NetBackup Hyper-V backups with Equallogic result in error 56

I've spent the better part of the last two days working with a co-worker who is a Hyper-V administrator trying to figure out why backing up VMs on his new Server 2012 R2 Hyper-V cluster were failing with error 156. We initially were using the default (auto) snapshot provider type, and that failed, showing a number of errors in the Hyper-V host's log relating to SNMP being unable to communicate with Physical Disks in the SAN.

This clued us in that it must be attempting to use snapshots at the array level rather than in the operating system. Changing the option to "Software" worked some of the time but still resulted in occasional 156 errors, changing it to "System" failed completely. But we opted not to engage Microsoft on this, but rather to try to solve the hardware-based snapshots first.

Today, he contacted Equallogic support, and together properly configured the Equallogic's support for VSS. Notably, enabling VSS access, and setting access permissions from "Volume Only" to "Volumes and Snapshots". That cleared up the error below:

In addition, the Host Integration Toolkit was upgraded to 4.7. After that was done, setting the snapshot type back to "auto" was successful, and performance is quite good, with the 1Gbps Hyper-V host NIC acting as the bottleneck for the backups.

Watching in the Equallogic management console, you can clearly see the snapshots being created at the array level during backups now.

I thought I'd post this, as administering vSphere on Equallogic, I would not have expected the array to get itself involved in backup operations unless explicitly configured, but apparently it does for Hyper-V deployed against Equallogic, at least when using NetBackup 7.6.0.2 with the snapshot provider type set to auto.

Reducing latency on Equallogic storage with VMware vSphere

I had an older Equallogic PS6000E SAN, configured for RAID 6 that was attached to a couple of vSphere hosts. Being comprised of a bunch of 1TB 7200 RPM SATA disks, it wasn't exactly built for performance and I would often see it top out on IOPS for long periods of time in SAN HQ. After a bit of shuffling in our other datacenter, I freed up a PS6000XV SAN (600GB 15,000 RPM disks, in RAID 10) and decided to add it to the same pool in order to utilize the auto-tiering capabilities and boost performance of the SATA SAN. My problems with IOPS were solved, but read latency remained stubbornly high. As I spent more time looking at the graphs, I realized that, strangely, the latency was highest when the IOPS were lowest, which is the opposite of what you'd expect. Shouldn't requests be answered faster when there is less work to do?

I did a bit of Googling, and decided to re-read the Best Practices for VMware guide for Dell's Equallogic storage. Buried inside there are two very helpful tips, that I don't remember being there years ago when I set up those SANs for the first time.

The important bits are found on pages 9-11. The section on Delayed ACK describes EXACTLY what I was seeing, so I disabled it, and Large Receive Offload (LRO) for good measure. Note that this will require a reboot of your hosts, but that's what we have vMotion for, right?

As you can see in the graphs below, the improvements in my read latency were pretty stunning and instant. If you are experiencing high latency during periods of relatively low IOPS with your Equallogic SANs, then definitely give this a try.

PowerShell script to create numerous DHCP scopes

Let's say you need to create a whole bunch of DHCP scopes in the Windows DHCP server, and you don't feel like spending hours using the wizard, or manually constructing all the netsh commands you need to do it from the command line. I had this very problem last week, so I hacked together this script to take a CSV file with all the details needed for the scopes, and output a .cmd file that I can simply run against my DHCP server to create them all. I've only included the DHCP options for router address, DNS servers, and DNS suffix, but you could certainly add more.

##==============================================================================
##==============================================================================
## SCRIPT.........: Create-Scope.ps1 
## AUTHOR.........: Seth H. Bokelman 
## EMAIL..........: seth.bokelman@uni.edu 
## VERSION........: 1 
## DATE...........: 2012-04-030 
## REQUIREMENTS...: Powershell v2.0 
## 
## DESCRIPTION....: Creates a CMD file to create numerous DHCP scopes 
## 
## NOTES..........: Requires CSV file with these fields: SCOPE, MASK, NAME, DESC 
## ROUTER, STARTIP, ENDIP, DNSSUFFIX 
## CUSTOMIZE......: 
##============================================================================== 
## START 
##============================================================================== 
# IP address of DHCP server 
$DHCPServer = "127.0.0.1"

#IP address of DNS servers 
$DNS1 = "127.0.0.1" 
$DNS2 = "127.0.0.1"

# Stores current date & time in a sortable format 
$date = Get-Date -format s

# Name of output batch file 
$outputfile = "C:DHCPscopes.cmd"

# Assumes a CSV with 8 columns listed above. 
$ips = import-csv "C:input.csv"

$ips | %{
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $DHCPServer add scope $($_.SCOPE) $($_.MASK) `"$($_.NAME)`" `"$date - $($_.DESC)`""
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) set optionvalue 3 IPADDRESS $($_.ROUTER)"
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) set optionvalue 6 IPADDRESS $DNS1 $DNS2"
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) set optionvalue 15 STRING `"$($_.DNSSUFFIX)`""
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) add iprange $($_.STARTIP) $($_.ENDIP)"
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $DHCPserver scope $($_.SCOPE) set state 1"
}
##==============================================================================
## END
##==============================================================================

Powershell script for creating DHCP reservation batch file

I've taken a handy script from Clint McGuire that creates batch files to aid in creating large groups of DHCP reservations and modified it a little bit to also insert the date in a sortable format at the start of the description field. Posting it here in case anyone else finds it useful:

##==============================================================================
##==============================================================================
## SCRIPT.........: Create-Reservation.ps1
## AUTHOR.........: originally: Clint McGuire, modified by Seth H. Bokelman
## EMAIL..........:
## VERSION........: 2
## DATE...........: 2012-04-020
## COPYRIGHT......: 2011, Clint McGuire
## LICENSE........:
## REQUIREMENTS...: Powershell v2.0
##
## DESCRIPTION....: Creates an CMD file to add reservations to DHCP.
##
## NOTES..........: Requires CSV file with 4 fields, IP, MAC, NAME and DESC
##
## CUSTOMIZE......:
##==============================================================================
## START
##==============================================================================
# IP address of DHCP server
$DHCPServer = "127.0.0.1"

# DHCP Scope you'd like reservations created for
$DHCPscope = "10.10.10.0"

# Stores current date & time in a sortable format
$date = Get-Date -format s

# Name of output batch file 
$outputfile = "C:\DHCPreservations.cmd"


# Assumes a CSV with four columns, MAC, IP, NAME and DESC.
$ips = import-csv "C:\accesspoints.csv"

$ips | %{
add-content -Encoding ASCII -Path $outputfile -Value "netsh Dhcp Server $DHCPServer Scope $DHCPScope Add reservedip $($_.IP) $($_.MAC) `"$($_.NAME)`" `"$date - $($_.DESC)`" `"DHCP`""
}
##==============================================================================
## END
##==============================================================================

My new PC, part III

Since at least a couple people reading this have expressed interest in building a similar machine, I'm going to continue explaining why I chose the parts I ddi for my new machine: Intel i5-2500K CPU

I typically prefer to use AMD processors whenever possible in both my home machines and in the servers I buy at work, but with the dominance that Intel is currently displaying on the desktop, choosing anything other than a "Sandy Bridge" chip didn't seem to make a lot of sense. I've always thought that AMD chips represented a very good value, and I think that continues to be true, but I was looking to get as much performance as I could reasonably afford, and had the budget for Intel. Intel has broken their mainstream chips into the i3, i5, and i7 families. The i3 chips are aimed at budget and entry-level machines, the i5 at mainstream machines, and the i7 at performance & enthusiast computers. Intel has also made a few parts, such as the i5-2500K and the i7-2600K that are purely aimed at enthusiasts building their own machines. The K designator in the model number indicates chips with an unlocked clock multiplier, chips that are practically guaranteed to overclock, as that's the only reason to care about an unlocked multiplier… I've had a few people ask me why I didn't go with the i7-2600 instead of my i5-2500K. My reasoning was simple, the only real difference between the two is hyperthreading support on the i7. Since I don't have a lot of apps (especially games) that can use more than 4 CPU cores, I didn't really feel it was worth the price premium. Sure, the i7 is 100mhz faster than the i5, but, using the stock cooler that came with my CPU, I easily overclocked my chip to 4.3Ghz just by using the auto-overclocking feature of my motherboard.

Antec Three Hundred Case

I chose the Antec case because it had pretty good reviews from real users, was from a quality manufacturer, and had a mounting space for a 2.5" hard drive. I like that all the edges inside are rounded, so you don't slice your hand up while working in it, and I also like that there's a handy chamber for stashing any extra power supply cables you're stuck with, if you're using a non-modular power supply like I am. The two included case fans are very quiet when run at their low setting, but they don't interface with the motherboard to let it throttle them, you have to open the case and flip their switches to the speed you desire. All-in-all, it's a solid choice, and my Radeon 6870 card easily fit between the case and the 3.5" hard drive spaces, and there are room for many drives. There is not an external 3.5" slot, however, so if you're dead-set on having a floppy drive in your case, you'll want to either get an adapter or look elsewhere. Same for a card reader.

ASUS LGA 1155 Intel Z68 Chipset Motherboard P8Z68-V PRO

I really waffled on what motherboard to purchase. I had a few requirements in mind. I really wanted 4 DIMM slots so I could expand beyond the initial 8GB of memory I purchased if I ever saw a smoking deal and wanted to go to 16GB. I also wanted the full ATX formfactor, and not one of the smaller variants, because my case had room for it. Sandy Bridge has three different chipsets, and the Z68 chipset is the top-of-the-line one, that incorporates the best parts of each of the two lesser chipsets. I initially had picked out this Gigabyte board, but it went out of stock as I was going to order it, so I stepped up to the more expensive Asus motherboard, and I can't say I'm disappointed. I was able to use the one-click overclock button in the Asus software to overclock my CPU to 4.3Ghz, and installing the drivers off the included DVD was pretty painless, with a nice little menu app that let you know what you were installing. The onboard audio works well, and this motherboard, combined with the i5-2500K, actually provides pretty decent integrated video as well. If you don't really plan on playing any modern 3D games on the system, you could easily get by with the HD3000 graphics chip integrated into the CPU until you find a good deal on a graphics card you want later. The motherboard manual actually was pretty helpful too, and everything is clearly labeled and explained, unlike the cheaper boards I've used in the past, where you get a poorly Xeroxed folded page of paper or two..

Tomorrow, I'll wrap this up talking about storage, and what the total cost of building this system was.