NetBackup Hyper-V backups with Equallogic result in error 56

I've spent the better part of the last two days working with a co-worker who is a Hyper-V administrator trying to figure out why backing up VMs on his new Server 2012 R2 Hyper-V cluster were failing with error 156. We initially were using the default (auto) snapshot provider type, and that failed, showing a number of errors in the Hyper-V host's log relating to SNMP being unable to communicate with Physical Disks in the SAN.

This clued us in that it must be attempting to use snapshots at the array level rather than in the operating system. Changing the option to "Software" worked some of the time but still resulted in occasional 156 errors, changing it to "System" failed completely. But we opted not to engage Microsoft on this, but rather to try to solve the hardware-based snapshots first.

Today, he contacted Equallogic support, and together properly configured the Equallogic's support for VSS. Notably, enabling VSS access, and setting access permissions from "Volume Only" to "Volumes and Snapshots". That cleared up the error below:

In addition, the Host Integration Toolkit was upgraded to 4.7. After that was done, setting the snapshot type back to "auto" was successful, and performance is quite good, with the 1Gbps Hyper-V host NIC acting as the bottleneck for the backups.

Watching in the Equallogic management console, you can clearly see the snapshots being created at the array level during backups now.

I thought I'd post this, as administering vSphere on Equallogic, I would not have expected the array to get itself involved in backup operations unless explicitly configured, but apparently it does for Hyper-V deployed against Equallogic, at least when using NetBackup 7.6.0.2 with the snapshot provider type set to auto.

Powershell script for creating DHCP reservation batch file

I've taken a handy script from Clint McGuire that creates batch files to aid in creating large groups of DHCP reservations and modified it a little bit to also insert the date in a sortable format at the start of the description field. Posting it here in case anyone else finds it useful:

##==============================================================================
##==============================================================================
## SCRIPT.........: Create-Reservation.ps1
## AUTHOR.........: originally: Clint McGuire, modified by Seth H. Bokelman
## EMAIL..........:
## VERSION........: 2
## DATE...........: 2012-04-020
## COPYRIGHT......: 2011, Clint McGuire
## LICENSE........:
## REQUIREMENTS...: Powershell v2.0
##
## DESCRIPTION....: Creates an CMD file to add reservations to DHCP.
##
## NOTES..........: Requires CSV file with 4 fields, IP, MAC, NAME and DESC
##
## CUSTOMIZE......:
##==============================================================================
## START
##==============================================================================
# IP address of DHCP server
$DHCPServer = "127.0.0.1"

# DHCP Scope you'd like reservations created for
$DHCPscope = "10.10.10.0"

# Stores current date & time in a sortable format
$date = Get-Date -format s

# Name of output batch file 
$outputfile = "C:\DHCPreservations.cmd"


# Assumes a CSV with four columns, MAC, IP, NAME and DESC.
$ips = import-csv "C:\accesspoints.csv"

$ips | %{
add-content -Encoding ASCII -Path $outputfile -Value "netsh Dhcp Server $DHCPServer Scope $DHCPScope Add reservedip $($_.IP) $($_.MAC) `"$($_.NAME)`" `"$date - $($_.DESC)`" `"DHCP`""
}
##==============================================================================
## END
##==============================================================================

High CPU load on domain controllers caused by imaging clients

Here’s an interesting issue we’ve encountered at work.  This also demonstrates how useful Cacti can be in graphing what’s going on in your server environment.  Even if you don’t have alerting, simply graphing what your “normal” usage is can alert you to problems as well as help you put timelines together that allow you to figure out the source.

I was glancing over some server utilization charts and saw this one, which shows CPU usage jumped way off the charts over Christmas break:

Why is that?  We didn’t suddenly quintuple in size?  Digging closer, I realized a definite pattern to the CPU load:

You can see that the load quiets down at night, and goes up in the morning, in pretty much a specific pattern.  Digging into the graphs revealed that it was almost down to the minute, which probably means it’s some type of automated source, as no user works from 7:00 a.m. to 10:30 p.m. every day.

The network graph showed a similar pattern, which gave me a good shot of capturing it.  I turned on Network Monitor, and ran a capture for 30 seconds, here’s what it saw:

Holy cow!  In 30 seconds I saw over 30,000 Frames from some of those clients!  The highest “normal” client I see is 136 frames in that time period.  Now that I knew where the traffic was coming from, I just had to figure out why.  I contact the administrator for those clients, who said that they had been re-imaged in the last few weeks.

Further investigation has shown these clients to have all been imaged with an improperly prepared image, causing them to essentially fight over the same records in Active Directory and DNS continuously.  The quiet periods were when the computer labs were shut down for the evening automatically to save energy, and then automatically powered back up in the morning before classes started in them.