My friend Sarah

This week, my friend and co-worker Sarah Sand suddenly passed away. Sarah was an exceptional co-worker and a great human being. Her obituary is here, if you’d like to read it.

Sarah and I didn’t work on many projects together, and she certainly had many closer friends at work, but we spent time together outside of work as well. When our company instituted a new policy that gave us Friday afternoons free in the summer, she was part of a group of friends who would come to my house and eat pizza, play board games, have a couple drinks, and maybe blow off a bit of steam after a week at work. In February, we all decided to get together on a Saturday to play, as we didn’t want to wait for summer to come, and right now I’m really thankful we did that.

Sarah was smart, and she was extremely good at her job. Not just good at the technical work, but personally dedicated to building tools and processes that would help others succeed as well. When I needed help figuring out how to use a new tool she oversaw she was happy to spend half an hour showing me how to do it, with infinite patience in dealing with my gripes. Those qualities served her well when we’d play games together too, she’d often quietly be amassing the points necessary to win a game of Settlers of Catan, while several of us guys would be loudly taunting each other and not seeing her strategy until it was too late to stop her even when we would gang up against her.

It’s not an exaggeration to say that Sarah was universally liked by all of us who worked with her. In IT, where egos and arguments and turf wars can often get in the way of progress, she was always agreeable, patient, and calm. She cared about her work, and one day at my house while waiting for others to arrive, she expressed some very real and raw exasperation at not being able to implement her plans for a work project, and then immediately apologized for it. I told her she had nothing to apologize for and that every one of us has had those moments, but what struck me was that her frustration wasn’t so much about not getting her way but that she felt it would hurt the users of the systems under her care.

Yesterday, while sharing a bit of mourning with a coworker, we both hit upon the same word that described her most of all. Kind. She would have kind words for those around her, every day. She was kind and supportive to complete strangers on the Internet. She had empathy at her core, and as someone who sometimes struggles to find that in myself, she awed me with her innate abilities.

In our February gaming session, I’d cobbled together a bag of random nerdy leftovers from various LootCrate boxes that I didn’t want or need, but thought others might, just to give us something to play for, as the winner of each game could pick something they wanted from the bag. She spied a particular item she wanted to win, a little paper notebook shaped like an original Star Trek communicator. I asked her if she liked it, and she said she really wanted to win it not for herself, but rather to give it to a coworker that she knew would love it.

That was Sarah. She was my friend, and I miss her.

How do you find those fares?

Most of our friends and family know that we like to travel, and that we don't like spending a ton of money to do so.  The Boarding Area has a great blog post today that explains a little bit about jumping on "mistake fares", and I'd strongly suggest reading it if you're interested in scoring a ticket to an exotic location for less than you'll often pay to fly domestically.

It's important to note that if you're the kind of person who wants to visit a specific city on a specific weekend, or you need a week to "think about it" before booking an international trip, you might as well not even bother.  The keys to scoring these deals are to notice them when they happen (I follow some key Twitter accounts that tend to broadcast them, such as The Flight Deal, which is mentioned in the article) and booking them immediately, often within an hour or two of them being found.  Many of the airlines will let you cancel a fare for free within the first 24 hours, so book the trip first, then ask your boss for the time off!

Often these cheap fares tend to be for travel in the winter or shoulder seasons, but we've found that the lack of crowds often makes up for slightly cooler weather.

Here are a few examples of crazy cheap sales and/or mistake fares that we've flown, all were round trip:

Chicago to London for $221
London to Florence for $16
Chicago to Paris for $250
Chicago to Johannesburg, South Africa for $390
Chicago to Beijing for $528
Minneapolis to Copenhagen for $351

Other Twitter accounts to follow:

Fare Deal Alert

Steal This Trip

AirFare Watchdog (Consider signing up to receive a daily email from their site showing you the fares that are on sale from your home airport as well)

Rest in peace, Grandpa

My grandfather, Henry Hollis Bokelman, passed away last month following a couple years of poor health.  I lived with him and my grandmother for my last three years of high school, and it was difficult watching someone so robust slide into the infirmity of old age the last couple of years.

Here is his obituary:

Henry “Hank” Hollis Bokelman, 88 of Ventura, died Monday, June 8, 2015, at Oakwood Care Center in Clear Lake.

A funeral service will be 1:30 p.m. Thursday, June 11, 2015, at Ward-Van Slyke Colonial Chapel, 101 N. 4th St., Clear Lake, with Pastor Scott Kozisek officiating. Burial will be in Clear Lake Cemetery, with military honors provided by the Clear Lake Veterans of Foreign Wars Post 4868. Visitation will be from 5:00-7:00 p.m. Wednesday, June 10, 2015, at Ward-Van Slyke Colonial Chapel.

Hank was born June 18, 1926, the son of Hollis and Vida (Rosenau) Bokelman, in Ventura. He grew up and attended school in Ventura, graduating from Ventura High School. He was united in marriage to Irma Schultz on June 22, 1946 at the Little Brown Church in Nashua. He was a United States Army veteran, stationed at Camp Sykes in Kyongju, Korea. After serving his country, he returned home to Ventura, where he farmed all of his life.

Hank was a member of the Veterans of Foreign Wars Post 5515 in Garner. Besides spending time with his family, Hank enjoyed fishing and hunting. He took many trips to Montana to hunt elk, and a few years ago took a dream trip to Canada to catch the big ones.

Hank is survived by his wife, Irma Bokelman of Ventura; four children, Hank (Ann) Bokelman of Hanlontown, John Bokelman of Ventura, Ruth (James) Conn of Volin, S.D., and Jane (David) Easton of Cedar Falls; seven grandchildren, Seth (Holly) Bokelman, Jessica (Scott) Rosendaul, William Conn, Elizabeth (Mitch) Hessman, Ashley (Lee) Geisinger, Cole Easton, and Tess Easton; and five great-grandchildren, Lela Geisinger, Jax Geisinger, Ella Rosendaul, Ethan Rosendaul, and Dylan Rosendaul.

He was preceded in death by his parents.

NetBackup Hyper-V backups with Equallogic result in error 56

I've spent the better part of the last two days working with a co-worker who is a Hyper-V administrator trying to figure out why backing up VMs on his new Server 2012 R2 Hyper-V cluster were failing with error 156. We initially were using the default (auto) snapshot provider type, and that failed, showing a number of errors in the Hyper-V host's log relating to SNMP being unable to communicate with Physical Disks in the SAN. 1stError-WhenVDS-VSS-accessNotEnabled

This clued us in that it must be attempting to use snapshots at the array level rather than in the operating system. Changing the option to "Software" worked some of the time but still resulted in occasional 156 errors, changing it to "System" failed completely. But we opted not to engage Microsoft on this, but rather to try to solve the hardware-based snapshots first.

Today, he contacted Equallogic support, and together properly configured the Equallogic's support for VSS. Notably, enabling VSS access, and setting access permissions from "Volume Only" to "Volumes and Snapshots". That cleared up the error below:

errorOnEQLwhenAccessPolicyOnlyAppliedToVolumesNotVolsAndSnapshots

In addition, the Host Integration Toolkit was upgraded to 4.7. After that was done, setting the snapshot type back to "auto" was successful, and performance is quite good, with the 1Gbps Hyper-V host NIC acting as the bottleneck for the backups.

Watching in the Equallogic management console, you can clearly see the snapshots being created at the array level during backups now.

I thought I'd post this, as administering vSphere on Equallogic, I would not have expected the array to get itself involved in backup operations unless explicitly configured, but apparently it does for Hyper-V deployed against Equallogic, at least when using NetBackup 7.6.0.2 with the snapshot provider type set to auto.

Reducing latency on Equallogic storage with VMware vSphere

I had an older Equallogic PS6000E SAN, configured for RAID 6 that was attached to a couple of vSphere hosts. Being comprised of a bunch of 1TB 7200 RPM SATA disks, it wasn't exactly built for performance and I would often see it top out on IOPS for long periods of time in SAN HQ. After a bit of shuffling in our other datacenter, I freed up a PS6000XV SAN (600GB 15,000 RPM disks, in RAID 10) and decided to add it to the same pool in order to utilize the auto-tiering capabilities and boost performance of the SATA SAN. My problems with IOPS were solved, but read latency remained stubbornly high. As I spent more time looking at the graphs, I realized that, strangely, the latency was highest when the IOPS were lowest, which is the opposite of what you'd expect. Shouldn't requests be answered faster when there is less work to do?

I did a bit of Googling, and decided to re-read the Best Practices for VMware guide for Dell's Equallogic storage. Buried inside there are two very helpful tips, that I don't remember being there years ago when I set up those SANs for the first time.

The important bits are found on pages 9-11. The section on Delayed ACK describes EXACTLY what I was seeing, so I disabled it, and Large Receive Offload (LRO) for good measure. Note that this will require a reboot of your hosts, but that's what we have vMotion for, right?

As you can see in the graphs below, the improvements in my read latency were pretty stunning and instant. If you are experiencing high latency during periods of relatively low IOPS with your Equallogic SANs, then definitely give this a try.

SAN HQ Latency Graph

latency2

PowerShell script to create numerous DHCP scopes

Let's say you need to create a whole bunch of DHCP scopes in the Windows DHCP server, and you don't feel like spending hours using the wizard, or manually constructing all the netsh commands you need to do it from the command line. I had this very problem last week, so I hacked together this script to take a CSV file with all the details needed for the scopes, and output a .cmd file that I can simply run against my DHCP server to create them all. I've only included the DHCP options for router address, DNS servers, and DNS suffix, but you could certainly add more.

##==============================================================================
##==============================================================================
## SCRIPT.........: Create-Scope.ps1 
## AUTHOR.........: Seth H. Bokelman 
## EMAIL..........: seth.bokelman@uni.edu 
## VERSION........: 1 
## DATE...........: 2012-04-030 
## REQUIREMENTS...: Powershell v2.0 
## 
## DESCRIPTION....: Creates a CMD file to create numerous DHCP scopes 
## 
## NOTES..........: Requires CSV file with these fields: SCOPE, MASK, NAME, DESC 
## ROUTER, STARTIP, ENDIP, DNSSUFFIX 
## CUSTOMIZE......: 
##============================================================================== 
## START 
##============================================================================== 
# IP address of DHCP server 
$DHCPServer = "127.0.0.1"

#IP address of DNS servers 
$DNS1 = "127.0.0.1" 
$DNS2 = "127.0.0.1"

# Stores current date & time in a sortable format 
$date = Get-Date -format s

# Name of output batch file 
$outputfile = "C:DHCPscopes.cmd"

# Assumes a CSV with 8 columns listed above. 
$ips = import-csv "C:input.csv"

$ips | %{
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $DHCPServer add scope $($_.SCOPE) $($_.MASK) `"$($_.NAME)`" `"$date - $($_.DESC)`""
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) set optionvalue 3 IPADDRESS $($_.ROUTER)"
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) set optionvalue 6 IPADDRESS $DNS1 $DNS2"
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) set optionvalue 15 STRING `"$($_.DNSSUFFIX)`""
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $dhcpserver scope $($_.SCOPE) add iprange $($_.STARTIP) $($_.ENDIP)"
add-content -Encoding ASCII -Path $outputfile -Value "netsh dhcp server $DHCPserver scope $($_.SCOPE) set state 1"
}
##==============================================================================
## END
##==============================================================================

Powershell script for creating DHCP reservation batch file

I've taken a handy script from Clint McGuire that creates batch files to aid in creating large groups of DHCP reservations and modified it a little bit to also insert the date in a sortable format at the start of the description field. Posting it here in case anyone else finds it useful:

##==============================================================================
##==============================================================================
## SCRIPT.........: Create-Reservation.ps1
## AUTHOR.........: originally: Clint McGuire, modified by Seth H. Bokelman
## EMAIL..........:
## VERSION........: 2
## DATE...........: 2012-04-020
## COPYRIGHT......: 2011, Clint McGuire
## LICENSE........:
## REQUIREMENTS...: Powershell v2.0
##
## DESCRIPTION....: Creates an CMD file to add reservations to DHCP.
##
## NOTES..........: Requires CSV file with 4 fields, IP, MAC, NAME and DESC
##
## CUSTOMIZE......:
##==============================================================================
## START
##==============================================================================
# IP address of DHCP server
$DHCPServer = "127.0.0.1"

# DHCP Scope you'd like reservations created for
$DHCPscope = "10.10.10.0"

# Stores current date & time in a sortable format
$date = Get-Date -format s

# Name of output batch file 
$outputfile = "C:\DHCPreservations.cmd"


# Assumes a CSV with four columns, MAC, IP, NAME and DESC.
$ips = import-csv "C:\accesspoints.csv"

$ips | %{
add-content -Encoding ASCII -Path $outputfile -Value "netsh Dhcp Server $DHCPServer Scope $DHCPScope Add reservedip $($_.IP) $($_.MAC) `"$($_.NAME)`" `"$date - $($_.DESC)`" `"DHCP`""
}
##==============================================================================
## END
##==============================================================================

My new PC, part III

Since at least a couple people reading this have expressed interest in building a similar machine, I'm going to continue explaining why I chose the parts I ddi for my new machine: Intel i5-2500K CPU

I typically prefer to use AMD processors whenever possible in both my home machines and in the servers I buy at work, but with the dominance that Intel is currently displaying on the desktop, choosing anything other than a "Sandy Bridge" chip didn't seem to make a lot of sense. I've always thought that AMD chips represented a very good value, and I think that continues to be true, but I was looking to get as much performance as I could reasonably afford, and had the budget for Intel. Intel has broken their mainstream chips into the i3, i5, and i7 families. The i3 chips are aimed at budget and entry-level machines, the i5 at mainstream machines, and the i7 at performance & enthusiast computers. Intel has also made a few parts, such as the i5-2500K and the i7-2600K that are purely aimed at enthusiasts building their own machines. The K designator in the model number indicates chips with an unlocked clock multiplier, chips that are practically guaranteed to overclock, as that's the only reason to care about an unlocked multiplier… I've had a few people ask me why I didn't go with the i7-2600 instead of my i5-2500K. My reasoning was simple, the only real difference between the two is hyperthreading support on the i7. Since I don't have a lot of apps (especially games) that can use more than 4 CPU cores, I didn't really feel it was worth the price premium. Sure, the i7 is 100mhz faster than the i5, but, using the stock cooler that came with my CPU, I easily overclocked my chip to 4.3Ghz just by using the auto-overclocking feature of my motherboard.

Antec Three Hundred Case

I chose the Antec case because it had pretty good reviews from real users, was from a quality manufacturer, and had a mounting space for a 2.5" hard drive. I like that all the edges inside are rounded, so you don't slice your hand up while working in it, and I also like that there's a handy chamber for stashing any extra power supply cables you're stuck with, if you're using a non-modular power supply like I am. The two included case fans are very quiet when run at their low setting, but they don't interface with the motherboard to let it throttle them, you have to open the case and flip their switches to the speed you desire. All-in-all, it's a solid choice, and my Radeon 6870 card easily fit between the case and the 3.5" hard drive spaces, and there are room for many drives. There is not an external 3.5" slot, however, so if you're dead-set on having a floppy drive in your case, you'll want to either get an adapter or look elsewhere. Same for a card reader.

ASUS LGA 1155 Intel Z68 Chipset Motherboard P8Z68-V PRO

I really waffled on what motherboard to purchase. I had a few requirements in mind. I really wanted 4 DIMM slots so I could expand beyond the initial 8GB of memory I purchased if I ever saw a smoking deal and wanted to go to 16GB. I also wanted the full ATX formfactor, and not one of the smaller variants, because my case had room for it. Sandy Bridge has three different chipsets, and the Z68 chipset is the top-of-the-line one, that incorporates the best parts of each of the two lesser chipsets. I initially had picked out this Gigabyte board, but it went out of stock as I was going to order it, so I stepped up to the more expensive Asus motherboard, and I can't say I'm disappointed. I was able to use the one-click overclock button in the Asus software to overclock my CPU to 4.3Ghz, and installing the drivers off the included DVD was pretty painless, with a nice little menu app that let you know what you were installing. The onboard audio works well, and this motherboard, combined with the i5-2500K, actually provides pretty decent integrated video as well. If you don't really plan on playing any modern 3D games on the system, you could easily get by with the HD3000 graphics chip integrated into the CPU until you find a good deal on a graphics card you want later. The motherboard manual actually was pretty helpful too, and everything is clearly labeled and explained, unlike the cheaper boards I've used in the past, where you get a poorly Xeroxed folded page of paper or two..

Tomorrow, I'll wrap this up talking about storage, and what the total cost of building this system was.

My new PC, part II

As I wrote yesterday, I’ve built a new gaming PC for home, and I promised to write some more today about why I chose certain components.  I’ll write about two of them today, and talk a little bit about power consumption.

Antec 620W Neo Eco Power Supply

The first component I bought was the power supply.  Power supplies aren’t very exciting components, and the main reason I picked this one is that I saw it on a special on Slickdeals.  I knew that I wanted a power supply with enough power that I could eventually run two video cards in my computer (ATI/AMD calls this Crossfire) and this power supply can do that.  Also, I wanted something relatively energy efficient, and since this one has the 80 Plus certification, I won’t be wasting a lot of power by using it.  Antec is generally a quality brand, and this is a quality (and heavy)  power supply, but it isn’t a “modular” power supply, where you only attach the cables to power the specific components needed in your build.  As a result, I have a couple extra cables in my case, and it makes for a bit of a mess when you’re wiring it all up.  If I was doing this over again, I’d wait a bit longer for a deal to pop up on a modular supply, as it would make the finished product look much tidier.

HIS Radeon 6870 Video Card (H687F1G2M)

I spent a lot of time agonizing over which video card to purchase.  There are a LOT of options for video cards, at many different price points, and typically the way I choose one is by finding the fastest card at the price point I’m willing to spend.  I read numerous benchmarks, and fortunately, AnandTech is currently using Civilization 5 as one of their benchmarking games.  I’ve been playing this game quite a bit lately, so it’s a very useful benchmark, and it was pretty clear that in the $150-ish price range, the NVidia GeForce 460 cards offered the most bang for the buck in Civ 5.  However, I’ve also been mining some Bitcoins lately with my hardware, and for Bitcoin mining, the Radeon cards are the only way to go.  They’re also sort of confusing to buy, as the performance doesn’t scale cleanly with price, due to the way that the mining software uses the processing power of the card.  After pouring over a lot of breakdowns of cost, mining performance, and energy consumption, I decided on a Radeon 6870 card as being something that would perform well in Bitcoin mining, in games, and still come in at a price I could live with.  It’s actually faster than the GeForce 460 I mentioned above in most games, except for Civilization 5, but honestly, if you’re not interested in Bitcoin mining, go for the 460, you can often find good deals on them listed on Slickdeals, and you can save at least $25 over the cost of the Radeon 6870, which ran me $165 after rebate.  If you want to stick with the Radeon family, the slightly-slower Radeon 6850 is also a good choice, it’s just not nearly as good at Bitcoin mining as its bigger brother.

 

Power Consumption

So, how much power does this new rig of mine use?  Can I actually turn a profit on my Bitcoin mining?  I plugged in my trusty Kill-A-Watt tonight to find out, and here are the results, not counting the monitor:

 

State Power consumption (watts)
Off 0
Booting 100-130
Idle/Login Screen 70
Mining Bitcoins 182
Civilization V 196
Sleep 1

So, I’m clearly not stressing my 620W power supply yet, but these numbers let us easily calculate what it’s costing me to mine Bitcoins.  Our power costs us about 7.8 cents per kw-hr, so when mining Bitcoins, I’m using about 34 cents worth of power per day versus leaving my computer turned off.  At my current rate, I can earn a Bitcoin about every 4 days, and they’re currently trading at over $15 each, so I could make $13 every four days in profit.  (Obviously, that heat goes somewhere, so my air conditioning will have to work slightly harder to dissipate that heat in the summer, but I'll save natural gas in the winter, so we’ll call it a wash).

 

Clearly, putting the computer to sleep is a good way to cut down on your power bill, but even leaving it idle isn’t going to break the bank, at a cost of only 13 cents per day.