High CPU load on domain controllers caused by imaging clients

Here’s an interesting issue we’ve encountered at work.  This also demonstrates how useful Cacti can be in graphing what’s going on in your server environment.  Even if you don’t have alerting, simply graphing what your “normal” usage is can alert you to problems as well as help you put timelines together that allow you to figure out the source.

I was glancing over some server utilization charts and saw this one, which shows CPU usage jumped way off the charts over Christmas break:

Why is that?  We didn’t suddenly quintuple in size?  Digging closer, I realized a definite pattern to the CPU load:

You can see that the load quiets down at night, and goes up in the morning, in pretty much a specific pattern.  Digging into the graphs revealed that it was almost down to the minute, which probably means it’s some type of automated source, as no user works from 7:00 a.m. to 10:30 p.m. every day.

The network graph showed a similar pattern, which gave me a good shot of capturing it.  I turned on Network Monitor, and ran a capture for 30 seconds, here’s what it saw:

Holy cow!  In 30 seconds I saw over 30,000 Frames from some of those clients!  The highest “normal” client I see is 136 frames in that time period.  Now that I knew where the traffic was coming from, I just had to figure out why.  I contact the administrator for those clients, who said that they had been re-imaged in the last few weeks.

Further investigation has shown these clients to have all been imaged with an improperly prepared image, causing them to essentially fight over the same records in Active Directory and DNS continuously.  The quiet periods were when the computer labs were shut down for the evening automatically to save energy, and then automatically powered back up in the morning before classes started in them.

Installing BlackBerry OS 5 on Sprint Blackberry 8330

I took the plunge earlier this week and installed the new BlackBerry operating system on my Sprint BlackBerry 8330 phone.  I’ve had really good luck with it, it’s been stable, and the improvements to the browser and messaging portions of the operating system have been well worth the time I spent getting it working.  In order to save others some time, here’s the procedure I used.  The only downsides I encountered were that I had less memory available afterwards in the phone (the new OS is larger) and I had to tweak permissions for some of the Google Apps to get them working.

Disclaimer:  This is unsupported by Sprint.  You may kill your phone doing this.  You may lose data.  You may trigger the apocalypse.  If you do so, it’s not my fault.  This worked for me, but you have been warned.  This very well may work on Verizon and U.S. Cellular 8330 phones as well, but I haven’t tested it.

Step 1. Plug your BlackBerry into your PC via USB and back up all your data via Desktop Manager.  I sync my contacts to Gmail using Google Sync, so it was easy for me to restore them back into the phone once I was done, but you’ll want to back up all the data on your phone before you start this.

Step 2. Download the Boost Mobile 5.0.0.438 version of the OS from this site. Install it on your PC.

Step 3. Download version 1.7 (or newer) of BBSAK from this site, you’ll need to register for a free account to access the download.  Install it on your PC.

Step 4. Make sure your backups are good before going any farther.  Launch BBSAK, (your password is probably blank).  Select “Wipe Device” and BBSAK will nuke your BlackBerry.  It’ll take a few minutes, and will automatically reboot, then give you an error message on the screen, indicating that your OS is gone. This is normal.

Step 5. Click the Load OS button in BBSAK.  This will now let you load OS 5 onto your phone.  You can select or deselect a few optional components that you want to load onto the phone, then wait as it copies the OS over, and reboots.

Step 6. Wait.  Be patient, seriously.  A progress bar will pop up on the phone as it boots for the first time.  It’ll appear to hang at around the 60% mark, but it’s not frozen.  Go find something else to do for 10-20 minutes, and it’ll be done by the time you come back.

Step 7. Start reinstalling all your data and apps.  It should now be up and running.  Performance may be lousy at first, as it re-downloads all the shortcuts from Sprint for things like Sprint Navigation, the IM clients, the software store, etc.  You’ll want to reinstall the pieces you use on the phone, then start adding in the third party software pieces you use, including BlackBerry App World.  I had to modify the permissions for some of the Google Applications, which you can do in Options –> Applications in the phone.  Google Sync refused to install properly using the installer that installs all Google applications, but going to the Google Sync web site with my BlackBerry directly and installing just that piece worked fine.

Step 8. Enjoy!  You’re now the first kid on the block with BlackBerry OS 5 on your 8330!  Performance will improve after the OS settles down from that initial boot, and I’ve had no issues with lockups or memory leaks, but as always, your mileage may vary!

Jungle Disk

I installed Jungle Disk on my home PC last night to use for backing up my personal data. I've been using Mozy for the last year, and was pretty happy with it, but since I don't back up a ton of data, I think Jungle Disk will be slightly cheaper, and has a few unique features. I chose RackSpace as the location to store my data, though Amazon's S3 service is the default. They both charge 15 cents per GB per month to host your data, but S3 charges an additional bit for every GB you transfer in or out, and RackSpace doesn't charge that. I'm not sure why it defaults to S3, when RackSpace is cheaper, but it does.

I'm very curious about their "Enterprise" offering. The rate they're charging is very very low, but at the same time, they aren't EMC, they aren't Symantec, and I wonder how many enterprises are really ready to trust their data to Jungle Disk...

I'm using the Desktop edition, and I like that it lets you also mount your backup space as a drive, though I was a bit dismayed at first that I had to schedule a particular time to do my backups, as I tend to leave my PC asleep. Going through the options, I saw it had the ability to wake my PC from sleep, as well as perform my backups at the next time the machine was on, which is how I actually prefer to run it. I also liked that it let me throttle bandwidth usage during certain hours of the day, so that I can turn it loose from midnight to 6:00 a.m., but it won't eat up all my bandwidth when I'm likely to be using it. I do wish that it would let me configure separate settings based on days of the week though, I'd be fine with giving up all of my bandwidth during weekdays too, but not on weekends.

reCAPTCHA

I've updated some of the software that drives this blog, so you can now post a comment without creating a username, as long as you solve the reCAPTCHA to post it. If you get one that you can't decipher, click the two arrows chasing each other to get a different one. If you're a registered user, you'll only have to solve one reCAPTCHA, at the time you register. I'd initially disabled un-registered comments to avoid spam, but was still getting registration spam. Hopefully, by using reCAPTCHA, both problems have now been solved!

Mozy

I've been using Mozy ( now owned by EMC) for the last couple of months for my home PC, and I'm pretty impressed. The initial backup takes a while, (14 hours for me) due to the slow uploads that are typical of residential cable internet, but my daily backups tend to take less than a minute, since the software is smart enough to back files up at the block level, rather than the file level. With a little Googling for coupons, it wound up costing me about $40 for a year of unlimited backup service, which seems reasonable to get all my files, photos, and tax records automatically out of the house daily to save me from disaster. I also looked at their MozyPro service, which is actually pretty decently priced as compared to the backup service I run at work. They charge $6.95 per server per month, plus 50 cents per GB per month. The major downside that I see is that you can only restore from the last 30 days. That's really all you need for disaster recovery, of course, but it doesn't protect you from the user who deletes something and doesn't notice it's gone until the next semester.

While I don't think Mozy is going to be replacing our campus backup system anytime soon, it might be a viable alternative for those areas who are doing workstation backups, as it eliminates the need to leave the machine on for the scheduled backup job (Mozy just runs whenever the user turns the box on). Workstations are cheaper than servers, at $3.95/month + 50cents/GB.

Storing Student Stuff, Part II

If you haven't already, read Part I first. So, what am I considering for hardware? I'm considering several options, here is my homegrown solution, if I couldn't afford a Filer/NAS device:

Homegrown student storage system:

Dell PowerEdge 2970 Server

Dell PowerVault MD3000i iSCSI SAN

Dell PowerVault DL2000 disk-based backup system

I planned on filling that MD3000i with 15 1TB disks, and when you get to that level of storage, you seriously need to consider how you're going to back it up. Right now, we protect a little over 15TB with our current backup systems on campus, so I'd nearly be doubling the size if I fed this data into our existing system. Doing full backups over the network each week would take far too long, so I'd need to utilize synthetic backups in order to reduce the amount of network traffic and the size of the backup window required. To do that, I looked at the DL2000 system, with two shelves of disk storage attached to it. This is essentially a 2U Dell Server, with Windows and Backup Exec (or CommVault) installed, and shelves of disk attached to the back of the server to use for storing the backups. While we typically use NetBackup, I've fairly familiar with BackupExec, and knew that it could perform synthetics necessary to reduce network traffic. Purchasing a dedicated backup system for this project also made some sense because I could predict the annual cost of the backups quite easily. Our NetBackup system is run on a charge-back basis, based upon how many GB each "customer" on campus is protecting with the system, as that number would fluctuate wildly, so would backup charges. Using our NetBackup system would be cost-effective if the student storage system stayed small, say under 2 TB, but if it hit 10TB, it would be extremely expensive to feed into our other backup boxes.

Now, if I go back to my list of possible protocols that would be useful in this project, which can I provide with this solution?

CIFS - This is easy, it's built into the Windows server I'd use

NFS - Should be possible to do in Windows, maybe with Services For UNIX, but we've really never done it before, and permissions can be VERY tricky. NetApp has a white paper laying out the problems associated with mapping NFS to CIFS permissions, and how they tackle it. I'm pessimistic about our ability to solve the problem nearly as well on our own.

WebDAV - Again, this is theoretically possible in Windows too, but we haven't done it before either. It may work easily, it may be extremely problematic for us.

HTTP/Web Access - I have yet to find a good piece of software for doing this, there are plenty for allowing web users to store files inside databases, I have yet to see a good one for sharing an NTFS file system through a web page.

SFTP - Again, should be possible, if a suitable bolt-on SFTP server can be found, though the ugly permissions monster may attack here too.

So what are the pros & cons of this approach?

Pros:

It's relatively cheap

It's as flexible as I can make a Windows server, if I can find the right software packages and make them work.

It's based on Dell hardware, which we have a lot of experience with.

Cons:

It's not redundant, if a hardware problem strikes the server, it's going to be down for 4 hours while we get replacement parts

Implementing so many simultaneous file protocols would be extremely difficult. I have grave doubts about our ability to make all of that work well, and it would take a very long time to accomplish it.

"Only" scales to 45TB. You can hook three MD3000i boxes in tandem, but that's it. I don't realistically expect to need more in the lifetime of this project, but it is a limitation that's smaller than my other options.

Monthly downtime for Windows patches, as well as patches for various file serving protocols. I know a Filer will have better uptime, just because you don't have to install security patches monthly.

Disaster Recovery - In the event that the system is destroyed, purchasing a new one will take time, and restoring the backups to it could take up to a couple days. No big deal if the problem happens over Christmas break, but if it's the weekend before Finals, it would be bad.

Stay tuned for Part III next week.

WordPress 2.7.1

Just testing my blog, now that I've updated to WordPress 2.7.1. It should look the same to the users, but the UI is waaaay different on my end...

Storing Student Stuff, Part I

For the last few months, I've been working on a project to provide some file storage space to all 12,500 of our students here at UNI. Though many (most?) other schools do provide storage space, we've never done it in a central organized fashion, though a few of the Colleges have dabbled with it over the years. I've met with the IT staff from the colleges to try to build a list of requirements, and met with students to ask what they want... Then, I tried to figure out what technology to use to meet those requirements. The obvious one is the typical file server, but one of the requirements I got back was high availability, which really translates to clustering, and clustering tends to translate to being a pain in the backside to set up and maintain, and the complexity it introduces can often turn into more downtime with configuration issues than the clustering prevents during hardware failures.

So, that got me thinking about Filer technology, something I'd never worked with, but I knew that Joe Breu, a former sysadmin at our municipal cable utility who now works for Rackspace did. I asked him about it, and he suggested I look at a NetApp filer. I was quite impressed, but I've also been looking at NAS type devices from EMC and Sun as well.

So how do all these choices stack up? What are our requirements? What is the best fit? What can we afford? Which of the requirements are "hard" requirements and which are just wishes? How much space does each student need? How big should we make the disk quotas? What percentage of students will use the service? What percentage of students will utilize their entire disk quota? How are we going to back all this storage up? What will backing this up cost? These are the questions I've been attempting to answer over the last few weeks, so I decided to blog it all to try and help keep it all straight.

The first thing we tried to establish is what protocols were possible for accessing the data by students.

CIFS - This is a no brainer, all the Windows boxes will use it

NFS - Probably not a hard requirement, but definitely a nicety for the Linux/UNIX users and Macintoshes.

WebDAV - This would provide the ability for students to mount their files as a drive while off-campus or on their personal machines.

HTTP/Web Access - Some sort of web page for students to easily upload and download a file, for quick remote access

SFTP - So tech-savvy students can easily upload/download large numbers of documents remotely, yet not broadcast their passwords in plain-text like FTP would

So, obviously these things can be accomplished on a traditional server, but, trying to make them all work together seamlessly would be difficult. Having NFS & CIFS permissions coexist can be tricky, and I have yet to find a good web access application for sharing files from a filesystem on the web. There are plenty for storing files in a database, but I need to do it from a filesystem...

Continue to Part II.

My first gamble

I made my first stock purchase today (not counting all the stuff in my retirement account, but I don't really pick those). I decided to toss a very small amount of money at Sprint, because I think they're undervalued now, and ripe for a takeover/buyout. I know there are fears of bankruptcy, but they have very real assets (spectrum & customers) that are worth real dollars. So, I risked a small amount of money (less than my last video game system) on some of their stock, now to see if it pays off...

Photosynth

I spent a little time this morning playing with Photosynth, the cool new photo tool released by Microsoft today. I snapped 62 pictures of my cubicle, then uploaded them to see if it would automatically stitch them together, and sure enough, it did. It's not the most exciting one on the site, but if you'd like to see my messing cube in some weird 3D-ish photo-mosaic, here it is. Note: You need to be on a PC, it doesn't work for Macs yet. Maybe this weekend I'll shoot a few hundred pictures of our house and see if I can make a decent one there from that of an outdoor environment.