I love it when a plan comes together

Part of my plan for 2010 was to get more organized, both at work and in my personal life. We all have things we should be doing but have put aside for one reason or another. In my case, it's usually just laziness or a lack of motivation. Recently, I took the time to document the wiring of our server rack, something I had been meaning to do for a long time. As part of this, I was able to then go through the Netapp vSphere storage best practices and identify a few places where I hadn't implemented certain features, most notably, iSCSI multipathing. This came just after we had gone through Netapp upgrades to enable MPHA (multi-path high availability) on our FiberChannel connections.

Following this wonderful blog post on implementing iSCSI multipathing in vSphere and the best practices guide, I turned our single path failover NIC iSCSI connection into a multi-path round-robin load balanced one. You can see from the graph above when it was activated and the two NICs began to work in harmony. Since we currently only have one iSCSI LUN, the benefits are not great, but I will be adding a second one later today which should start to show the benefits of the configuration.

BackupExec, VSS, and Exchange (Oh My!)

It turns out that my problem yesterday with BackupExec is a strange interaction between BackupExec 11d, Volume Shadow Services, and Exchange backups.

What normally happens is BackupExec calls out to the Volume Shadow Services on a Windows system and tells it to snapshot anything related to what it is about to back up. This allows the applications to put themselves into a consistent state in preparation for a backup. This is critical for databases like Exchange. In my case, this activates services such as Exchange, the Event logs, System State (registry), and even SnapDrive for preparing the disks living on our NetApp filer for being backed up.

All this is fine and good, the backup proceeds as expected, but when it finishes, you can check the status of the VSS service using:

vssadmin list writers

from a command prompt and you end up seeing a bunch of services listed as

[5] Waiting for completion

This is not normal, but no errors were thrown at all from BackupExec. It ended nicely and reported that everything was fine. But, later in the evening, I have a second backup scheduled using SnapManager for Exchange from NetApp which provides quickly recoverable snapshots of our Exchange data stores. This software also leverages VSS to put Exchange in a consistent state but when it would try...

You guessed it, failure. The VSS writers would report back that they were busy and all hell would break loose.

Turns out the solution is stranger than the problem. If I run 3 backup jobs from BackupExec, each one backing up a different component: files, system state, and information stores, then this problem is gone. The VSS test after the backup shows:

[1] Stable

While not the most sensible solution, it was easy enough to create three selection lists and jobs.

The strangest part is that I can run all three separate jobs at the same time without a problem, but running the three pieces in the same job throws the errors.