BackupExec, VSS, and Exchange (Oh My!)

It turns out that my problem yesterday with BackupExec is a strange interaction between BackupExec 11d, Volume Shadow Services, and Exchange backups.

What normally happens is BackupExec calls out to the Volume Shadow Services on a Windows system and tells it to snapshot anything related to what it is about to back up. This allows the applications to put themselves into a consistent state in preparation for a backup. This is critical for databases like Exchange. In my case, this activates services such as Exchange, the Event logs, System State (registry), and even SnapDrive for preparing the disks living on our NetApp filer for being backed up.

All this is fine and good, the backup proceeds as expected, but when it finishes, you can check the status of the VSS service using:

vssadmin list writers

from a command prompt and you end up seeing a bunch of services listed as

[5] Waiting for completion

This is not normal, but no errors were thrown at all from BackupExec. It ended nicely and reported that everything was fine. But, later in the evening, I have a second backup scheduled using SnapManager for Exchange from NetApp which provides quickly recoverable snapshots of our Exchange data stores. This software also leverages VSS to put Exchange in a consistent state but when it would try...

You guessed it, failure. The VSS writers would report back that they were busy and all hell would break loose.

Turns out the solution is stranger than the problem. If I run 3 backup jobs from BackupExec, each one backing up a different component: files, system state, and information stores, then this problem is gone. The VSS test after the backup shows:

[1] Stable

While not the most sensible solution, it was easy enough to create three selection lists and jobs.

The strangest part is that I can run all three separate jobs at the same time without a problem, but running the three pieces in the same job throws the errors.