Backup Exec vs. the NDMP agent

Once again I find myself slapping my forehead over the behavior of Backup Exec. In this episode, we find our intrepid administrator transitioning his NetApp Filer backups from traditional backups of CIFS shares to NDMP backup to a tape library attached directly to the filer. All appears well as the tape library is registered and shows up properly in the Backup Exec devices screen.

Flash forward to the weekend. All backups of the CIFS shares using non-NDMP methods fail. No NDMP backups have been configured. What could be the problem?

The problem appears to be that the Backup Exec agent and the NDMP agent share a port. Probably something within the bowels of the Backup Exec agent that it really is the bastard spawn of NDMP. However, what this means is that once you turn on NDMP on your NetApp filer, you can no longer do normal backups of CIFS shares from that filer!

It's just incredible to me that time and time again, I come across absolutely insane things that Backup Exec does. Why in the world would turning on NDMP on a filer *have* to require that Backup Exec now refuses to talk to the filer except via NDMP. Obviously this is some bug (one can hope) but upgrading to 12.5 from 11d isn't on my roadmap right now as that opens an entirely new can of worms.

Well, NDMP is now off until my new library license arrives along with my new stock of LTO4 tapes. Apparently it's all or nothing for NDMP and the filer.

BackupExec, VSS, and Exchange (Oh My!)

It turns out that my problem yesterday with BackupExec is a strange interaction between BackupExec 11d, Volume Shadow Services, and Exchange backups.

What normally happens is BackupExec calls out to the Volume Shadow Services on a Windows system and tells it to snapshot anything related to what it is about to back up. This allows the applications to put themselves into a consistent state in preparation for a backup. This is critical for databases like Exchange. In my case, this activates services such as Exchange, the Event logs, System State (registry), and even SnapDrive for preparing the disks living on our NetApp filer for being backed up.

All this is fine and good, the backup proceeds as expected, but when it finishes, you can check the status of the VSS service using:

vssadmin list writers

from a command prompt and you end up seeing a bunch of services listed as

[5] Waiting for completion

This is not normal, but no errors were thrown at all from BackupExec. It ended nicely and reported that everything was fine. But, later in the evening, I have a second backup scheduled using SnapManager for Exchange from NetApp which provides quickly recoverable snapshots of our Exchange data stores. This software also leverages VSS to put Exchange in a consistent state but when it would try...

You guessed it, failure. The VSS writers would report back that they were busy and all hell would break loose.

Turns out the solution is stranger than the problem. If I run 3 backup jobs from BackupExec, each one backing up a different component: files, system state, and information stores, then this problem is gone. The VSS test after the backup shows:

[1] Stable

While not the most sensible solution, it was easy enough to create three selection lists and jobs.

The strangest part is that I can run all three separate jobs at the same time without a problem, but running the three pieces in the same job throws the errors.