A while back, I investigated how to optimize backups in an EqualLogic storage environment. I wanted to make use of all the (EqualLogic) features available to me.

I created a list of ‘must haves’ for this design:

  • Zero-cost, zero-impact on production environment while creating backups
  • Maintain data consistency and integrity
  • Be able to replicate backups to a secondary location (prevent using tapes as a off-site backup and archival tool)
  • Separation of back-up creation and processing: production environment should have a very short back-up time frame, all the processing and logic happens ‘behind the curtains’
  • Keep the backup server ‘light’ by using the SAN as a backup-to-disk target.

Utilizing SAN snapshots

I wanted to see if I could use the snapshot technologies available to me to fulfill the zero-cost, zero-impact item on the list. A snapshot is made without impact or cost, is fast to complete and is a static copy of the production data for my backup server to work with. On the downside, a snapshot isn’t application-consistent out of the box.

We need some kind of interaction between the array and the application to make the snapshot (and the data contained within) completely consistent. Luckily, the EqualLogic array has some cool Microsoft VSS integration. The device acts as a hardware VSS Provider. If you’re using applications that are VSS-aware (NTFS, Exchange, SQL all include a VSS Writer), you can leverage this integration. Just install the EqualLogic Host Integration Toolkit (Microsoft Edition) or HIT/ME for short, which includes the Auto-Snapshot Manager on the application server. The ASM/ME acts as a VSS Requestor and creates these snapshots (while maintaining data consistency using the various VSS components). Information about the snapshot is stored in XML and transferred from the array to the backup server using the ‘VSS-Control’ volume. The backup server picks up on this and mounts the volume based on the information inside the XML. This snapshot is then used as a source for backups rather than the original volume.

There is, how could there not be, a caveat. The major version of Windows should be consistent across backup- and application servers. Nowadays, this means Windows Server 2008 or 2008 R2. Please don’t mix 2003 and 2008 versions. Don’t blame EqualLogic for this, it’s a limitation due to Microsoft changing the XML format in Windows Server 2008, lacking backward compatibility.

Off-host backups

Next, I wanted to alleviate the application server. Although it now has a fast way to create a static copy of the application data (contained in the snapshot), the server still is actively involved in mounting the snapshot and moving data from the snapshot to the backup media or backup server. I want the back-up server to coördinate and execute all backup related tasks (including mounting the snapshot and copying the data). The only thing left to do for the application server is to prepare for the snapshot. The backup server then mounts the snapshot locally and copies the data to the backup target. When you’re using 3rd party backup software, this software will act as the VSS Requestor. The HIT/ME will do the VSS Provider-related tasks (in conjunction with the array hardware).

This requires off-host capabilities on the backup server, meaning that the backup software supports the VSS hardware provider included in the EqualLogic array and enables VSS transportable snapshots. Symantec Backup Exec, CommVault Galaxy, CA BrightStor ARCserver and Legato NetWorker all support this. For the off-host functionality to actually do its work, you’ll need to make sure that the SAN-snapshot is visible on the backup server. This usually means the backup server is connected to the iSCSI network. Also make sure both the application and backup servers can access the VSS control volume on the array. This is done while installing the HIT/ME. Create persistent, snapshot-only connections to the volumes to be backed up for the backup server (use a ACL entry for the volume inside the EqualLogic management tool for this). For the record: giving the backup server access to the original volume as well as the snapshot could lead to data corruption. Be warned.

An example

For example, Symantec Backup Exec has an option called ‘Advanced Disk-based Backup Option’ or ADBO to enable off-host backups. You won’t need the ‘Advanced Open File Option’ if using the ADBO. If you want to backup Exchange or SQL using VSS and the ADBO, you still need the agents for Exchange and SQL, but they need to be installed on the backup server (along with the Remote Agent for Windows and the EqualLogic HIT/ME). They’re not needed on the application servers themselves. You should however install the Backup Exec Remote Agent for Windows and the EqualLogic HIT/ME on the application servers.

To create transportable snapshots, specify that you want to use the ADBO  by selecting ‘Use offhost backup to move backup processing from the remote computer to media server’ and select the ‘Hardware – Use technology provided by hardware manufacturer’ snapshot provider. Be sure that you deselect ‘Use Advanced Open File Option’.

Full Granularity

Cool thing is, you can backup an entire volume, Exchange Information Store (or Storage Group) at the database level or SQL at the instance or database level, while still being able to do granular restores. Just enable ‘Use Backup Exec Granular Recovery Technology’ for each backup job.

Off-site backups

By leveraging the data replication features of the EqualLogic array, we can replicate the backup media target volume on the array to a secondary array in a different location. We can extend this functionality by adding a backup media server on the secondary location to do backup-to-tape operations. These two methods of adding resiliency to your backup strategy are configured using standard recommendations from both EqualLogic (for replication) and Symantec (for backup-to-tape). For simplicity sake, I won’t explain these in this post.

VMware vSphere

Things get a bit complicated if you’re doing all this with virtualized application servers. One of the requirements for the HIT/ME to function is that the volumes are mounted through the Microsoft iSCSI Software Initiator. This means that data volumes you’re backing up aren’t VMDK’s or RDM’s, but iSCSI volumes straight from the EqualLogic array. This might and probably will complicate your storage and virtualization platform designs, so you’ll have to make the trade-off between ‘simple’ and ‘awesome features’.

If you decided that off-host backups are worth the extra complexity, you might think: “Hey, I’m now using the ASM/ME, I could use the ASM/VE, too!“. ASM/VE is the Auto-Snapshot Manager for VMware vSphere, obviously. It creates virtual machine snapshots for all virtual machines on a given iSCSI volume before snapshotting the volume itself. Unfortunately, there’s not much you can do with those snapshots-in-snapshots from a backup perspective. If you’re after off-host backups for individual virtual machines, just use the VMware vStorage APIs for Data Protection plugin for your backup software, like the Backup Exec Agent for VMware Virtual Infrastructures and use ‘SAN transport mode’. Through the vADP and the VMware Tools, the virtual machine itself is somewhat VSS-aware (and thus consistent), too. Again, install the Remote Windows Agents for Granular Recovery stuff.

If you are using the ASM/VE, making a Smart Copy presents you with an option to “include PS series volumes accessed by guest iSCSI initiator.” I’ll quote Pete Koehler for his opinion on this matter: “I do not use this option for a few very good reasons, and rely completely on ASM/ME for properly capturing guest attached volumes”. Thanks Pete, couldn’t agree more.

Multipath I/O for performance

If you’re desperate for more performance, add MPIO. Your application servers (connected to the EqualLogic through the Microsoft iSCSI Software Initiator) can use MPIO. The backup server is no different. Use the specific DSM for MPIO from EqualLogic (installed alongside with the HIT/ME). Even if you use the VMware vADP to backup virtual machines, you can use MPIO. Beat that, HP LeftHand P4000.