I’ve been meaning to dig into something technical for a long time as a more than welcome change in daily routine. I’ve been on the hunt for a new house (in Utrecht), have been busy with the VMware Enterprise Solution Provider partnership and did a complete rebuild of my internal Lab Manager environment. Also, I’m beginning my preparations for the VCDX Design Exam (which I’m going to take at the end of February). A couple of months ago, I visited the subject of RDM’s from a different angle (link), but now I wanted to give a more complete report on the matter.

What is a Raw Device Mapping?

The VMware VMFS filesystem supports a special type of file. This file, an RDM, is a mechanism for a virtual machine to have direct access to a LUN on the SAN. This LUN can be formatted in any way desired, without the need to format it to VMFS and place a VMDK on it. This removes two additional layers of complexity (the VMFS and the VMDK).

An RDM is a symbolic link from a VMFS volume to a raw LUN. The mapping makes LUNs appear as files in a VMFS volume. The mapping file, not the raw LUN, is referenced in the virtual machine configuration. The mapping file acts as a proxy for a raw physical device. This file contains metadata for managing and redirecting disk access.

When a LUN is opened for access, the mapping file is read to obtain the reference to the raw LUN. Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.

The file gives you some of the advantages of direct access to a physical device while keeping some advantages of a virtual disk in VMFS. As a result, it merges VMFS manageability with raw device access.

Although VMware recommends that you use VMFS datastores for most virtual disk storage, on certain occasions, you might need to use raw LUNs or logical disks located in a SAN.


Virtual Compatibility Mode

Virtual mode for an RDM specifies full virtualization of the mapped device. It appears to the guest operating system exactly the same as a virtual disk file in a VMFS volume. The real hardware characteristics are hidden. Virtual mode enables you to use VMFS features such as advanced file locking and snapshots. Virtual mode is also more portable across storage hardware than physical mode, presenting the same behavior as a virtual disk file. When you clone the disk, make a template out of it, or migrate it (if the migration involves copying the disk), the contents of the LUN is into a virtual disk (.vmdk) file.

Physical Compatibility Mode

Physical mode for the RDM specifies minimal SCSI virtualization of the mapped device, allowing the greatest flexibility for SAN management software. In physical mode, the VMkernel passes all SCSI commands to the device, with one exception: the REPORT LUNs command is virtualized, so that the VMkernel can isolate the LUN for the owning virtual machine. Otherwise, all physical characteristics of the underlying hardware are exposed. Physical mode is useful to run SAN management agents or other SCSI target based software in the virtual machine. Physical mode also allows virtual-to-physical clustering for cost-effective high availability. LUNs attached to powered-on virtual machines and configured for physical compatibility cannot be migrated if the migration involves copying the disk. Such LUNs cannot be cloned or cloned to a template either.

Dynamic Name Resolution

Provides a user-friendly name for a mapped device. When you use an RDM, you do not need to refer to the device by its device name. You refer to it by the name of the mapping file, for example:

/vmfs/volumes/myVolume/myVMDirectory/myRawDisk.vmdk

This reference is permanent and will not change between reboots. VMFS uniquely identifies all mapped LUNs, and the identification is stored in its internal data structures. Any change in the SCSI path, such as a Fibre Channel switch failure or the addition of a new host bus adapter, can change the device name. Dynamic name resolution compensates for these changes by adjusting the data structures to retarget LUNs to their new device names.

User-friendly Persistent Names

Stores unique identification information for each mapped device. VMFS associates each RDM with its current SCSI device, regardless of changes in the physical configuration of the server because of adapter hardware changes, path changes, device relocation, and so on.

When do you use a RDM?

SAN-based features

If you are using SAN-based features, sometime you have no choice but to use a RDM. Examples are Dell/Equallogic’s VSS Writer for snapshot and backup purposes, NetApp’s SnapManager for Exchange/SQL and so forth. Using a RDM enables these kinds of software to ‘see’ inside the LUN and to create a consistent snapshot of the data contained within the LUN. This allows for replication of data (snapshots) to a second SAN (on a different physical location) for Disaster Recovery. These types of software do require communication with the application, thus usually require an agent inside the VM. With VSS, the ‘agent’ is installed by default by the Guest OS, as it is a Windows feature.
It might also be use to create backups of the data on those LUNs in a much more efficient way as compared to traditional backup agents or even VM-based snapshots (using VCB or the vStorage API and Data Recovery). With these techniques, say, your Exchange database is backed up using resources (i.e. the hardware) of your SAN and thus relieving the hypervisor of any backup or snapshot tasks. This is also known as off-host backups. Depending on your environment, you can either use a physical backupserver (and integration between the backupsoftware such as Symantec Backup Exec and the SAN such as Equallogic), a NDMP-enabled VTL or a second SAN.
Also, in-band management software for your SAN running in Virtual Machines might require direct access to a management LUN. This usually requires physical compatibility mode.

Clustering

A second reason why you would want to use RDMs is in the case of a clustering service, such as Microsoft Clustering Services, Heartbeat and Novell Clustering Services (based on Heartbeat nowadays). There are three types of clustering, each with its own requirements on the storage side, which I will explain below.

Big ol’ disks

The last case for RDMs is ‘big datadisks’, with which you feel uncomfortable having it all in a single VMDK. In the words of Duncan Epping, ‘large’ is a disk larger than 800GB. This however does not mean that any virtual disk larger than 800GB should be a RDM. You could, for manageability and other reasons, create a dedicated VMFS Datastore with just this VMDK. Choosing between either is a matter of requirements as mentioned above and confidence in the technical feasibility of RDMs.

Requirements and limitations

  • Blockbased storage. Both iSCSI and FC are supported. No NFS. No local storage. You need a VMFS datastore to store the mapping file.
  • No partition mapping – RDM requires the mapped device to be a whole LUN.
  • You cannot create a snapshot of a Physical Compatibility Mode RDM.
  • You cannot enable Fault Tolerance for a VM using a Physical Compatibility mode RDM.
  • VCB cannot backup a RDM in Physical Compatibility mode.
  • VMware Data Recovery cannot backup a RDM in Physical Compatibility mode.
  • When using VMware vCenter Site Recovery Manager, make sure you include both the raw LUN as well as the VMFS datastore holding the mapping file to SAN replication schedules. Placing the mapping file alongside the VMX file is highly recommended.
  • VMware vCenter Server does not support LUN number changes within the target. When a LUN number changes, the vml identifier does not change accordingly. Remove and rebuild the RDM to generate a new vml ID that the LUN recognizes as a workaround.
  • The maximum size of an RDM is 2TB minus 512 bytes.
  • Use the Microsoft iSCSI initiator and MPIO inside the VM when you need more than 2TB on a disk.
  • You cannot use a RDM (in conjuncture with NPIV) to give a VM access to a VMFS volume. This seems to make no sense at all, unless you want to run your VCB proxy as a virtual machine to access Fibre Channel LUNs. For iSCSI, you can still use the software iSCSI Initiator in your Guest OS. Mental note: obviously, it isn’t supported at all.

Compatibility with…

Storage VMotion and cold migration

A RDM can be included when you perform a Storage VMotion or cold migration. When using Physical Compatibility mode, only the mapping will be migrated (choose ‘Same as Source’ in the Migration Wizard). When using Virtual Compatibility mode, the raw disk will be migrated to either a thin provisioned or thick VMDK disk. Physical Compatibility mode RDMs cannot be migrated to a VMDK file. RDM converted to VMDK virtual disks cannot be converted back. Please note that NFS is not supported as a destination for a RDM disk migration. Use an intermediate step (Storage VMotion to a VMFS volume, either local or on an iSCSI/FC SAN) as a workaround.

RDM’s are fully supported for normal VMotion operations.

N-Port ID Virtualization

Makes it possible to use the NPIV technology that allows a single Fibre Channel HBA port to register with the Fibre Channel fabric using several worldwide port names (WWPNs). This ability makes the HBA port appear as multiple virtual ports, each having its own ID and virtual port name. Virtual machines can then claim each of these virtual ports and use them for all RDM traffic.
NPIV is supported only for virtual machines with RDM disks. Virtual machines with regular virtual disks continue to use the WWNs of the host’s physical HBAs.

Clustering

There are a couple of different ways to build a cluster environment, each with its own requirements on the storage side:

Cluster-in-a-box

Most often used for test- and development use cases, this type of clustering has two or more VM’s running on the same physical box. You can use both kinds of RDMs, as well as VMDK’s. VMware recommends using VMDK virtual disks, unless you are using Microsoft Clustering, in which case you need to use a RDM. I would recommend Physical Compatibility mode.

Cluster-across-boxes

Instead of running all cluster nodes on a single physical box, you run the VM’s across multiple physical boxes. This requires either type RDM. VMware recommends Virtual Compatibility mode. In the case of Microsoft Clustering Services, use Physical Compatibility mode.

Physical-to-Virtual Cluster

Using both physical boxes (without hypervisor) and virtual boxes requires the use of Physical Compatibility mode RDMs.

For Microsoft Clustering Services, there are some extra caveats. For instance, using Windows Server 2008 in a Failover Cluster scenario, you cannot use Virtual Compatibility mode. For clustered Continuous Replication Environment for Microsoft Exchange: use Physical Compatibility mode. Also, iSCSI is still not supported. You’ll need to use Fibre Channel.

FAQ

If using a RDM, what is written to physical disk regarding information about the RDM?
Nothing, the LUN contains only information about the filesystem the Guest OS formatted it with. There is no additional data on the disk.
Can I use a local disk as a RDM?
No, local RDM’s are not supported. Using the vSphere Client, you cannot assign local storage as an RDM to a VM. There are ways to achieve local RDM’s, though, for instance, connecting a SCSI device using SCSI-passthrough.
Can I detach a RDM and attach the LUN to a physical server?
Just like using normal LUNs or even simple external USB hard drives: yes you can, but it is up to the OS to whom the LUN is attached if this will work without a hitch. For the usual filesystems like NTFS, EXT3, ZFS and NSS, this is no problem at all.
How is a RDM presented to the Guest OS in a VM?
The Guest OS sees a local SCSI hard drive, just as it would when using a VMDK or Guest OS initiated iSCSI connected disk. Depending on the chosen compatibility, the Guest OS would see a ‘VMware Virtual SCSI Disk Device’ or a disk named by the storage array (it would be likely to have the vendor’s name somewhere in there).
Can I use RDM-like features on other products than vSphere?
Sure you can. VMware Workstation can give access to a complete LUN (or physical disk) or partitioned parts of a disk.

Sources for this blogpost