I’ve been meaning to dig into something technical for a long time as a more than welcome change in daily routine. I’ve been on the hunt for a new house (in Utrecht), have been busy with the VMware Enterprise Solution Provider partnership and did a complete rebuild of my internal Lab Manager environment. Also, I’m beginning my preparations for the VCDX Design Exam (which I’m going to take at the end of February). A couple of months ago, I visited the subject of RDM’s from a different angle (link), but now I wanted to give a more complete report on the matter.
What is a Raw Device Mapping?
The VMware VMFS filesystem supports a special type of file. This file, an RDM, is a mechanism for a virtual machine to have direct access to a LUN on the SAN. This LUN can be formatted in any way desired, without the need to format it to VMFS and place a VMDK on it. This removes two additional layers of complexity (the VMFS and the VMDK).
An RDM is a symbolic link from a VMFS volume to a raw LUN. The mapping makes LUNs appear as files in a VMFS volume. The mapping file, not the raw LUN, is referenced in the virtual machine configuration. The mapping file acts as a proxy for a raw physical device. This file contains metadata for managing and redirecting disk access.
When a LUN is opened for access, the mapping file is read to obtain the reference to the raw LUN. Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.
The file gives you some of the advantages of direct access to a physical device while keeping some advantages of a virtual disk in VMFS. As a result, it merges VMFS manageability with raw device access.
Although VMware recommends that you use VMFS datastores for most virtual disk storage, on certain occasions, you might need to use raw LUNs or logical disks located in a SAN.
Virtual mode for an RDM specifies full virtualization of the mapped device. It appears to the guest operating system exactly the same as a virtual disk file in a VMFS volume. The real hardware characteristics are hidden. Virtual mode enables you to use VMFS features such as advanced file locking and snapshots. Virtual mode is also more portable across storage hardware than physical mode, presenting the same behavior as a virtual disk file. When you clone the disk, make a template out of it, or migrate it (if the migration involves copying the disk), the contents of the LUN is into a virtual disk (.vmdk) file.
Physical Compatibility Mode
Physical mode for the RDM specifies minimal SCSI virtualization of the mapped device, allowing the greatest flexibility for SAN management software. In physical mode, the VMkernel passes all SCSI commands to the device, with one exception: the REPORT LUNs command is virtualized, so that the VMkernel can isolate the LUN for the owning virtual machine. Otherwise, all physical characteristics of the underlying hardware are exposed. Physical mode is useful to run SAN management agents or other SCSI target based software in the virtual machine. Physical mode also allows virtual-to-physical clustering for cost-effective high availability. LUNs attached to powered-on virtual machines and configured for physical compatibility cannot be migrated if the migration involves copying the disk. Such LUNs cannot be cloned or cloned to a template either.
Dynamic Name Resolution
Provides a user-friendly name for a mapped device. When you use an RDM, you do not need to refer to the device by its device name. You refer to it by the name of the mapping file, for example:
/vmfs/volumes/myVolume/myVMDirectory/myRawDisk.vmdk
This reference is permanent and will not change between reboots. VMFS uniquely identifies all mapped LUNs, and the identification is stored in its internal data structures. Any change in the SCSI path, such as a Fibre Channel switch failure or the addition of a new host bus adapter, can change the device name. Dynamic name resolution compensates for these changes by adjusting the data structures to retarget LUNs to their new device names.
User-friendly Persistent Names
Stores unique identification information for each mapped device. VMFS associates each RDM with its current SCSI device, regardless of changes in the physical configuration of the server because of adapter hardware changes, path changes, device relocation, and so on.
When do you use a RDM?
SAN-based features
If you are using SAN-based features, sometime you have no choice but to use a RDM. Examples are Dell/Equallogic’s VSS Writer for snapshot and backup purposes, NetApp’s SnapManager for Exchange/SQL and so forth. Using a RDM enables these kinds of software to ‘see’ inside the LUN and to create a consistent snapshot of the data contained within the LUN. This allows for replication of data (snapshots) to a second SAN (on a different physical location) for Disaster Recovery. These types of software do require communication with the application, thus usually require an agent inside the VM. With VSS, the ‘agent’ is installed by default by the Guest OS, as it is a Windows feature.
It might also be use to create backups of the data on those LUNs in a much more efficient way as compared to traditional backup agents or even VM-based snapshots (using VCB or the vStorage API and Data Recovery). With these techniques, say, your Exchange database is backed up using resources (i.e. the hardware) of your SAN and thus relieving the hypervisor of any backup or snapshot tasks. This is also known as off-host backups. Depending on your environment, you can either use a physical backupserver (and integration between the backupsoftware such as Symantec Backup Exec and the SAN such as Equallogic), a NDMP-enabled VTL or a second SAN.
Also, in-band management software for your SAN running in Virtual Machines might require direct access to a management LUN. This usually requires physical compatibility mode.
Clustering
A second reason why you would want to use RDMs is in the case of a clustering service, such as Microsoft Clustering Services, Heartbeat and Novell Clustering Services (based on Heartbeat nowadays). There are three types of clustering, each with its own requirements on the storage side, which I will explain below.
Big ol’ disks
The last case for RDMs is ‘big datadisks’, with which you feel uncomfortable having it all in a single VMDK. In the words of Duncan Epping, ‘large’ is a disk larger than 800GB. This however does not mean that any virtual disk larger than 800GB should be a RDM. You could, for manageability and other reasons, create a dedicated VMFS Datastore with just this VMDK. Choosing between either is a matter of requirements as mentioned above and confidence in the technical feasibility of RDMs.
Requirements and limitations
- Blockbased storage. Both iSCSI and FC are supported. No NFS. No local storage. You need a VMFS datastore to store the mapping file.
- No partition mapping – RDM requires the mapped device to be a whole LUN.
- You cannot create a snapshot of a Physical Compatibility Mode RDM.
- You cannot enable Fault Tolerance for a VM using a Physical Compatibility mode RDM.
- VCB cannot backup a RDM in Physical Compatibility mode.
- VMware Data Recovery cannot backup a RDM in Physical Compatibility mode.
- When using VMware vCenter Site Recovery Manager, make sure you include both the raw LUN as well as the VMFS datastore holding the mapping file to SAN replication schedules. Placing the mapping file alongside the VMX file is highly recommended.
- VMware vCenter Server does not support LUN number changes within the target. When a LUN number changes, the vml identifier does not change accordingly. Remove and rebuild the RDM to generate a new vml ID that the LUN recognizes as a workaround.
- The maximum size of an RDM is 2TB minus 512 bytes.
- Use the Microsoft iSCSI initiator and MPIO inside the VM when you need more than 2TB on a disk.
- You cannot use a RDM (in conjuncture with NPIV) to give a VM access to a VMFS volume. This seems to make no sense at all, unless you want to run your VCB proxy as a virtual machine to access Fibre Channel LUNs. For iSCSI, you can still use the software iSCSI Initiator in your Guest OS. Mental note: obviously, it isn’t supported at all.
Compatibility with…
Storage VMotion and cold migration
A RDM can be included when you perform a Storage VMotion or cold migration. When using Physical Compatibility mode, only the mapping will be migrated (choose ‘Same as Source’ in the Migration Wizard). When using Virtual Compatibility mode, the raw disk will be migrated to either a thin provisioned or thick VMDK disk. Physical Compatibility mode RDMs cannot be migrated to a VMDK file. RDM converted to VMDK virtual disks cannot be converted back. Please note that NFS is not supported as a destination for a RDM disk migration. Use an intermediate step (Storage VMotion to a VMFS volume, either local or on an iSCSI/FC SAN) as a workaround.
RDM’s are fully supported for normal VMotion operations.
N-Port ID Virtualization
Makes it possible to use the NPIV technology that allows a single Fibre Channel HBA port to register with the Fibre Channel fabric using several worldwide port names (WWPNs). This ability makes the HBA port appear as multiple virtual ports, each having its own ID and virtual port name. Virtual machines can then claim each of these virtual ports and use them for all RDM traffic.
NPIV is supported only for virtual machines with RDM disks. Virtual machines with regular virtual disks continue to use the WWNs of the host’s physical HBAs.
Clustering
There are a couple of different ways to build a cluster environment, each with its own requirements on the storage side:
Cluster-in-a-box
Most often used for test- and development use cases, this type of clustering has two or more VM’s running on the same physical box. You can use both kinds of RDMs, as well as VMDK’s. VMware recommends using VMDK virtual disks, unless you are using Microsoft Clustering, in which case you need to use a RDM. I would recommend Physical Compatibility mode.
Cluster-across-boxes
Instead of running all cluster nodes on a single physical box, you run the VM’s across multiple physical boxes. This requires either type RDM. VMware recommends Virtual Compatibility mode. In the case of Microsoft Clustering Services, use Physical Compatibility mode.
Physical-to-Virtual Cluster
Using both physical boxes (without hypervisor) and virtual boxes requires the use of Physical Compatibility mode RDMs.
For Microsoft Clustering Services, there are some extra caveats. For instance, using Windows Server 2008 in a Failover Cluster scenario, you cannot use Virtual Compatibility mode. For clustered Continuous Replication Environment for Microsoft Exchange: use Physical Compatibility mode. Also, iSCSI is still not supported. You’ll need to use Fibre Channel.
FAQ
If using a RDM, what is written to physical disk regarding information about the RDM?
Nothing, the LUN contains only information about the filesystem the Guest OS formatted it with. There is no additional data on the disk.
Can I use a local disk as a RDM?
No, local RDM’s are not supported. Using the vSphere Client, you cannot assign local storage as an RDM to a VM. There are ways to achieve local RDM’s, though, for instance, connecting a SCSI device using SCSI-passthrough.
Can I detach a RDM and attach the LUN to a physical server?
Just like using normal LUNs or even simple external USB hard drives: yes you can, but it is up to the OS to whom the LUN is attached if this will work without a hitch. For the usual filesystems like NTFS, EXT3, ZFS and NSS, this is no problem at all.
How is a RDM presented to the Guest OS in a VM?
The Guest OS sees a local SCSI hard drive, just as it would when using a VMDK or Guest OS initiated iSCSI connected disk. Depending on the chosen compatibility, the Guest OS would see a ‘VMware Virtual SCSI Disk Device’ or a disk named by the storage array (it would be likely to have the vendor’s name somewhere in there).
Can I use RDM-like features on other products than vSphere?
Sure you can. VMware Workstation can give access to a complete LUN (or physical disk) or partitioned parts of a disk.
Sources for this blogpost
- Introduction to VMware vSphere
- vSphere Basic System Administration
- ESX Configuration Guide
- vSphere Availability Guide
- Fibre Channel SAN Configuration Guide
- VMware Data Recovery 1.1 Administration Guide
- VMware vCenter SRM 4 Release Notes
- VMware vCenter Server 4.0 Update 1 Release Notes
- Configuration Maximums for VMware vSphere 4.0
- Virtual Machine Backup Guide
- Setup for Failover Clustering and Microsoft Cluster Service
- 2vcps and a Truck: Using iSCSI to get some big ole disk in a Virtual Machine
- The VMguy: vSphere and MSCS
- Virtual Black Hole: To Use RDM’s or Not To Use RDM’s
- boche.net: Setup for Microsoft cluster service
- Leo’s Ramblings: Deciding between iSCSI RDMs and the Microsoft’s iSCSI initiator
FYI, concerning cold migration…there is a known issue in vSphere where migrating physical mode RDMs creates thick provisioned VMDKs as you’d expect with virtual mode RDMs. I experienced this issue just the other day. Very frustrating!
Andy: wow, that is weird. Is this bug still alive in Update1?
I was able to attach a 10 TB RDM to a VM Guest so far no problems….we actually created it in a seperate datacenter directly zoned to a physical box loaded it up with data and moved the box to datacenter 2 across town and reattaced it to the vm as RDM and everything was there no problems!
but when i tried to attach a 20 TB RDM it throws an error , im still tryoingm to work around it but any advice is well recieved!!!!
Matt
Andy: wow, that is weird. Is this bug still alive in Update1?
Yes, it still happens.
In Physical Compatibility mode, with a Windows 2003 Cluster, the RAW LUNs presented to the ESX are configured with VMWARE Host Type or W2003 Host Type?
I have constant fail overs due to heavy IO, and the RAW LUNs are in the same HostGroup as the rest of the LUNs, so, they are all configured as “VMWARE” LUNs.
The fact is, those RAW LUNs are accessed exclusively by the MSCS, so I understand that they MUST be configured as “Windows 2003 Cluster” LUNs. Is this correct? Wich HostType is correct for a RAW LUN in Physical Mode?
Thanks in advance.-
Andi: If you use a physical compatibility mode RDM, make sure to configure the LUN with settings for the Guest OS, i.e. Windows 2003. Don’t set these LUNs to any VMware-format, as VMware does not interact with these LUNs. For any LUNs that contain the VMFS filesystem, configure as VMware LUNs. For Virtual Compatibility mode, set according to your SAN vendor’s recommendation.
Joep, thanks for your reply. I’m trying to understand how it works. That’s what I thought and told to my client in the first place… but is there any official document that states this?
I believe this is stated in either the iSCSI or the FC SAN configuration guide.
Matt,
Can you (or someone else) explain how you attached a 10TB RDM to vSphere, when the stated maximum is 2TB?
Is this possible but unsupported?
Mika
Hello,
I’m currently on the process of redesigning my MSA1000 that connected to 4 vSphere hosts and i have MSCS SQL 2000 and Exchange 2007.
I’m migrating the entire VMs to a new iSCSI SAN based on StarWind as a Temporary till I swapped everything in the MSA1000 and reconfigure it to host my VMs.
I’m wondering about my SQL and Exchange which using RDM LUNs, could you please give me some tips and tricks to migrate my RDM to the new iSCSI Storage, recreate them on the MSA1000 and re-migrate the VMs back to the MSA1000.
Thanks once again,
Habibalby
How many RDM per virtual machine i can assign??
i have vsphere 4 update 1, i try assign 3 RDM but i cant.
do you know???
Very nice article. First one I’ve found which catagorically states partitions cannot be RDMd. That explains why I can’t find any way to do so ;-(. I want to use half an SSD for ESXi and pass half of it to a client VM for use as ZFS cache.
I thought I’d just pass the whole LUN and be smart enough not to touch the ESXi partition from the client, but a lock ESXi claims prevents that.
As you’re now my acknowledged expert, could you offer any suggestions to work around this?
Thank you!
Hi Jonathan,
There simply isn’t a way to use a partition as an RDM. RDM’s just work with LUNs (or disks). Also, local disks aren’t supported.
Hi Jonathan,
There simply isn’t a way to use a partition as an RDM. RDM’s just work with LUNs (or disks). Also, local disks aren’t supported.
Excellent article, however “I would recommend Physical Compatibility mode.” for cluster in a box? VMware seems to say virtual RDM or vmdk only for cluster in a box, in the MSCS vsphere 4.1 white paper.
Hello,
I am studying for the VCP 410 and I came across your blog. This is an excellent post! It really helped me understand how RDM works. Thank you so much.
Great post on the RDMs, however one question I have is how they are treated by an ESX farm in relation to the maximum number of LUNs.
We have a 20 server farm (just one of our farms). Our service service provider is telling us that to enable a guest to vMotion the RDM has to be presented to all hosts in the farm, effectively making it 20 LUNs for the purposes of the number of LUNs we can present.
Is this correct or does the RDM itself only need to be mapped once and it just moves with the guest as vMotion moves the guest ?
Hi Jason,
The RDM is presented to the host first, and ‘routed’ to the guest VM. This means that you do need to present the RDM to each and every host in your cluster in order for vMotion to actually work.
If you are in any way limited to the number of LUNs you can present to the physical hosts, you could consider using in-guest iSCSI (using the software Microsoft iSCSI Initiator from within the Guest OS inside the VM, for instance). This way, you still have vMotion compatibility without having to present the LUN to all physical hosts.
Very nice and detailed explanation . thanks a lot!!