Tuesday, 6 November 2012

Raw Device Mapping (RDM) DeepDive


What is it?

RDM is mapping file used to map a LUN directly to VM bypassing VMFS layer. This LUN can be formatted using any File System (NTFS or FAT32) without the need to format it using VMFS and placing VMDK file on top of it.

The mapping file is located with the virtual machine directory. Here is an example of RDM mapping file for a VM called VM-01 which has pRDM LUN and vRDM LUN (will cover them later in details).

~ #  cd /vmfs/volumes/SAN-DATASTORE/VM-01

/vmfs/volumes/507fd84d-e9b583ac-b370-001b244d6cb2/VM-01 # ls -la
drwxr-xr-x    1 root     root          2800 Nov  3 12:18 .
drwxr-xr-t    1 root     root          2100 Nov  2 21:33 ..
-rw-r--r--    1 root     root            73 Oct 20 10:21 VM-01-e61f5754.hlog
-rw-------    1 root     root     2147483648 Nov  3 11:39 VM-01-e61f5754.vswp
-rw-------    1 root     root     42949672960 Nov  3 12:49 VM-01-flat.vmdk
-rw-------    1 root     root          8684 Nov  3 11:40 VM-01.nvram
-rw-------    1 root     root           541 Nov  3 11:38 VM-01.vmdk
-rw-r--r--    1 root     root             0 Oct 19 15:10 VM-01.vmsd
-rwxr-xr-x    1 root     root          3679 Nov  3 12:18 VM-01.vmx
-rw-------    1 root     root             0 Oct 20 19:04 VM-01.vmx.lck
-rw-r--r--    1 root     root           260 Nov  3 12:18 VM-01.vmxf
-rwxr-xr-x    1 root     root          3680 Nov  3 12:18 VM-01.vmx~
-rw-------    1 root     root     32212254720 Nov  3 12:18 VM-01_1-rdm.vmdk ---------------------> vRDM mapping file
-rw-------    1 root     root           482 Nov  3 12:18 VM-01_1.vmdk ----------------------> Descriptor File
-rw-------    1 root     root     32212254720 Nov  3 12:18 VM-01_2-rdmp.vmdk ---------------------> pRDM mapping file
-rw-------    1 root     root           494 Nov  3 12:18 VM-01_2.vmdk ----------------------> Descriptor File
-rw-r--r--    1 root     root        579252 Oct 20 10:21 vmware-1.log
-rw-r--r--    1 root     root        139522 Oct 20 19:03 vmware-2.log
-rw-r--r--    1 root     root        307364 Nov  3 12:19 vmware.log
-rw-------    1 root     root     112197632 Oct 20 19:04 vmx-VM-01-3860813652-1.vswp

Notice the difference between the descriptor file of a VMDK Hard drive and RDM LUN Hard Drive.
Important Note:  The size of the mapping file equals to the size of the LUN. Although RDM LUN is obtained directly from SAN, the same space of the LUN should be maintained in VMFS datastore due to mapping file.

How it works?

When a LUN is opened for access from inside the VM, the mapping file is read to obtain the reference to the raw LUN (because mapping file is having the metadata for the LUN). Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.
VM .vmx file is having reference pointing to RDM mapping file and not the LUN. For our example VM-01

scsi0:1.fileName = "VM-01_1.vmdk"
scsi0:1.mode = "persistent"
scsi0:1.ctkEnabled = "FALSE"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:1.present = "TRUE"
scsi0:1.redo = ""
scsi0:2.fileName = "VM-01_2.vmdk"
scsi0:2.mode = "independent-persistent"
scsi0:2.ctkEnabled = "FALSE"
scsi0:2.deviceType = "scsi-hardDisk"
scsi0:2.present = "TRUE"
scsi0:2.redo = ""

When to use?

Always VMFS is preferred over RDM. But there are some scenarios where RDM is required:

SAN-based features

- SAN snapshots: Using RDM enables backup software's to ‘see’ inside the LUN and to create a consistent snapshot of the data contained within the LUN. This allows for replication of data (snapshots) to a second SAN (on a different physical location) for Disaster Recovery. 
- In-band management software for your SAN running in Virtual Machines might require direct access to a management LUN. This usually requires physical compatibility mode.

Clustering

In any MSCS clustering scenario that spans physical hosts cluster data and quorum disks should be configured as RDMs rather than as virtual disks on a shared VMFS.  The reason is that cluster members need to have access to shared LUN (quorum).

What are the available types?

RDM has two compatibility modes:


1. Virtual Mode 
In this mode all SCSI commands from the VM to LUN will pass through VMFS layer except READ & WRITE. Those will be passed directly to the LUN.

If you check the device manager of the VM you will see the RDM LUN listed as VMware Virtual Disk SCSI Disk Device. This is exactly the same as VMDK Drive added to the VM. The physical characteristics of the LUN are hidden from the VM.

The advantage of vMode is that you can still use Distributed Locking Mechanism of VMFS as well as snapshots of RDM LUN drives.

2. Physical Mode
In this mode all SCSI commands are passed to RDM LUN directly from the VM expect REPORT command virtualized by VMkernal.

If you check the device manager of the VM you will see the RDM LUN listed based on the physical name/characteristics of the storage array.

The advantage of the full exposure of the LUN characteristics to the VM can be seen when using SAN Management Tools and MSCS services. Also, RDM LUNs in physical mode can have a size greater than 2TB.
 
Another important feature inherited with RDM is Dynamic Name Resolution. Each LUN is having unique ID in RDM mapping file used by VMFS. In case the path to the LUN modified due to failover, for example, the ID of the LUN will be change. Dynamic Name Resolution feature will automatically discover the new name and associate it in the RDM mapping file to resume operations.

VMFS vs. RDM

Here is a comparison between the types of disk drives available and the supported features.


RDM interoperatibility with vSphere Advanced Features
Both pRDM and vRDM support vSphere advnaced features such as HA, DRS, SDRS, VM, SVM. However there are some limitations for some features.

vMotion
- Files for a virtual machine are not relocated when it is vMotioned. 
- The virtual machine is re-registered to the destination host. 
- Any RDMs remain as RDMs when the virtual machine is registered to another host. That is, no changes to the virtual machine itself are made.


Cold Migration
Cold migration can be for resources, datastores, or both while the VM is powered-off. In case migrating datastore is selected, here are some considerations for RDM:

- Any non-RDM virtual disks are physically moved to the destination. 
- The virtual machine configuration files are physically moved to the destination. 
- Raw LUNs themselves cannot be moved, as they are raw disks presented from the SAN. However, the pointer files (RDMs) can be relocated if required. 
- During the migration you can use the advanced section of the migration wizard and select if you would like to maintain the same format of the files at the destination (remain as RDM) or convert it to a thick/thin disk. 
- The virtual machine registration changes, but the files are left untouched.


Cloning
When cloning a VM with RDMs, pRDM & vRDM contents will be copied into VMDK virtual disks. If you want to clone a virtual machine without its RDMs, remove them from the configuration of the virtual machine before cloning. You can delete the RDM from the disk when removing it (the raw LUN contents are not changed, only the RDM mapping file is deleted). Re-add them to the configuration when completed.

Storage vMotion
- When you perform Storage vMotion, the virtual machine files are physically relocated to a destination datastore. The same host retains ownership or registration of the virtual machine after Storage vMotion completes. 
- For vSphere 4.0 and higher: 
# Virtual disks and virtual mode RDM pointer files can be relocated to the destination datastore. 
# Virtual disks can be converted to thick-provisioned or thin-provisioned disks during migration as long as  the destination is not an NFS datastore. 
# Physical mode RDM pointer files can be relocated to the destination datastore, but physical mode RDM data cannot be migrated. 
# Virtual mode RDM data can be migrated with storage vMotion, provided sufficient licensing is present.

Notes:

- In ESX 5.0, during the Storage vMotion, virtual mode RDMs can be converted to thick/thin provisioned disks by selecting the appropriate options under the Advanced section of the migration wizard. Physical mode RDMs can only be converted with a cold migration. 
- If you are attempting a Storage vMotion of a virtual mode RDM using the Advanced method and the RDM pointer mapping file is already present in the target datastore, the Storage vMotion finishes quickly, but without moving the data. This is because the Storage vMotion detects that the source and target datastores for the mapping file are the same, and therefore concludes that no movement is needed.


To work around this issue: 
1. Remove the RDM from the virtual machine, and delete it from disk to ensure the mapping file is deleted. This will not delete the data in the RDM. 
2. Add the RDM again, and put the mapping file in a different datastore than the ultimate target of the storage vMotion. 
3. After the virtual machine reconfiguration is complete, attempt the storage vMotion again.

RDM Restrictions

1. RDM can't be used with NFS datastore. 
2. SVM can't be used with NPIV

RDM Tips

In case RDM is greyed out:

1. Make sure that you have a free LUN which support RDM 
2. Make sure that RDMFiler Advanced parameter is unchecked

Note: In case a LUN is used by ESXi host (even if partially), you can't use it as RDM to VM and vice versa. The reason is that OS or Hypervisor will apply lock into the LUN which will prevent it to be used by other system.

After adding the RDM to VM, you don't need to format it (assuming that it was used on previous system and was previously formatted). You need ONLY to bring it online using disk management. In case of fresh new RDM assigned to VM, it needs to be formatted for first use.

Note: RDM isn’t support with vCloud and should be avoided.

14 comments:

  1. Very nice post about "what is an RDM" and clear difference between vRDM and pRDM!

    ReplyDelete
  2. This is great, thank you.

    ReplyDelete
  3. Hi,
    Absolutely great post!
    Maybe its a good idea to also cover the .rdmp files that are left in the datastore after erasing a LUN.
    I know many people ask questions about this issue.

    ReplyDelete
  4. Will the block size of RDM be the same as the OS(4KB if NTFS) or will VMFS play a part and make it at 1MB?

    ReplyDelete
    Replies
    1. If your LUN was original formatted using NTFS (for example), the block size will remain 4KB when attached to VM as RDM. Even if you attach new LUN to windows VM as RDM and formatted the LUN from windows OS, the block size will be 4KB for NTFS formatting.

      In case of VMFS, the hypervisor will format the LUN to join a datastore. For RDM, the hypervisor won't see the LUN and will attach to directly to VM. It just uses RDM mapping file to point the VM to the LUN.

      Hope this clarified.

      Delete
    2. This comment has been removed by the author.

      Delete
  5. Hi MohammedI, have left doubts.

    1) pRDM vs VRDM

    With pRDM you can do vMotion -> Change host with VM on? (Hot vMotion)

    With VRDM you can do vMotion -> Change host with VM on? (Hot vMotion)

    2) Convert

    you can convert a disk pRDM in VRDM?


    Thanks for the info.

    ReplyDelete
  6. Hi
    nice post
    I have an issue with a VM, with 2 hdisks (one a classical vmdk and one in virtual RDM mode)?
    They are both seen as vmware virtual disk SCSI on the windows side

    When I try to migrate with GUI (right click and then migrate ), trying to migrate config and hdisk 1 to datastore 1 and hdsk 2 (the vRDM to datastore 2), I have the message :
    The requested Storage VMotion would move a virtual machine's disks without assigning the virtual machine a new home, but such a move is not supported on the Source host
    I have a Vcenter 4 and unfortunatly some old ESX hosts in 3.5..

    My goal is to move the RDM to a vmdk as it is said in the manuals ..

    Any idea ?

    regards

    Patrick

    ReplyDelete
  7. Hi Mohammed,

    We have been trying to attach a Samsung 840 SSD as an RDM for a Linux VM. The Raw Devices option is always disabled in vCenter during Add Hard Disk wizard, and attempts to manually create via CLI using vmfsktools -z or -r also fail.

    Any suggestions on how to troubleshoot?

    Thanks
    Rick

    ReplyDelete
  8. Hi Mohammed,
    I have deleted the VM from disk (Decomissioned) which had RDM mappings but missed to take a note of RDM details also missed to remove them from VM. Now I need those RDM details to share with storage person so that he can take that space from the SAN end & reassign those LUNs as RDM to cluster so that I can use them for another VM.
    Can you please let me know is there any way I can get RDM details of deleted VM?
    Your help would be much appreciated!
    Thanks in advance!

    ReplyDelete
  9. Great article!! This is very important information for us.I like all content and information.I have read it.You know more about this please visit again.

    VMware Training in Chennai

    ReplyDelete
  10. Good morning guys , Great Article .

    I am however concerned about the statement that says mapping file on VMFS is same size as backing RDM . I have seen the files in the past but it looks like a "fake" file . For example , are you saying if you have a 50TB RDM , you would need a VMFS volume that also has 50TB free . That sounds like a waste .

    Any comments welcome .


    Important Note: The size of the mapping file equals to the size of the LUN. Although RDM LUN is obtained directly from SAN, the same space of the LUN should be maintained in VMFS datastore due to mapping file.

    ReplyDelete
    Replies
    1. I did some checking . The pRDM.VMDK are indeed fake . The VMFS Volume containing the Symlink is only 1 TB , but it hosts a total of 6 TB of physical RDMs . So at least when dealing with Physical raw disks , in ESX55 , you do not require the same amount of Free Space on VMFS pointer data store .

      This VMFS store is 1 TB , see below up to 6TB Pointers >

      /vmfs/volumes/513dabd6-f5b30e38-fe6e-0025b50a008f/XXXX # ls -ltr
      total 0
      -rw------- 1 root root 2201282334720 Jun 5 2016 XXXX_5-rdmp.vmdk
      -rw------- 1 root root 2418939694080 Jun 5 2016 XXXX_6-rdmp.vmdk
      -rw------- 1 root root 118123246080 Jun 5 2016 XXXX_2-rdmp.vmdk
      -rw------- 1 root root 1649333145600 Jun 5 2016 XXXX_11-rdmp.vmdk
      -rw------- 1 root root 2287647774720 Jun 5 2016 XXXXL_1-rdmp.vmdk
      -rw------- 1 root root 499 Apr 7 18:53 XXXX_2.vmdk
      -rw------- 1 root root 501 Apr 7 18:54 XXXX_5.vmdk
      -rw------- 1 root root 501 Apr 7 19:35 0XXXX_1.vmdk
      -rw------- 1 root root 501 Apr 7 19:56 XXXX_6.vmdk

      Delete
  11. Thanks so much for this information. I have to let you know I concur on several of the points you make here and others may require some further review, but I can see your viewpoint.
    vmware training uk

    ReplyDelete