Boy-oh-boy-oh-boy, this is the best day since I know what VCB is! Finally, it’s being retired!

Dear Valued Customer,

The purpose of this letter is to inform you of our vSphere backup product strategy, ongoing enhancements, and end of availability plans for VMware Consolidated Backup.

VMware Backup Product Strategy
VMware released vStorage APIs for Data Protection (VADP) with the vSphere 4.0 release in May, 2009. VADP is the next generation of VMware’s backup framework. We have also been working with several backup partners to integrate VADP into their solutions to make backup of vSphere Virtual Machines fast, efficient and easy to deploy compared to VCB and other backup solutions. Several of our major backup partners have already released VADP integrated backup products and we expect most of the major backup partners to have VADP integrated backup software by the upcoming feature release of the vSphere platform in 2010.

Future Product Licensing
Given the strong interest and adoption of VADP by our backup eco-system and the benefits offered by VADP compared to VMware Consolidated Backup (VCB), we are announcing the End of Availability for VCB starting with next vSphere feature release in 2010. Starting with the next vSphere platform feature release, VCB will be removed from vSphere platform. VADP integrated backup products (including VMware Data Recovery) will be the recommended option for efficient backup and restoration of vSphere Virtual Machines. This will allow us to focus new value added feature development on VADP instead of two backup frameworks (VCB and VADP). You can find more information about the use of vStorage APIs for Data Protection in our Developer Community. For information on the availability of VADP integrated release of your backup product please contact your backup vendor.

End of Availability
With the release of the next vSphere platform, we will continue to provide the binaries for VCB, but they will not be compatible with the next platform release. We will continue to provide support for VCB on the current vSphere platform per the VMware support policy.

If you need assistance in the migration from VMware Consolidated Backup to the vStorage APIs for Data Protection, please contact your local reseller or storage backup vendor.

Best regards,
VMware Product Management

I’ve been preparing for the Design exam portion of VCDX for the last couple of days. Tomorrow is D-day for me! Meanwhile, some tweeps asked my what materials I used to prepare. Well, here you go:

Obviously, use the VMware VCDX Community Documents and the VCDX Design Exam Blueprint. I’ve read ‘Mastering vSphere 4′ by Scott Lowe, which is a nice addition on some of the details on vSphere. Only read if you’ve got the time, it weighs in at a hefty 700 pages.

The absolutely most valuable resources for me were:

• VMware Technical Paper - Business Continuity and Disaster Recovery
  • I highly recommend this VMbook!
• VMware Infrastructure 3 in a Cisco Network Environment
• iSCSI Design Considerations and Deployment Guide
• VMware VI3 Security Hardening
• Chad Sakacc - A “Multivendor Post” on using iSCSI with VMware vSphere
• Duncan Epping - HA Deepdive

Also, these were nice to read:

• Duncan Epping - Impact of decisions…
• Frank Denneman - Impact of Host local VM Swap on HA and DRS

The usual linkage on the blogs show some other useful documents, too:

• Scott Lowe - VCDX Design Exam Post-Mortem
• Jason Boche - VCDX Design Exam: been there, done that!
• Rick Scherer - VCDX Design Exam – Complete
• John Arrasjid - VCDX Tips from VCDX 001 John Arrasjid

As I’m a bit rusty on iSCSI, I’ve read up on those as well. Remember that the VCDX Design exam is on VI3.5, not vSphere, so these might not apply!

• iSCSI Design Considerations and Deployment Guide
• NetApp and VMware vSphere Storage Best Practices
• VMware vSphere and ESX 3.5 Multiprotocol Performance Comparison Using FC, iSCSI, and NFS
• Best Practices for File System Alignment in Virtual Environments

Specifically, I was also interested in the way Dell/EQL handles iSCSI

• Dell VMware vSphere Reference Architecture for SMB
• Dell EQL – Configuring VMware vSphere swiSCSI with PS-Series storage
• Jon Owings / 2vcps and a Truck – iSCSI Connections on EqualLogic PS Series

Digging deeper into templates and linked clones in VMware Workstation, we can create templates of ESX and ESXi. Using the previous post as a basis for the ESX template, it’s relatively simple:

Creating the template

When creating a virtual machine for ESXi, be sure to adhere to the 2 vCPU default value and the 2GB memory, as these are the minimum requirements for ESXi. There are, however, ways to reduce the minimum amount of RAM (see VCDX 007’s post). From experience, I know that selecting 1 vCPU also works. I’ve selected the LSI Logic SAS controller, and added three more Intel E1000 NICs to the configuration. Also, as I’m restricted in the amount of disk space, I’ve used a 2GB disk to install ESXi on.

One other thing to note, when you’re using the ‘hack’ to allow ESXi to boot with less than 2GB of memory is a chance you can’t add ESXi to vCenter.

[Show as slideshow]

 

Install ESXi

Just follow the default [F11] [Enter] installation procedure. A side note: ESXi does allow installation on a small 2GB drive:

Power off the VM after installation. Make sure that VM does not reboot and loads ESXi. This way, you are guaranteed a fresh, unique instance of ESXi. This prevents possible problems with generated SSL and SSH keys, among other things.

Make a snapshot of the VM

[Show as slideshow]

 

Deploying a virtual machine based on a template

[Show as slideshow]

 

After cloning and booting, ESXi is loaded with default settings. You can now assign a password, unique hostname and IP address settings. Make sure to reboot once after configuration.

In my earlier post, I described how I use my laptop as a mobile server. One of the optimizations I’ve applied is the use of templates and linked clones within VMware Workstation 7. Using templates and linked clones gives you a couple of advantages:

• Saves you time: no need to install the Guest OS every time you need a clean virtual machine.
• Saves you disk space: the template takes up the normal amount of disk space, as it contains the complete Guest OS. The linked clone only takes up a relatively small amount of disk space, as it contains only the changed data.

Creating a Virtual Machine

Create a virtual machine to your liking. I’ve used the following values:

[Show as slideshow]

 

Installing a guest OS in the Virtual Machine

Install a Guest OS like you want it. I’ve installed Microsoft Windows Server 2003 R2 Enterprise Edition SP2 in English. Make sure to install the VMware Tools. If you are planning to use this virtual machine in an Active Directory domain, you need to have a way to change the SID of the cloned machine, as VMware Workstation, unlike vCenter Server doesn’t do that for you. I’ve copied ‘newsid.exe’ to the desktop for this purpose.

Marking the VM as a template

Enabling Template Mode is merely a protection against accidental deletion of the parent VM (i.e. the template).

Make a snapshot of the VM

[Show as slideshow]

 

Deploying a virtual machine based on a template

[Show as slideshow]

 

My home lab isn’t actually some server that I’ve got lying around. As I’m on the road a lot, I need to have my VMs with me at all times. That’s why I came up with the ‘bag lab’. A laptop mobile enough to carry around with me on a day-to-day basis, but powerful enough to provide me with the resources to build a complete VMware vCenter Site Recovery Manager setup (which requires 5x Windows 2003, 2x ESX, 2x NetApp Sim) on the laptop.

As I explained earlier, the machine I’ve chosen is a Dell Latitude E6500. While maxed out specifications-wise, I’ve even added some extra tricks, like the SSD and the tray to accommodate the second hard drive. I’ve used this laptop for a few weeks now, and I’m thoroughly impressed. I’ll try to explain how I’ve organized the OS, applications and various types of data to my liking, as well as experiences with the weight & size while on the road.

Software layout

The laptop has two storage devices. A 80GB SSD (SSDSA2MH080G2C1 with 02HD firmware for TRIM support) from Intel is used to accommodate the operating systems, all applications, virtual machine templates and linked clones based on these templates. Also, it runs a 10GB iSCSI Target (using the free StarWind iSCSI Target software).

The 250GB 7200RPM 2,5″ SATA disk (Seagate ST9250410ASG) is used to store data that does not need high transfer speeds, such as application installers and ISO’s (a semi-complete library of all VMware software), my music, video and picture collections. Also, a (slower but bigger) iSCSI Target of 25GB is placed on this disk.

With the virtual machines placed on the SSD, they are lightning fast. Because I am using the linked clone technology in VMware Workstation in combination with template virtual machines, I do not have to worry about used disk space. Cloned VMs only use a little bit of storage. A complete SRM setup, which requires five Windows 2003 VM’s, two ESXi machines and two NetApp Simulator take up only 14GB’s of disk space, including the parent template VMs.

From a disk performance view, the laptop is more than capable to run the aforementioned VMware vCenter Site Recovery Manager configuration. As always, the amount of available memory is a problem. Too bad this laptop doesn’t accept more than 8GB…

Weight and size

The E6500 is a big laptop. Measuring in at 14.1″ (358mm) wide, 10.1″ (257mm) deep and 1.5″ high (38mm) and weighing in at 6.6 lbs (3kg), you’ll notice you are carrying a mobile datacenter around. That God I’ve got just the right item for lugging it with me:

Comparing the weight and size with its performance, I have to say that it’s worth the trade-off:

Obviously, the laptop lacks serious strength in the graphics department. As I’m not a gamer at all, frankly, I couldn’t care less. More graphics power will only drain the laptop’s battery power even faster.

I’ve been meaning to dig into something technical for a long time as a more than welcome change in daily routine. I’ve been on the hunt for a new house (in Utrecht), have been busy with the VMware Enterprise Solution Provider partnership and did a complete rebuild of my internal Lab Manager environment. Also, I’m beginning my preparations for the VCDX Design Exam (which I’m going to take at the end of February). A couple of months ago, I visited the subject of RDM’s from a different angle (link), but now I wanted to give a more complete report on the matter.

What is a Raw Device Mapping?

The VMware VMFS filesystem supports a special type of file. This file, an RDM, is a mechanism for a virtual machine to have direct access to a LUN on the SAN. This LUN can be formatted in any way desired, without the need to format it to VMFS and place a VMDK on it. This removes two additional layers of complexity (the VMFS and the VMDK).

An RDM is a symbolic link from a VMFS volume to a raw LUN. The mapping makes LUNs appear as files in a VMFS volume. The mapping file, not the raw LUN, is referenced in the virtual machine configuration. The mapping file acts as a proxy for a raw physical device. This file contains metadata for managing and redirecting disk access.

When a LUN is opened for access, the mapping file is read to obtain the reference to the raw LUN. Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.

The file gives you some of the advantages of direct access to a physical device while keeping some advantages of a virtual disk in VMFS. As a result, it merges VMFS manageability with raw device access.

Although VMware recommends that you use VMFS datastores for most virtual disk storage, on certain occasions, you might need to use raw LUNs or logical disks located in a SAN.

Virtual Compatibility Mode

Virtual mode for an RDM specifies full virtualization of the mapped device. It appears to the guest operating system exactly the same as a virtual disk file in a VMFS volume. The real hardware characteristics are hidden. Virtual mode enables you to use VMFS features such as advanced file locking and snapshots. Virtual mode is also more portable across storage hardware than physical mode, presenting the same behavior as a virtual disk file. When you clone the disk, make a template out of it, or migrate it (if the migration involves copying the disk), the contents of the LUN is into a virtual disk (.vmdk) file.

Physical Compatibility Mode

Physical mode for the RDM specifies minimal SCSI virtualization of the mapped device, allowing the greatest flexibility for SAN management software. In physical mode, the VMkernel passes all SCSI commands to the device, with one exception: the REPORT LUNs command is virtualized, so that the VMkernel can isolate the LUN for the owning virtual machine. Otherwise, all physical characteristics of the underlying hardware are exposed. Physical mode is useful to run SAN management agents or other SCSI target based software in the virtual machine. Physical mode also allows virtual-to-physical clustering for cost-effective high availability. LUNs attached to powered-on virtual machines and configured for physical compatibility cannot be migrated if the migration involves copying the disk. Such LUNs cannot be cloned or cloned to a template either.

Dynamic Name Resolution

Provides a user-friendly name for a mapped device. When you use an RDM, you do not need to refer to the device by its device name. You refer to it by the name of the mapping file, for example:

/vmfs/volumes/myVolume/myVMDirectory/myRawDisk.vmdk

This reference is permanent and will not change between reboots. VMFS uniquely identifies all mapped LUNs, and the identification is stored in its internal data structures. Any change in the SCSI path, such as a Fibre Channel switch failure or the addition of a new host bus adapter, can change the device name. Dynamic name resolution compensates for these changes by adjusting the data structures to retarget LUNs to their new device names.

User-friendly Persistent Names

Stores unique identification information for each mapped device. VMFS associates each RDM with its current SCSI device, regardless of changes in the physical configuration of the server because of adapter hardware changes, path changes, device relocation, and so on.

When do you use a RDM?

SAN-based features

If you are using SAN-based features, sometime you have no choice but to use a RDM. Examples are Dell/Equallogic’s VSS Writer for snapshot and backup purposes, NetApp’s SnapManager for Exchange/SQL and so forth. Using a RDM enables these kinds of software to ‘see’ inside the LUN and to create a consistent snapshot of the data contained within the LUN. This allows for replication of data (snapshots) to a second SAN (on a different physical location) for Disaster Recovery. These types of software do require communication with the application, thus usually require an agent inside the VM. With VSS, the ‘agent’ is installed by default by the Guest OS, as it is a Windows feature.
It might also be use to create backups of the data on those LUNs in a much more efficient way as compared to traditional backup agents or even VM-based snapshots (using VCB or the vStorage API and Data Recovery). With these techniques, say, your Exchange database is backed up using resources (i.e. the hardware) of your SAN and thus relieving the hypervisor of any backup or snapshot tasks. This is also known as off-host backups. Depending on your environment, you can either use a physical backupserver (and integration between the backupsoftware such as Symantec Backup Exec and the SAN such as Equallogic), a NDMP-enabled VTL or a second SAN.
Also, in-band management software for your SAN running in Virtual Machines might require direct access to a management LUN. This usually requires physical compatibility mode.

Clustering

A second reason why you would want to use RDMs is in the case of a clustering service, such as Microsoft Clustering Services, Heartbeat and Novell Clustering Services (based on Heartbeat nowadays). There are three types of clustering, each with its own requirements on the storage side, which I will explain below.

Big ol’ disks

The last case for RDMs is ‘big datadisks’, with which you feel uncomfortable having it all in a single VMDK. In the words of Duncan Epping, ‘large’ is a disk larger than 800GB. This however does not mean that any virtual disk larger than 800GB should be a RDM. You could, for manageability and other reasons, create a dedicated VMFS Datastore with just this VMDK. Choosing between either is a matter of requirements as mentioned above and confidence in the technical feasibility of RDMs.

Requirements and limitations

• Blockbased storage. Both iSCSI and FC are supported. No NFS. No local storage. You need a VMFS datastore to store the mapping file.
• No partition mapping – RDM requires the mapped device to be a whole LUN.
• You cannot create a snapshot of a Physical Compatibility Mode RDM.
• You cannot enable Fault Tolerance for a VM using a Physical Compatibility mode RDM.
• VCB cannot backup a RDM in Physical Compatibility mode.
• VMware Data Recovery cannot backup a RDM in Physical Compatibility mode.
• When using VMware vCenter Site Recovery Manager, make sure you include both the raw LUN as well as the VMFS datastore holding the mapping file to SAN replication schedules. Placing the mapping file alongside the VMX file is highly recommended.
• VMware vCenter Server does not support LUN number changes within the target. When a LUN number changes, the vml identifier does not change accordingly. Remove and rebuild the RDM to generate a new vml ID that the LUN recognizes as a workaround.
• The maximum size of an RDM is 2TB minus 512 bytes.
• Use the Microsoft iSCSI initiator and MPIO inside the VM when you need more than 2TB on a disk.
• You cannot use a RDM (in conjuncture with NPIV) to give a VM access to a VMFS volume. This seems to make no sense at all, unless you want to run your VCB proxy as a virtual machine to access Fibre Channel LUNs. For iSCSI, you can still use the software iSCSI Initiator in your Guest OS. Mental note: obviously, it isn’t supported at all.

Compatibility with…

Storage VMotion and cold migration

A RDM can be included when you perform a Storage VMotion or cold migration. When using Physical Compatibility mode, only the mapping will be migrated (choose ‘Same as Source’ in the Migration Wizard). When using Virtual Compatibility mode, the raw disk will be migrated to either a thin provisioned or thick VMDK disk. Physical Compatibility mode RDMs cannot be migrated to a VMDK file. RDM converted to VMDK virtual disks cannot be converted back. Please note that NFS is not supported as a destination for a RDM disk migration. Use an intermediate step (Storage VMotion to a VMFS volume, either local or on an iSCSI/FC SAN) as a workaround.

RDM’s are fully supported for normal VMotion operations.

N-Port ID Virtualization

Makes it possible to use the NPIV technology that allows a single Fibre Channel HBA port to register with the Fibre Channel fabric using several worldwide port names (WWPNs). This ability makes the HBA port appear as multiple virtual ports, each having its own ID and virtual port name. Virtual machines can then claim each of these virtual ports and use them for all RDM traffic.
NPIV is supported only for virtual machines with RDM disks. Virtual machines with regular virtual disks continue to use the WWNs of the host’s physical HBAs.

Clustering

There are a couple of different ways to build a cluster environment, each with its own requirements on the storage side:

Cluster-in-a-box

Most often used for test- and development use cases, this type of clustering has two or more VM’s running on the same physical box. You can use both kinds of RDMs, as well as VMDK’s. VMware recommends using VMDK virtual disks, unless you are using Microsoft Clustering, in which case you need to use a RDM. I would recommend Physical Compatibility mode.

Cluster-across-boxes

Instead of running all cluster nodes on a single physical box, you run the VM’s across multiple physical boxes. This requires either type RDM. VMware recommends Virtual Compatibility mode. In the case of Microsoft Clustering Services, use Physical Compatibility mode.

Physical-to-Virtual Cluster

Using both physical boxes (without hypervisor) and virtual boxes requires the use of Physical Compatibility mode RDMs.

For Microsoft Clustering Services, there are some extra caveats. For instance, using Windows Server 2008 in a Failover Cluster scenario, you cannot use Virtual Compatibility mode. For clustered Continuous Replication Environment for Microsoft Exchange: use Physical Compatibility mode. Also, iSCSI is still not supported. You’ll need to use Fibre Channel.

FAQ

If using a RDM, what is written to physical disk regarding information about the RDM?
Nothing, the LUN contains only information about the filesystem the Guest OS formatted it with. There is no additional data on the disk.
Can I use a local disk as a RDM?
No, local RDM’s are not supported. Using the vSphere Client, you cannot assign local storage as an RDM to a VM. There are ways to achieve local RDM’s, though, for instance, connecting a SCSI device using SCSI-passthrough.
Can I detach a RDM and attach the LUN to a physical server?
Just like using normal LUNs or even simple external USB hard drives: yes you can, but it is up to the OS to whom the LUN is attached if this will work without a hitch. For the usual filesystems like NTFS, EXT3, ZFS and NSS, this is no problem at all.
How is a RDM presented to the Guest OS in a VM?
The Guest OS sees a local SCSI hard drive, just as it would when using a VMDK or Guest OS initiated iSCSI connected disk. Depending on the chosen compatibility, the Guest OS would see a ‘VMware Virtual SCSI Disk Device’ or a disk named by the storage array (it would be likely to have the vendor’s name somewhere in there).
Can I use RDM-like features on other products than vSphere?
Sure you can. VMware Workstation can give access to a complete LUN (or physical disk) or partitioned parts of a disk.

Sources for this blogpost

• Introduction to VMware vSphere
• vSphere Basic System Administration
• ESX Configuration Guide
• vSphere Availability Guide
• Fibre Channel SAN Configuration Guide
• VMware Data Recovery 1.1 Administration Guide
• VMware vCenter SRM 4 Release Notes
• VMware vCenter Server 4.0 Update 1 Release Notes
• Configuration Maximums for VMware vSphere 4.0
• Virtual Machine Backup Guide
• Setup for Failover Clustering and Microsoft Cluster Service
• 2vcps and a Truck: Using iSCSI to get some big ole disk in a Virtual Machine
• The VMguy: vSphere and MSCS
• Virtual Black Hole: To Use RDM’s or Not To Use RDM’s
• boche.net: Setup for Microsoft cluster service
• Leo’s Ramblings: Deciding between iSCSI RDMs and the Microsoft’s iSCSI initiator

Besides the two Labs I have at work, I’ve gotten myself a new laptop to use as a lab-inbetween-home-and-work-lab.

Hypervisor Lab @Work

Two Dell PowerEdge T605 (see my previous post)

Lab Manager @Work

We’re running Lab Manager on three Dell PowerEdge 1950’s (16GB RAM, dual core Xeon’s) and a Dell/EMC AX150i with 12×500 GB SATA. Nothing really fancy, but an adequate solutions for our training and testing purposes.

Mobile Lab @Bag

A brand new Dell Latitude E6500

• Intel Core 2 Duo P9700
• 8GB 800MHz DDR2 RAM (2x 4GB)
• 250GB SATA-II HDD 7200RPM
• 15.4″ Wide Screen WUXGA (1920×1200) 2CCFL
• Mobile Intel GMA X4500HD
• Intel WiFi Link 5100 (802.11 a/g/n 1X2) 1/2 MiniCard with Centrino label
• US/Int – Internal Qwerty LED Backlight Keyboard
• 6 Cell 54WHr LI-ION Primary Battery
• Microsoft Windows 7 Ultimate (x64, English)
• 8x DVD+/RW Drive

I’m planning to replace the DVD-drive with an caddy with an Intel 80GB SSD:

I’ve upgraded my employer’s Lab Manager installation to 4.0.1 last thursday. I followed procedure by disallowing new deployments, undeploying all virtual machines, making sure nobody could login, double-decked that no event were currently running and make backups of the virtual machine and database files.

I commenced the upgrade, and clicked through the installer. The upgrade went smooth as far as I could see. After the upgrade, I went to the webpage of the LM-server to finish the upgrade wizard. Instead of the upgrade wizard, I was welcomed by the initialization wizard. It scared the hell out of me, this was not supposed to happen!

I assumed the installer did a boo-boo with the database, replacing it with a fresh one instead of upgrading the old one. I was right, in the installation directory of SQL I found the ‘old’ database in a *.renamed format. The new database files were smaller and had newer modified dates.

To restore the database, I had to do the following:

• Stop the HTTP SSL, IIS Admin Service, World Wide Web Publishing Service, VMware vCenter Lab Manager Monitor and VMware vCenter Lab Manager SupportLink services.
• Make sure that the ‘SQL Server (LABMANAGER)’ service is running.
• Restore the database using osql.
• Restored user rights to the LabManager database using osql.
• Restarted the above services (including the SQL Server).

I used osql for all the database interaction. You can use Microsoft SQL Server Management Studio Express as well.

Restore the database:

OSQL -S localhost\labmanager -E -Q "restore database labmanager
 FROM DISK='c:\lm_backup.bak' with replace"

Restore the user rights to the database:

osql -S localhost\LabManager -E
USE [LabManager]
GO
CREATE USER [LAB\VMwareLMClientAccts] FOR LOGIN [LAB\VMwareLMClientAccts]
GO
USE [LabManager]
GO
EXEC sp_addrolemember N'db_datareader', N'LAB\VMwareLMClientAccts'
GO
USE OLTP
GO
EXEC sp_change_users_login 'auto_fix','LAB\VMwareLMClientAccts'
GO

For future reference: backups of the database are made using this command:

OSQL -S localhost\LabManager -E -Q "backup database labmanager
 TO DISK='c:\lm_backup.bak'"

At the fifth Dutch VMUG Event, I’ve given a presentation on VMware vCenter Lab Manager 4. I’ve uploaded the slidedeck of this (Dutch) presentation to SlideShare, a nifty way to share your presentations.

Follow the Dutch VMUG Event 2009 live on twitter using the #DutchVMUG tag. Enjoy this video of Steve Herrod, CTO of VMware, congratulation all Dutchmen on the fifth Dutch VMUG Event:

Check out the registration and entrance as well:

Check out Viktor van den Berg, the Dutch VMUG leader, doing the keynote:

You’ll know when you’ve come up with a good title when other people start copying it. Thanks Jason!

I’m setting up a VMware vSphere environment for a demo and presentation I’m giving later this week on VMware vCenter Lab Manager 4 at the Dutch VMUG Event. I prepared a couple of VM’s on my laptop running VMware Workstation, among wich a vCenter Server installation. I installed two physical servers with ESX4, attached them to vCenter and configured a dvSwitch on both. As these hosts are really simple desktops with just a single NIC, I had to configure the dvUplinks group with one uplink per host.  I migrated the Service Console Port (vswif0), created a couple of port groups for Virtual Machine networking and created a VMKernel port for NFS and VMotion. Since there was no need for the standard vSwitch anymore, I simply deleted the associated Port Groups and the vSwitch itself:

Now I had all ESX services (vswif0 and vmk0) and virtual machine port groups running on the dvSwitch. The end result was pretty:

So now for the migration of those local VM’s to the newly set up ESX-hosts using VMware Converter Standalone. I shut down the four VM’s and created the Convert task to v2v them. As I reviewed the options, I noticed that Converter picks up Port Groups from a dvSwitch, but does not advertise them as such:

Oh well, as long as they’re there, right? I started the v2v tasks and prepared some home-made pizza in the mean time. After an hour or so, I came back to the computer (the pizza was great!) and found all tasks successfully completed. However, the VM’s, now running on vSphere, were not present on the network. I could not ping them or find their MAC-adresses in the switch. They were simply not online. They booted correctly, had VMware Tools running, had the correct TCP/IP settings, but their virtual network adapters weren’t connected to the dvSwitch:

To make matters worse: I could not select the correct network label (Port Group) in the drop down list. After some long and hard thinking, I figured out why: ESX couldn’t communicate with vCenter to update the dvSwitch’s status. This is simply because the vCenter VM was one of the migrated VM’s, and thus suffered from the same problem: it wasn’t connected to the network. How’s that for a catch-22!

As I said earlier, the physical hosts run on a single vmnic. No easy fix here then, I cannot create a standard vSwitch, create a port group on it, add a vmnic and migrate the vCenter VM to this port group to get the VM online and thus be able to get the other VM’s attached to the right (dvSwitch) Port Group, after which I can migrate the vCenter VM to the right PG.

I had to break the dvSwitch on the physical host by removing vmnic0 and vswif0 from it and attach it to a standard vSwitch. As you cannot migrate a vmnic and a virtual adapter like the Service Console Port at the same time, I had a major problem. Migrating either would mean that I would lose all connectivity to the host. Getting them both attached to a standard vSwitch was my only way out of this conundrum. I need to fall back to the CLI for this one!

First, I removed vswif0 and vmnic0 from the dvSwitch:

esxcfg-vswif -d vswif0
esxcfg-vswitch -Q vmnic0 -V 969 dvSwitch

I created a new standard vSwitch and Port Group and attached vmnic0 as an uplink. I also recreated vswif0:

esxcfg-vswith -a vSwitch0
esxcfg-vswitch -A SCTMP vSwitch0
esxcfg-vswitch -A VMTMP vSwitch0
esxcfg-vswif -a -i 10.10.10.202 -n 255.255.255.0 -p SCTMP vswif0

I had restored the original setup: a simple standard vSwitch with a Service Console Port and a Virtual Machine Port Group. VMnic0 provided the link to the outside world. I moved the vCenter VM to the VMTMP Port Group. vCenter and ESX could finally negotiate details on the dvSwitch, so I could get on with it, attaching all other VM’s to the correct (dvSwitch) Port Group. The last VM to go was obviously the vCenter VM, as doing so would result in the VM losing connectivity to the network again. I quickly dove back into the CLI and reversed the commands I had just given to attach the Service Console to the dvSwitch again.

esxcfg-vswif -d vswif0
esxcfg-vswitch -D VMTMP vSwitch0
esxcfg-vswitch -D SCTMP vSwitch0
esxcfg-vswitch -d vSwitch0

And recreated the vswif0 interface on the dvSwitch, and attached vmnic0 to the dvSwitch:

esxcfg-vswif -a -i 10.10.10.202 -n 255.255.255.0 -V dvSwitch -P 141
esxcfg-vswitch -P vmnic0 -V 969 dvSwitch

Wow, what a way to go to get a couple of VMs to the correct Port Group, phew! Of course, having just a single NIC in a dvSwitch environment is BAD. I was just too lazy to arrange some extra NICs for these desktops because I’ll only be using them temporarily. That being said, I learned first hand why a hybrid solution (which requires multiple physical NICs) is a great idea. For further reading on hybrid environments, please do check out VCDX 007’s post on it!

I’ve been experiencing a problem with a very small VMware ESXi 4 host on which I run two VM’s for VMware vCenter Lab Manager management (a vCenter VM and a LM VM). I was rebuilding the environment, so I had both VM’s running on this host with (at that time) 3GB of RAM. When adding this host to vCenter, I got weird and cryptic error messages:

Cannot install the vCenter agent service. Unknown installer error.

I Googled my ass off, but never found an explanation or workaround. Finally, I decided to move the VM’s to another host, put it in maintenance mode, reboot it, and tried to add it to vCenter once more. For some reason or another, this worked!

Mike Laverick of RTFM Education posted a link to a VMware KB article exactly described what went wrong in this situation, and how to fix it. Like Mike, I find it funny that the solution is to migrate the running VM’s to another host and put more memory in the box:

This issue can occur if the ESXi host is low on RAM. The host can be low
on RAM either because it does not have enough physical memory allocated
to it, or because too many virtual machines are consuming
RAM at the same time.

If the host does not have enough physical memory allocated to it, you
can allocate more physical RAM:

 1. Power off the virtual machines on the host.
 2. Power off the ESXi host.
 3. Allocate more physical RAM to the host.
 4. Reboot the ESXi host.

In any case, I’m happy Mike found this KB article, because I now know that more (RAM) is always better, especially in this case! Thanks Mike!

Zoals Arne, Bouke en Viktor, Ivo, Eric, Arnim, Duncan en Gerben al besproken hebben vindt op 11 december 2009 het jaarlijkse Dutch VMUG Event 2009 plaats! Dit jaar is het extra feest, want het evenement wordt voor de vijfde keer georganiseerd!

Het evenement staat dit jaar uiteraard in het teken van de release van vele versie 4 producten, allen onder de vSphere vlag. Zo gaat Eric Sloof een sessie verzorgen over het beheren van een vSphere 4 omgeving met behulp van de Virtualization EcoShell en Luc over PowerCLI, Verron over de integratie van NetApp systemen met de vStorage API, Viktor over de vNetwork Distributed vSwitches, ikzelf over vCenter Lab Manager 4, Willem over Citrix XenApp op vSphere en tenslotte Gabrie over de best practices bij het ontwerpen van een vSphere omgeving.

De volledige agenda is hier te bekijken, en de diverse sprekers hier. Naast deze parallelle sessies is er dit jaar ook ruimte voor een workshop over vCenter Hearbeat. Check ook de sponsors, die ook op de beursvloer aanwezig zullen zijn. Voor de locatie en andere praktische informatie ga je hierheen.

Zoals gezegd, ik zal zelf ook een sessie verzorgen. Ik ga jullie wegwijs maken in de wereld van VMware vCenter Lab Manager 4. Lab Manager biedt de gebruiker de mogelijkheid om de interne test- en ontwikkelomgeving te automatiseren.

Net zoals de laatste paar jaar is er ruimte voor rond de 600 deelnemers. Zoals een concert van Foreigner is dit congres meestal snel helemaal volgeboekt. Schrijf je dus snel in op de site van de VMUG: http://www.vmug.nl/modules.php?name=Inschrijven

I’ve created a small video on how to reproduce the hostsfile bug in NetApp System Manager 1.0.1:

While tinkering around with the new NetApp System Manager 1.0.1, I found something that could be considered a bug. I’m connecting the manager to a FAS2020 with the most recent ONTAP version (7.3.2), and needed to set up some (temporary) virtual IP’s and DNS-names. For this last item, I edited the filer’s hostfile using NetApp System Manager:

While scrolling through the syslog, I noticed an error statement about the hosts file:

After a reboot, I saw all kinds of error scroll by, basically saying the hosts file was broken and the hostname of the filer could not be read. This means that the filer won’t boot up properly. This means it won’t present any storage services like CIFS, NFS and iSCSI to clients. I need to reset the hostname trough the console, for which I have not found any usable method yet. The usual command, ‘hostname’, won’t budge, it gives the following error:

I confirmed this bug on both the FAS2020 and the NetApp Simulator (7.3.1). I can only assume this is a major bug in the NetApp System Manager 1.0.1…

Update:

After consulting NetApp support, I figured out how to fix this error on the filer: just re-run setup from the console. In my case, all the pre-entered values such as network settings were recognized and used in the wizard, I only had to fill the hostname in again.

I went to a customer today to configure a NetApp FAS2020 for SnapMirror replication. For sake of argument, let’s call this filer FLR2. After hooking up the fibre channel interconnect to a second diskshelf (containing SATA-disks), I noticed that all disks from the onboard shelf were missing. They were added to a second FAS2020 (FLR1), which is running all the production VM’s and thus has more benefit from the 300GB 10K SAS disks than FLR2, which was going to do only SnapMirror replication from FLR1 over a 20 Mbit internet connection. Without the SAS-disks in place, which hold aggr0 and the root volume (vol0), there wasn’t much to boot from. Hence, I had myself a very expensive piece of useless storage.

Because my experience with NetApp filers has been relatively minimal up until this point, I needed to dive into it deeply to figure out how the root volume works, why NetApp has placed it onto the ‘data’ disks (and not onto the CompactFlash storage), and most important, how to fix this mess.

The bootlog

I began by examining the output of the boot process. I hookup up the serial connection, and saw the following:

AMI BIOS8 Modular BIOS
Copyright (C) 1985-2006,  American Megatrends, Inc. All Rights Reserved
Portions Copyright (C) 2006 Network Appliance, Inc. All Rights Reserved
BIOS Version 3.0
+++++++++++++++

Boot Loader version 1.3
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.
Portions Copyright (C) 2002-2006 Network Appliance Inc.

CPU Type: Mobile Intel(R) Celeron(R) CPU 2.20GHz

Starting AUTOBOOT press Ctrl-C to abort...
Loading:.................0x200000/33899484 0x22543dc/31974692 0x40d2900
 /2557760 Entry at 0x00200000
Starting program at 0x00200000
cpuid 0x80000000: 0x80000004 0x0 0x0 0x0
Press CTRL-C for special boot menu

[fci.initialization.failed:error]: Initialization failed on
 Fibre Channel adapter 0b.

NetApp Release 7.2.6.1: Wed Dec 10 21:26:02 PST 2008
Copyright (c) 1992-2008 NetApp.
Starting boot on Mon Nov  2 15:50:00 GMT 2009

[nvram.battery.turned.on:info]: The NVRAM battery is turned ON.
 It is turned OFF during system shutdown.
[diskown.isEnabled:info]: software ownership has been enabled
 for this system
[coredump.spare.none:info]: No sparecore disk was found.
 WARNING: 0 disks found!

Storage Adapters found:
2 Fibre Channel Storage Adapters found!
1 SAS Adapters found!
0 Parallel SCSI Storage Adapters found!
1 ATA Adapters found!

Target Adapters found:
0 Fibre Channel Target Adapters found!
1 iSCSI Target Adapters found!
1 Unknown Target Adapters found!

WARNING: there do not appear to be any disks attached to the system.

Check that disks have been assigned ownership to this
system (ID 135058219) using the 'disk show' and 'disk assign'
commands from maintenace mode.

No root volume found.
Rebooting... (ctrl-c to break reboot loop)

OK, so the bootloader is still functioning properly, but complaining about the lack of disks, which makes perfect sense, because the ownership of the disks hasn’t been claimed yet. More importantly, even if the filer had claimed the disks, there wouldn’t be anything on those disks, so the boot process would’ve stopped in a similar fashion anyway.

The solution

So, the filer doesn’t have any disks assigned to it and there are no volumes defined on those disks. Let’s fix that! See the red bits in the log below for details:

1. We’ll start out by pressing [CTRL] – [C] to enable the very special boot menu.
2. Select option 4a to assign ownership of a couple of disks, initialize them and create a flexible root volume.

AMI BIOS8 Modular BIOS
Copyright (C) 1985-2006,  American Megatrends, Inc. All Rights Reserved
Portions Copyright (C) 2006 Network Appliance, Inc. All Rights Reserved
BIOS Version 3.0
+++++++++++++++

Boot Loader version 1.3
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.
Portions Copyright (C) 2002-2006 Network Appliance Inc.

CPU Type: Mobile Intel(R) Celeron(R) CPU 2.20GHz

Starting AUTOBOOT press Ctrl-C to abort...
Loading:.................0x200000/33899484 0x22543dc/31974692 0x40d2900
 /2557760 Entry at 0x00200000
Starting program at 0x00200000
cpuid 0x80000000: 0x80000004 0x0 0x0 0x0
Press CTRL-C for special boot menu
Special boot options menu will be available.

[fci.initialization.failed:error]: Initialization failed on
 Fibre Channel adapter 0b.

NetApp Release 7.2.6.1: Wed Dec 10 21:26:02 PST 2008
Copyright (c) 1992-2008 NetApp.
Starting boot on Mon Nov  2 15:57:08 GMT 2009

[nvram.battery.turned.on:info]: The NVRAM battery
 is turned ON. It is turned OFF during system shutdown.
[diskown.isEnabled:info]: software ownership has
 been enabled for this system

(1)  Normal boot.
(2)  Boot without /etc/rc.
(3)  Change password.
(4)  Assign ownership and initialize disks for root volume.
(4a) Same as option 4, but create a flexible root volume.
(5)  Maintenance mode boot.
Selection (1-5)? 4a

The system has 0 disks whereas it needs 3 to boot, will try to assign the
required number.

[diskown.changingOwner:info]: changing ownership for disk 0a.18
 from unowned (ID -1) to  (ID 135058219)
[diskown.changingOwner:info]: changing ownership for disk 0a.23
 from unowned (ID -1) to  (ID 135058219)
[diskown.changingOwner:info]: changing ownership for disk 0a.27
 from unowned (ID -1) to  (ID 135058219)

Zero disks and install a new file system? y
This will erase all the data on the disks, are you sure? y
Zeroing disks takes about 280 minutes.

[coredump.spare.none:info]: No sparecore disk was found.

[raid.disk.zero.done:notice]: Disk 0a.18 Shelf 1 Bay 2
 [X269_SMOOS01TSSX NA01]: disk zeroing complete
[raid.disk.zero.done:notice]: Disk 0a.27 Shelf 1 Bay 11
 [X269_SMOOS01TSSX NA01]: disk zeroing complete
[raid.disk.zero.done:notice]: Disk 0a.23 Shelf 1 Bay 7
 [X269_SMOOS01TSSX NA01]: disk zeroing complete

[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.27 Shelf 1 Bay 11 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully
[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.23 Shelf 1 Bay 7 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully
[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.18 Shelf 1 Bay 2 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully

[wafl.vol.add:notice]: Aggregate aggr0 has been added to the system.
[fmmbx_instanceWorke:info]: no mailbox instance on local side
[fmmb.current.lock.disk:info]: Disk 0a.18 is a local HA mailbox disk.
[fmmb.current.lock.disk:info]: Disk 0a.23 is a local HA mailbox disk.
[fmmbx_instanceWorke:info]: normal mailbox instance on local side

After configuring the hostname, network settings for e0a and the bmc-adapter, I was ready to hit NetApp FilerView to configure the rest. But the web server that should be running FilerView wasn’t running. Back to the console log once more:

[init_java:warning]: Java disabled:  Missing /etc/java/rt131.jar.
[httpd_servlet:warning]: Java Virtual Machine not accessible
[asup.config.minimal.unavailable:warning]: Minimal Autosupports
 unavailable. Could not read /etc/asup_content.conf
[sysconfig.sysconfigtab.openFailed:notice]: sysconfig: table of
 valid configurations (/etc/sysconfigtab) is missing.

Hmm, it seems the filer is still missing a lot of files, all in the /etc folder, which should reside on the root volume. Most importantly, the Java bits needed to use FilerView weren’t there. It looks like the Data ONTAP installation still isn’t recovered fully. As I noticed this filer’s ONTAP version (7.2.6.1) was old anyway, I decided to upgrade the software in the usual fashion. I browsed to the NOW page for ONTAP v7.3.2, Checked out the release notes, changed features page, the requirements for running this version, read up on the known problems and limitations and important cautions.

I was convinced that this release was the right one for me. As this was the first time I did an actual upgrade of ONTAP, I decided to read up on the Data ONTAP Upgrade Guide as well. While we’re at it, I downloaded the complete set of documentation for Data ONTAP here. Lastly, I grabbed the ‘System Files‘ and ‘Netboot Image‘.

Upgrading itself seemed easy: the filer wasn’t performing any tasks or I/O and isn’t in any cluster. This means I can simply jank out the power cables if I wanted to. Not that that’s such a good idea when upgrading firmware, but still. The second thing that made the upgrade easier was that the BIOS/NABL (v3.0), Diagnostics (v5.3.7) and the BMC controller firmware (v1.2) are all up to date.

Since the wipe-out of ONTAP by removing the SAS disks also destroyed any reference to licenses, I could not place the ONTAP upgrade software on the filer using CIFS or NFS, or even iSCSI or FCP. The only option I could use was to download it through HTTP. So I placed the System File onto a HTTP server, ready for the filer to pick up the bits and do its magic.

flr2> software update http://webserver/ontap/732_setup_e.exe
software: copying to 732_setup_e.exe software:
100% file read from location.
software: /etc/software/732_setup_e.exe has been copied.
software: installing software, this could take a few minutes...
software: installation of 732_setup_e.exe completed.

[rc:info]: software: installation of 732_setup_e.exe completed.
 Please type "download" to load the new software, and "reboot"
 subsequently for the changes to take effect.

flr2> download 
[download.request:notice]: Operator requested download initiated
 download: Downloading boot device download: If upgrading from a
 version of Data ONTAP prior to 7.3, please ensure download: there
 is at least 3% of available space on each aggregate before upgrading.
 Additional information can be found in the release notes.

Version 1 ELF86 kernel detected.............
download: Downloading boot device (Service Area).........
[download.requestDone:notice]: Operator requested download completed

flr2> reboot 
[kern.shutdown:notice]: System shut down because : "reboot".

Wow, that’s way easier than I expected. Copy the file to the filer, install the .exe image, activate the new code on the filer’s boot device and actually reboot the unit. During reboot, a couple of messages about updating software on one of the shelfs flew by:

[kern.version.change:notice]: Data ONTAP kernel version was changed from
 NetApp Release 7.2.6.1 to NetApp Release 7.3.2.
[sas.adapter.firmware.download:info]: Updating firmware on SAS adapter 0c
 from version 1.24.03.00 to version 1.26.03.00.
[dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
[sfu.firmwareDownrev:warning]: Disk shelf firmware needs to be updated on
 1 disk shelf.
[sfu.ctrllerElmntsPerShelf:info]: [storage download shelf]: 1 ES
 controller element can be updated on 0c.shelf0.
[sfu.downloadStarted:info]: Update of disk shelf firmware started
 on 1 shelf.
[sfu.downloadingController:info]: [storage download shelf]: Downloading
 SAS.0500.SFW on disk shelf controller module B on 0c.shelf0.
[sfu.suspendDiskIO:info]: Suspending disk I/O to reboot shelf modules for
 shelf firmware update.
[sfu.rebootRequest:info]: Issuing a request to reboot disk shelf
 0c.shelf0 module B.
[ses.shelf.fw.update.pfu.nrdnt:info]: SCSI Enclosure Services (SES)
 shelf firmware update will be performed using Package Firmware
 Update (PFU) due to not having redundant paths.
[sfu.adapterSuspendIO:info]: Suspending I/O to SAS adapter 0c for
 40 seconds while shelf firmware is updated.
[sfu.resumeDiskIO:info]: Resuming disk I/O after shelf firmware update.
[sfu.downloadSuccess:info]: [storage download shelf]: Firmware file
 SAS.0500.SFW downloaded on 0c.shelf0.
[sfu.downloadSummary:info]: Shelf firmware updated on 1 shelf.
[sfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk shelves.

Nothing shocking to notice here, just confirmation that the ONTAP kernel version has been upgraded, that the SAS Adapter 0c and ES shelf controller 0c firmware will be upgraded and rebooted afterwards. Also, good to see the extra confirmation that all disk shelves and disk drives have current firmware versions after upgrading!

Concluding

Don’t ever take out the disks if the root volume is running on them. Period.

If you do, like this customer did, you’ll have yourself a couple of hours o’ fun restoring and upgrading ONTAP, as well as a heart-to-heart with the people who removed them in the first place, which is likely to take up the remainder of the day.

I’ve created a separate page (click the ‘vSphere Client on Win7‘ link above) to accomodate the most popular blog post on Virtual Lifestyle. No more wading through my site to get to the good stuff, just click that easy to spot, easy to use button above

I was planning a quick upgrade of a small cluster of ESX3 hosts to the latest and greatest yesterday. After upgrading vCenter, Update Manager and the vSphere Client, I set up an upgrade baseline in VMware vCenter Update Manager, took an ESX-host out of production and tried to remediate the host. After some thirty minutes, the task failed. No decent error was displayed in the ‘events’ view within the vSphere Client.

As I was wading through log files (/esx4-upgrade/ is a great place to start, so is /var/log/vmware/), I noticed something about the /boot partition being too small. I had finally encountered this dreaded drawback in upgrading ESX3 to ESX4. Sadly, not much can be done about it, although I did try some magic to resize the /dev/sda1 partition on which /boot was residing, but to no avail. Today, the three servers in the cluster will be receiving a clean installation of ESX.

So, if you are having trouble upgrading to ESX4 using Update Manager (or the Host Upgrade Utility, for that matter), please remember that it could very well be caused by a small boot partition. Start your troubleshooting endeavours at /esx4-upgrade/. Also, make sure you read up on the vSphere Upgrade Guide, because it gives some invaluable information on the /boot partition:

The default ESX 2.5.5 installation creates a /boot partition that is too small to enable
upgrades to ESX 4.0. As an exception, if you have a non-default ESX 2.5.5 installation on which at least 100MB
of space is available on the /boot partition, you can upgrade ESX 2.5.5 to ESX 3.x and then to ESX 4.0.

As I was Googling for an updated version of the APC PCNS Agent for ESX, I stumbled upon this forumtopic:

PowerChute Network Shutdown for ESX and ESXi 4, part number SSPCNSV, is out and on our web site here . The downloadable version is SEPCNSV found here.

So there finally is a version specifically for VMware ESX and ESXi! Hooray! The downside is, you need to grab your checkbook to get the bits, as the software is 99 USD per physical host…

To install the PCNS on ESXi, you’ll need the following:

• VMware vMA (dedicated to the PCNS Agent);
• DRS and HA are supported;
• Manual VMotion isn’t supported;
• Every guest needs to have the VMware Tools installed.

I can only assume that having the VMware Tools as a requirement will gracefully shut down all running VM’s when the PCNS is triggered. That’s an added bonus!

Check out this page for more documentation. Please refer to VM /ETC for more documentation.

A recap on the history of virtualization

Full virtualization or binary translation allows an unmodified guest OS to run as a virtual machine. The earlier VMware products used this technique to allow for virtualization. This is the oldest of the virtualization methods, and also the slowest. Up until the arrival of hardware-assisted virtualization, full virtualization was the most versatile way to get things done.

The second option is to use para-virtualization. This uses the same binary translation, although a lot less has to be translated. This requires the guest OS to be heavily modified to be aware of the fact that it is virtualized. As many guest Operating Systems were not modified to work with para-virtualization, the improved speed (as compared to full virtualization) did not win many people over: the platform was way too restrictive for mainstream adoption. Later VMware-products used VMI for para-virtualization. VMI has retired this technique recently. Xen is the leader in para-virtualization, with the para-virtualization modules built into many open-source (UNIX and Linux) operating systems today.

With these two techniques in place, virtualization became more and more mainstream. Processor manufacturers began to feel the pressure by the market to speed up virtualization. They began to implement many of these binary translations into the processor, calling them ‘Hardware Assisted Virtualization’. AMD-V and Intel VT are the most commonly used implementations of the platform. Hardware Assisted Virtualization is the commonly used platform for virtualization, giving customers the best of both worlds: the speed of para-virtualization (without compromising on modifying the guest OS) and the easy of use of full virtualization. Hardware Assisted Virtualization has therefore become the industry standard, and the platform is improving all the time. More and more features are being implemented, making virtualization even faster and more flexible. Hardware Assistance is reaching out to other forms of virtualization. VMDirectPath allows for virtualization of I/O devices, such as Network Interface Cards or Storage Controllers.

Future directions of the hypervisor

With the move to even heavier integration of the virtualization layer into the processor and other pieces of hardware, would it be wierd to think that the hypervisor as we know them now will disappear? From a performance point of view, I’d like my entire Virtual Machine Monitor to dissipate into the hardware. I’d call it Hypervisor-on-die. This could evolve from a modular chip, separated from the CPU itself (like the old memory controller was separated from the CPU) to a hypervisor completely integrated in the CPU’s instructionset. This hypervisor would run as an instructionset, like AMD-V and Intel VT do nowadays. This would require that the current architecture of a hypervisor would have to be pulled apart and heavily optimized.

Future directions of hypervisor management

Consider how management of VMware ESX is going to be. VMware has already stated that ESXi is the future. ESX with its Service Console is going to die eventually. Management functions are going to be executed on a freely movable and independent “Service Console”, now called the vMA. This reduces the number of excess weight each hypervisor has to carry around, while still being able to do all management tasks. But with the hypervisor completely embedded into the CPU-architecture, how will it be managed?

A first and simple way is using the vMA like it exists today. But consider the physical Remote Access Cards (like the Dell iDRAC and HP iLO). These cards are parasites feeding of the hosts’ capabilities, used to manage the hardware they live on. What if the vMA is evolving into a similar parasite, integrated into the hardware, using independent hardware? This would mean that there’s no requirement for separate software products, allowing for even tighter integration of the hypervisor and management software into the hardware.

Concluding

With this post, I tried to work out a possible scenario of the future of virtualization on a technical level. Do you think this is a feasible way that the hypervisor and its management tools might evolve?