Archive for August, 2010

Booting a Lefthand SAN/iQ node requires a serial port

Aug 25, 2010 Blogs

When you’re booting a Lefthand Virtual SAN Appliance (this also applies to the Failover Manager), make sure you don’t disable ‘Serial port A’. It is required for console output, which is set to ‘ttyS0′ in the syslinux.cfg file of the underlying Linux OS:

So, make sure that the BIOS is set like this:

and not like this:

If you want more information about the boot process, use the ‘ESC’ key to abort the bootloader and type ‘vga’ in the ‘boot:’ prompt. This wil disable output to the serial port and enable output to the screen:

HP LeftHand Multi-Site SAN & VMware vSphere

Aug 25, 2010 Blogs

I’d like to warn you for a not-so-obviously documented fluke in the combination of a HP LeftHand (P4000) Multi-Site Cluster and VMware vSphere hosts on multiple sites.

The situation

HP LeftHand (P4000) Multi-Site Cluster

HP LeftHand Multi-Site Cluster requires the storage nodes to be divided into two subnets, one per site. Each subnet (or VLAN) requires a Virtual IP in that subnet.

Example

I.e.: VLAN 1 with Virtual IP 10.10.1.100 (/24) for site 1 which hold storage node 1 and storage 3 and VLAN 2 with Virtual IP 10.10.2.100 (/24) for site 2 which hold storage node 2 and storage node 4.

LUN Ownership

Whenever a LUN (or volume, in Lefthand’s CMC) on the LeftHand cluster is accessed by an iSCSI Initiator, that volume is bound to the Virtual IP (and thus subnet / VLAN) by which the LUN is accessed.

Example

I.e.: when an ESX-host on Site 1 rescans it’s vmhba38, the volumes presented to that ESX-server get bound to this site’s Virtual IP, which is in VLAN 1.

VMware ESX software iSCSI Initiator and multiple subnets

The VMware software iSCSI Initiator (and, by extension, the dependent hardware iSCSI Initiator) do not support accessing a iSCSI Target (or LeftHand Virtual IP) outside it’s own subnet.

Example

I.e.: the ESX-hosts on Site 2, which have their iSCSI adapter in VLAN 2, cannot traverse the network to access the LUN that is bound to VLAN1. They to a discovery to the Virtual IP in VLAN 2, but get redirected to the Virtual IP in VLAN 1, because that’s where the LUNs are bound to. You’ll see errors in  /var/log/messages about iscsid not being able to connect to the Virtual IP in VLAN 1.

The solution

Multiple VMkernel ports for iSCSI in multiple VLANs

The only way to maintain synchronous replication (‘Network RAID 10′ in LeftHand naming convention) is to maintain the HP LeftHand Multi-Site Cluster. This means maintaining the multiple VLANs for iSCSI. To enable hosts on site 2 to access the LUNs, you’ll need to configure additional VMkernel ports for iSCSI in the other VLAN. So in addition to the two VMkernel ports in VLAN 2 on an ESX-host in site 2, you’ll add two VMkernel ports in VLAN 1 on that ESX-host. On the ESX-hosts in site 1, you’ll add two VMkernel ports in VLAN 2.

The impact of this change is additional iSCSI traffic over the intersite link: not only does the replication traffic travel between sites, now also iSCSI traffic from the ESX-hosts in site 2 travels the intersite link to the storage nodes in site 1.

Another change in this environment is the added complexity to complete a site failover. You’ll need to rescan all software iSCSI adapters after the primary site has failed, because the LUNs are still attached to the (now failed) Virtual IP of the primary VLAN. By rescanning, the LUNs are attached to the Virtual IP of the other VLAN. If site 2 fails, only the replication is lost, as site 2 doesn’t do any iSCSI traffic with any of the ESX-hosts but merely does Network RAID 10 replication.

If you dare create a single vSphere Cluster, spanning HA/DRS across sites, VMware HA can take care of VM failover. If hosts on site 1 fail but storage is still alive, a rescan on the hosts on site 2 is needed before HA can restart the VM’s. You’d better be quick with that rescan! If hosts on site 2 fail, no rescan or other action is needed, as HA will restart VMs in the ESX-hosts in site 1 (because ESX-hosts in site 1 have an active session with the LUNs).

Dependent hardware iSCSI

Aug 25, 2010 Blogs

I’ve implemented a new five-node VMware vSphere cluster for a customer. The hardware we’re using are HP ProLiant DL360 G7′s. They’ve got the new Broadcom 5709c Gigabit Ethernet Adapters, which present themselves as both standard Ethernet network cards as well as storage controllers (‘Broadcom iSCSI Adapters’ in the Storage Adapter section of the host’s Configuration Tab). The main advantage of these dependent hardware iSCSI HBAs is the offloading of iSCSI traffic to the TCP/IP Offload Engine on the NICs, which reduces the amount of processor power needed for the iSCSI traffic. VMware describes dependent hardware iSCSI as “Depends on VMware networking, and iSCSI configuration and management interfaces provided by VMware. This type of adapter can be a card that presents a standard network adapter and iSCSI offload functionality for the same port. The iSCSI offload functionality depends on the host’s network configuration to obtain the IP, MAC, and other parameters used for iSCSI sessions.”

As they are dependent hardware iSCSI HBAs, you’ll need to use standard VMkernel components to make them work. In a nutshell, these are the steps you need to go through:

  • There is a static relationship between a NIC port (vmnic#) and a dependent iSCSI HBA (vmhba##). Use ‘esxcli swiscsi vmnic list -d vmhba##’ to view which vmnic belongs to this vmhba.
    In my case, vmnic0 corresponds to vmhba32. vmnic1 corresponds to vmhba33.
  • Create a VMkernel port on a vSwitch with a single uplink (choose one of the vmnics which have a corresponding vmhba).
    I’ve created two vSwitch Port Groups: iSCSI0 and iSCSI1. iSCSI0 has vmk0 with uplink vmnic0, iSCSI1 has vmk1 with uplink vmnic1.
  • Bind the vmk# to a vmhba in such a way that the uplink on the vSwitch of the VMkernel port corresponds with the vmhba##. Use ‘esxcli swiscsi nic add -n vmk# -d vmhba##’.
    I’ve bound vmk0 to vmhba32 and vmk1 to vmhba33.
  • Verify that all is well with ‘esxcli swiscsi nic list -d vmhba##’

Some points to take into account:

  • Make sure that each VMkernel port  only has a single vmnic uplink. Make sure that the vmhba corresponding to the vmnic is the same one you’ll configure with esxli swiscsi
  • Use 1:1 mapping of vmk# interfaces to vmnic#/vmhba##
  • You can use iSCSI multipathing, as long as you use multiple VMkernel ports (each with it’s own vmnic/vmhba pair)
  • You can put all the VMkernel ports (vmk#) on the same vSwitch or you can create a separate vSwitch for each. Your choice!
  • Broadcom iSCSI adapters do not support IPv6 and Jumbo Frames. You should enable flow control though, especially if the network is congested.
  • The vmk/vmnic/vmhba pair you use for dependent hardware iSCSI (and software iSCSI for that matter) cannot establish a session to an iSCSI Target that is in a different subnet / VLAN. Use a second  vmk/vmnic/vmhba pair in that different subnet if you need to connect a single ESX-host to multiple iSCSI subnets / VLANs.
  • Make sure that each vmhba has a unique IQN.
  • Add the iSCSI Target IP address to the Dynamic Discovery tab of each of the vmhba adapters you want to use.
  • Performance Reporting for iSCSI Traffic is unavailable, because the traffic bypasses the regular network stack, where the performance reporting does is monitoring.