So, you guys might know I have this wacky IaaS platform called ZE KLAUWD. Imagine the Governator saying ‘ the cloud’ after he’s had a couple of beers.

Anyway, this platform started out as a single host deployment using consumer-grade components (which I decommissioned in August ’11), evolved to two Dell PowerEdge R310’s and now has a third member to the team. This 2U SuperMicro SuperServer 6026T-NTR+ has a X8DTN+ motherboard, dual Intel Xeon 5630 processors and a total of 48GB’s of RAM. In addition, an Areca ARC-1222 SAS RAID-controller with BBU and 256MB cache packs a decent storage punch. Two spare 250GB drives in RAID-1 on a HP 8 Internal Port SAS Host Bus Adapter with RAID make up the bootstrap VMFS volume for the infrastructure VMs. Specifically, a Vyatta router / firewall and a Nexenta Community Edition storage appliance.

My VM has 4 vCPU’s, 16GB of vRAM, two vmxnet3 NICs with the Areca controller in VT-d pass through mode. Three SSD’s and 5 SATA disks complete the rig. I have a 120GB SSD as L2ARC and a mirror of two 80GB SSDs for ZIL (although mirrored ZIL is not a requirement anymore). The five SATA disks are spread into a RAID-Z2 and a mirror. The 16GB of vRAM should give me enough ARC. This should give me plenty of IOps to grow the platform and host more VMs.

image2Image graciously stolen from Marco’s blog post


Using @MBroeken‘s guide to building a Nexenta VM, I learned that Nexenta has a NAS VAAI plugin. I’ve installed it on all three of my hosts, and I can confirm it works on the Nexenta CE 4.0 M20 beta build. I was doubting the 8-16k record size. The NexentaStor ‘NFS Concept and Reference Guide‘ has a good explanation, though:

Nexenta recommends a record size of 8 KB for virtualization workloads. By default, most file systems use block sizes of 4 KB. Although the hypervisor does its best to coalesce writes, most of the I/O seen from the storage appliance are between 4 KB — 16 KB of random operations. You can turn record size at any time. New writes to the share inherit the new record size. For data that is already stored on a share, Nexenta recommends storage vMotion to, optionally, convert all of the data to a new record size.

In a previous post about this platform and Nexenta, I mentioned I was going to use asynchronous ZFS-based snapshots and replication. I have abandoned this idea, opting for the Veeam option instead, since Veeam is capable of producing more consistent copies of the data. I keep an on- as well as an off-site copy of the backup files. I’m still investigating how to best leverage the new options in the Cloud Edition of Veeam.


Check out some screenshots of the build: