Nimble Storage launched about three years ago. In fact, it was at one of the other Tech Field Day events, Tech Field Day 3 (Seattle, 2010). They joined TechFieldDay again at Storage Field Day 2 in 2012 presenting their ‘scale-up‘ approach. They just kept coming back for more. In 2013, they presented their ‘scale-out’ architecture at Storage Field 4.


Let’s recap the scale-to-fit (Nimble-ese for ‘scales either way’) philosophy in two parts:


Scale-up  is very flexible due to the CASL architecture. Any specific dimension (capacity, cache and compute) in a Nimble Storage array can scale-up with just what’s needed. However, there’s only so much hardware you can throw at a single array.

The base array, consisting of two clustered controllers (containing compute and flash), one internal disk shelf and optionally up to three external disk shelves, is perfectly capable of delivering both enough performance and capacity as well as data protection. The challenge was to make it scale further than the physical maximums of a single array, performance- and capacity-wise, without changing the underlying base architecture.


In their #SFD4 presentation, Umesh Maheshwari, Founder and CTO of Nimble Storage did an excellent job of highlighting the critical area’s of Nimble’s past success. He explained how their approach of ‘scale-up (in the required dimension) first, then scale-out‘ from both an engineering perspective and the customer’s perspective makes perfect sense: it’s simply very difficult to build true scale-out without the limitations of monolithic or clustered architectures.

They succeeded in scaling out without changing the base architecture. All the hardware’s the same, and there’s no clunky interconnects between arrays to deal with. It’s just the same old hardware you’re used to. If you want to learn more about the CASL architecture and the hardware, please read Chris Wahl’s post as he did an excellent job.

So how did they achieve scale-out?

Well, by looking at the competition.

The EqualLogic architecture has done scale-out for a long, long time now, and has done it very successfully. Their peer storage architecture is one of their best features and very well thought out: the group, storage pools, storage members, performance load balancing (including the hybrid array load balancer), capacity load balancing and much more. It is still hard to grow beyond about a dozen members in this architecture, but that’s a major limitation of the clustered / peered architecture approach and not necessarily something on their part.

Imitation is the sincerest form of flattery. Nimble has taken on a very similar approach, albeit with other limitations and maximums. Given the pretty decent scale-up limitations per array, I think a single Nimble Storage array group can handle almost all use cases in the market they’re targeting: they’re not into low-end cheap ‘n’ deep or high-end performance storage but do a little of both while giving customers the choice to add either.

Single point of administration

The scale-out architecture allows you to group up to four arrays in a single multi-master management group. Nimble is taking the conservative road; this 4-array limit will likely be removed in an upcoming software release. But, for now, this scale-out architecture allows us to manage four arrays with a maximum of 16 disk shelves as a single system in one group.


As far as the comparison with EqualLogic goes, there are a couple of subtle but crucial differences:

  • Arrays with existing configurations can be added to a group while preserving data and configuration. This also applies when merging multiple arrays or groups into a single group.
    The use case here is that there are about 4000 systems out there with current data and configuration that, in the future, might be added into a group.
    Forced evacuation of the arrays just to add an array to a group wouldn’t be pretty.
  • Since each and every Nimble array offers performance not through disks (or tiering) but through flash cache and compute power, there is no performance load balancing.
  • Each volume’s slices (couple MiB in size) are spread equally across arrays. it is assumed that the equal split will split performance as well and prevent any I/O hotspots in a pool. Since each array contains all components to perform adequately and roughly equally, performance load balancing is implicit. This means that a homogeneous cluster of arrays is preferred in this version of the scale-out architecture to maintain performance characteristics.
  • Rebalancing of slices (couple of MiB) only really happens when adding or removing arrays (including individual shelves) to/from a pool, as long as each connected host is running the MPIO DSM / PSP plugin.

Single pool of capacity and performance

The concept of an independent storage container acting as a layer between the array and the volume is not new. Nimble has implemented this well-known concept in the form of storage pools. Each host can be joined in a storage pool (and one pool only), and a storage pool can consist of multiple storage arrays. With the current limitation of 4 arrays per group, this means 4 pools per group, too.

This gives us the flexibility to isolate an array into a single pool to isolate that array’s performance and capacity. We can also add multiple arrays into a single pool for consolidation of capacity and performance. In the latter, volume data will be fanned out across all arrays. Each volume carved out of a multi-array pool is able to provide the performance of multiple arrays combined.

A note on performance load balancing

Nimble pools don’t do (sub-)volume tiering because the CASL architecture relies on the compute and flash for performance features (such as read caching, write coalescing and compression, etc). In other words: hot blocks in any given volume in a pool (on any given array) are stored in the read cache for performance anyway. The system will figure out which blocks are hot enough to include in the cache. The SSD flash isn’t a separate tier, but a cache only.

You can exercise some influence on the performance of a volume:

SSD caching can be enabled on a per-volume basis, allowing a fine grain configuration for what you deem worthy of consuming cache space.

As “cache worthy” data is identified, it is collected into a smaller sequential stripe the size of an SSD’s page and dumped into one of the flash drives. Using a First In, First Out (FIFO) log structure, data is eventually evicted as it grows cold. When a cache miss occurs, the system pulls the data from the NL-SAS disks and performs a read-ahead and prefetch.

Data that still resides in the NVRAM or DRAM (via a “shadow” cache) can be read, bypassing the need to chat with the SSD layer.

So while there is caching in a single array, there is no performance load balancing across arrays. This means that, depending on the distribution of data in a volume across the arrays, a single volume might not be getting all-array performance since too many hot blocks live on one array (and not the other).

Multipath I/O

Again, this is very similar to what we’ve seen in other storage systems. No fancy mumbo-jumbo or proprietary client protocols. It’s p(l)ain ol’ iSCSI with a host-side MPIO driver for parallel array access for load balancing and high availability. This Nimble Connection Manager creates and optimizes paths to the storage arrays. Nimble has provided this plugin for both Windows and vSphere (ESXi). To efficiently use multiple arrays in a pool, this plugin is required on any host connecting to the pool to utilize the parallel access method.

Parallel and direct access guarantees best performance and avoids forwarding of foreign requests. This however requires the Nimble-assisted MPIO path selection policy, not unlike the Multipathing Extension Module for EqualLogic environments. Now what this basically does is share the slice map with the host, so it can make the correct decision on where to send I/O. This removes the need for special interconnects between the arrays and avoids path thrashing on the controllers.

Future Scale

There are a couple of obvious ‘next steps’:

  • Remove the 4-array / 4-pool limit
  • Add performance load balancing on top of capacity load balancing by implementing heat maps (hot vs. cold tag) for blocks to prevent a single controller bottleneck.
  • Add relative capacity / performance tags to each array (and specifically, for each compute, flash and disk shelf component individually) to aid in performance load balancing
  • Implement site-local replication for data redundancy (maybe comparable to LeftHand’s Network RAID) to be able to sustain array failure with the group. A failure in the current architecture will cause the pool to go offline with all volumes along with it.
  • Per-VM Snapshots are not included in this release; they will be added in the future, probably leveraging vVol support
  • Nice quote from Hans, I couldn’t have phrased it better myself

    I do want to give Nimble a challenge for the future: NetApp has of their one-size-fits-all until the market really got fed up with it and in the end they lost the battle because you simply cannot keep beating a dead horse. For Nimble that dead horse in the long run could be iSCSI. iSCSI simply doesn’t scale to dozens like Nimble is doing it today. They will either have to chose not to distribute their volumes over ALL nodes or they will have to step away from iSCSI as such. But that’s not for today! For today this solution with up to 4 dual controller NimbleStorage boxes fits perfect for their target audience.


Nimble’s Scale to Fit scaling supports both scale-up and scale-out. This mix ‘n’ match approach is fairly new in their iSCSI market segment, and I believe Nimble Storage has struck the nail on it’s head. Their CASL architecture is a very interesting and promising approach to delivering performance from a single array. The scale-out approach, leveraging existing concepts and features, is a sound investment in the future given their target audience. It’s a flexible, clean architecture, and it’s going to be very interesting to see how Nimble is going to ramp up their scale-out architecture.

However, it’s not just their storage architecture that sparked my interest. Their InfoSight product is what strikes me as the real differentiator and absolutely unique in this space. Please stay tuned for a separate post on Nimble’s InfoSight product!