High Capacity VSAN Nodes Part 1

Summary

I was recently asked for some help engineering high capacity VSAN nodes for a large storage solution which brought up some issues I thought would be interesting to share. By high capacity, I mean the customer was looking for 400TB with the ability to grow to a petabyte. I found that the standard building blocks didn’t make economic sense for the customer’s ratio of workloads to storage. As a result, we examined the available platforms for greater capacities per node.

The first thing I’d like to examine are the various density trade-offs that can be made in choosing a configuration.

Density

Typically, storage density compares total storage to physical space (often times vertical space in the rack, terabytes per rack unit or TB/U). With VSAN, we have another consideration, the amount of storage addressable per VSAN CPU socket license (TB/license). As VSAN lists at US$2.5K/socket, a solution which is dense on a TB/U basis but not a TB/license basis can inflate the total project budget. Consider two hypothetical servers at maximum configuration:

[table id=1 /]

The first has 12TB of storage in a 1U form factor while the second has 24TB of storage in 2U.

[table id=2 /]

The two servers are equally dense in space at 12TB/U, but the second server has the potential to be twice as dense at 24TB/license (vs. 12TB/license for the first server).

What are we giving up, then? Clearly, this configuration is giving up density measured in a third way, the number of CPUs we can have in the amount of space available (CPU/U).

[table id=3 /]

This becomes relevant when we begin to look at real-world configurations, where almost all 2U rack servers come with at least two CPU sockets which can be populated.

[table id=4 /]

A quick note on the 21 drive count: I’m assuming the typical 2U SFF host which comes with 24 SFF drives. 21 refers to the number of capacity drives assuming three drive groups, each with one SSD cache and seven capacity drives.

At this point we can see another trade-off when choosing fewer, high-capacity, low-RPM drives. We’re getting much fewer IOPS for the underlying capacity layer. 7.2K LFF drives are much less dense (IOPS/TB) than 10K SFF drives. I’ll pull some numbers here from the Wikipedia article on IOPS.

[table id=5 /]

This becomes relevant when we calculate the minimum speed at which data needs to be de-staged from the SSD write cache to the capacity disk layer.

Next time, I’ll work through an example high-capacity application to see the implications of these numbers.

Photo by twicepix