Know Your Storage Constraints: IOPS and Throughput

Application performance can often hinge on how well your storage can serve data to end clients. For this reason you must correctly design or choose your storage tier speed in terms of both IOPS and throughput, which rate the speed and bandwidth of the storage respectively.

Introduction to IOPS

IOPS stands for input/output operations per second. It’s a measurement of performance for hard drives (HDDs or SSDs) and storage area networks. IOPS represents how quickly a given storage device or medium can read and write commands in every second.

The “back end” IOPS relates to the physical constraints of the media itself, and equals 1000 milliseconds / (Average Seek Time + Average Latency), with the latter two also measured in milliseconds.

Back end IOPS is dependent on the rotational speed of the HD, if applicable (solid state drives do not rotate, while traditional hard drive disks do). The Average Latency in the above formula is the time it takes the disk platter to spin halfway around. It is calculated by dividing 60 seconds by the Rotational Speed of the disk, then dividing that result by 2 and multiplying everything by 1,000 milliseconds. ((60 / RPM)/2)*1,000.

Of course, for solid state drives, the average latency drops significantly, as there is no rotating disk inside. Therefore you can just plug in 0.1 ms as the average latency to account for network traffic between the processor in your server/virtual machine and the storage array or device. More on network issues shortly.

Average Seek Time is the time it takes for the head (the piece that reads data) to reach the area on the disk upon which that data is stored. The head needs to move around the storage area in order to locate the targeted data. You must average both write and write seek times in order to find the average seek time.

Most of these ratings are given to you by the manufacturers. Generally a HDD will have an IOPS range of 55-180, while a SSD will have an IOPS from 3,000 – 40,000.

Different applications require different IOPS and block sizes to function properly. A single application may even have different components that function at different size ranges for blocks. It is vital to check software providers’ recommendations around block size and performance.

Block Sizes

Block loosely translates to any piece of data. File systems write entire blocks of data rather than individual bits and bytes. A file system block can stretch over multiple sectors, which are the physical disk sections. Blocks are abstract representations of the hardware that can (or may not) be a multiple of physical block size. Every file takes up one block no matter its size, so choosing the correct block size to efficiently consume storage can make a big difference when it comes to performance.

For example. SQL server logs in 64 kb blocks, while Windows Server might use 4 kb blocks, and the underlying vSphere hypervisor uses 1 MB blocks.  As you’ll see, the block size and IOPS have a snowballing effect on network traffic throughput.

Small block sizes are preferred for a large volume of smaller files, because you are more efficiently using the storage (remember that even small files must consume an entire block — so a file that is only a couple of bytes will use an entire 4 KB block).

You’ll want to architect storage to properly align according to multiples of the base sectors. For example, four sectors of 4 KB will fit on a block of 16 KB, which fits on a stripe of 64KB nicely. Otherwise you run into additional latency and will have to defragment frequently as your blocks will overlap different blocks and sectors (and perhaps even disks in a storage array) as you read and write. vSphere 5.0 and up will automatically align with a 1 MB block, as mentioned above.

OK, So What About Throughput?

Throughput measures the data transfer rate to and from the storage media in megabytes per second. While your bandwidth is the measurement of the total possible speed of data movement along the network, throughput can be affected by IOPS and packet size. Network protocol can also change the overall throughput. It is a measurement of the ultimate amount of data that actually moves along the network path — while bandwidth is the potential capacity of the path without an extenuating factors.

IOPS can therefore become a bottleneck for throughput. It is likely that your CPU and memory will not be a limiting factor, as they can handle more information per second than most storage devices and networks.

A serially-attached storage device might only have a maximum throughput of up to 2,812 MBps, while Ethernet reaches up to 12,500 MBps and fiber channel networks reach 4,000 MBps.

The size and number of blocks traversing the network can make a big difference in throughput. A 1 MB block will take longer to pass through the network but you may not move as many of them. A 4 KB block will move very quickly, but you’ll likely face a high volume.

With all of this in mind, it is vital to plan according to manufacturer and developer recommendations as well as real-world benchmarks to maximize your storage (and subsequently application) performance. Take a look at peak IOPS and throughput ratings, read/write ratios, RAID penalties, and physical latency. One of our cloud engineers will also be happy to help you select the proper cloud storage tier for each of your applications.