Talk to Aerospike and they will tell you “yes, and fairly soon.” Here’s the skinny.
Spinning disk, or hard disk drive (HDD), has been under threat for quite a while with the cost of solid-state drive (SSD) falling at about 50% per annum. But in the past year or so, the price of SSD – or flash memory devices – has fallen faster and it is becoming disruptive. Last spring IBM launched FlashSystem, a line of SSD products, which cost about $10 per gigabyte. By comparison, high performance HDD comes in at about $6 per gigabyte, but there is a little more to the comparison than that.
You tend not to use all of an HDD, but you can and are eventually likely to fill an SSD to capacity – so SSD has greater capacity than the raw store capacity figures suggest. HDD consumes more power, takes more data center space and requires more data center labor. SSD is faster to read and write than high performance HDD, and that matters for many applications, but there is no easy comparison to make here – it is very contextual. It’s also a changing picture, mainly because SSDs are on an accelerating technology curve.
In any event, IBM is convinced that the tipping point has arrived where SSD has become a better choice than HDD for Tier 1 storage and that SSD will gradually become the dominant form of storage in the data center. It is also investing $1 billion in research in this area, which counts as a major strategic initiative. Note also that IBM has a history of making savvy bets in the hardware area. It tends to be ahead of the curve.
There is another factor to think about here. Moore’s Law’s impact has been dramatic, as we are all well aware, on processor speeds, memory speeds, networking speeds, switch speeds, controller speeds and pretty much everything in the data center except HDD. In the last 10 years HDD speed has not kept pace. The consequence is that disk I/O has increasingly become the most common bottleneck, and that is a particularly inconvenient problem in the era of big data.
SSD technology is on the Moore’s Law curve, for the next 10 years at least, and thus it will likely leave HDD in the dust. The speed advantage will soon become extreme. As a rough rule of thumb, Moore’s Law speeds up a technology to which it applies by a factor of 10 every six years of so. Hence, wave goodbye to spinning disk.
Of course, in one respect, this may not be entirely delightful news for the data center. This of it like this: pretty much all the applications that run in the data center were built for spinning disk. Spinning disk is a big industry – it has been for decades. Currently its run rate is about $20 billion per year, and to a very significant extent, data centers are organized around the nature and the foibles of spinning disk. Think SANs, think NAS, think back-up and archive procedures, etc.
You cannot just slip SSD into this environment and expect to port all those mission critical applications onto SSD with the flick of a switch. There is work to do. And by the way, what about all that database technology that’s deployed? Very few databases were built for SSD, and even fewer exploit the possibilities of SSD.
Which brings me back to Aerospike. Aerospike is a database company that specializes in truly high performance database. Currently it is used mainly by big companies that need real-time OLTP performance in the region of 2 to 3 milliseconds per transaction with thousands of transactions per second, although it is also used for some BI applications that have the need for speed.
The company has been smart enough to “bet the farm” on SSD, and for good technology reasons. The Aerospike database has been engineered specifically to run on a combination of DRAM and flash.
When we say “engineered,” we mean it. It doesn’t use the Linux file system directly, because that, like many other technologies, was built for spinning disk. Instead, it implements its own log structured file system. It optimizes for the way that flash works. It uses small block reads and large block writes for speed. It keeps indexes in DRAM to reduce wear and tear on flash memory and runs in a parallel manner across multiple SSDs, which significantly improves throughput. As a consequence, it is lightning fast.
There is a game change in progress. The sun is going down on spinning disk and it is rising on SSD. Both Aerospike and IBM look to be well positioned to take advantage.