Measuring the scalability of SQL and NoSQL systems

YCSB is a new benchmark for Cloud Serving Systems developed by a team of researchers at Yahoo! Research Silicon Valley lead by Raghu Ramakrishnan.

"Over the last few years, we observed an explosive rise in the number of new large-scale distributed data management systems. This trend created a lot of excitement throughout the community of web application developers and among data management developers and researchers. But it also created a lot of debate. Which systems are most stable and mature? Which have the best performance? Which is best for my use case? We saw these questions being asked in the community, but also within Yahoo! " say Silberstein and Ramakrishnan.

The YCSB benchmark, available as open source, helps companies measuring the scalability of SQL and NoSQL systems.

The initial YCBS benchmark measured four different systems: Cassandra, HBase, PNUTS (Yahoo! cloud system) and an implementation of a sharded MySQL.

"The big selling point of cloud serving systems is their ability to scale upward by adding more servers. In our initial work we observed systems doing well on scalability, but having erratic behavior when we added capacity online and increased workloads (i.e., they were often weak on elasticity) " remark Silberstein and Ramakrishnan.

Read more.

Paper: Benchmarking Cloud Serving Systems with YCSB. (.pdf), by Adam Silberstein, Brian F. Cooper, Raghu Ramakrishnan, Russell Sears, Erwin Tam, Yahoo! Research.
1st ACM Symposium on Cloud Computing, ACM, Indianapolis, IN, USA (June 10-11, 2010)