The canonical workload simulates a common workload on typical hardware, as defined in the methodology section. The canonical workload reveals the performance of a database under a mixed workload, and provides a one-stop figure for overall database performance under typical conditions. Both MySQL and Membase provide stable performance at around 200k-300k queries per second. RethinkDB starts at more than 1.2M qps and settles into a stable throughput of about 900k qps after ten minutes.

This benchmark runs a canonical workload (as defined in the methodology section) under low memory conditions to simulate significant disk I/O. This benchmark provides insights into how different database systems behave when performance becomes almost entirely I/O bound. RethinkDB shows a ~5.5 factor improvement over MySQL. Controlling the cache size is not possible in Membase, so were unable to collect meaningful results for Membase's performance under these conditions.

In this benchmark, we test the database with a read-only workload. During the benchmark, data gets randomly selected from a database containing 50 million keys, each with a corresponding value size of 8-128 bytes. Due to the large cache size used in this benchmark (32GB), most of the data is loaded into memory within roughly the first 30 seconds. From that point on, the in-memory index structures and query processing capabilities determine the throughput actually available to the user. Both MySQL and RethinkDB provide high throughput rates of more than 1200k queries per second, with a steady throughput of 1500k qps for RethinkDB. Membase on the other hand is unable to achieve more than about 250k qps under this workload. Please note that (as in all benchmarks) we use prepared statements for MySQL to minimize the overhead of SQL query parsing. Both RethinkDB and Membase are accessed through the Memcached text protocol, which comes with comparably low parsing overhead out of the box.

In this benchmark, we test the database with an insert-only workload. We continuously insert keys of sizes between 8 to 16 bytes with corresponding values of sizes between 8 and 128 bytes via a large number of concurrent connections. Both Membase and MySQL achieve a steady insert throughput of 100k - 150k queries per second. RethinkDB maintains an insertion rate of more than 500k keys per second for the first 750 seconds.

The eventual drop in RethinkDB's performance is due to the fact that the in-memory cache gets almost entirely filled with dirty pages, and disk throughput saturation is reached. The next version of RethinkDB will address this problem by making better use of locality (two random small modifications should result in a single small write, not two full block writes).

Running in strong durability mode guarantees that whenever a write operation returns a success status to the client, the change has actually been committed to non-volatile memory. Due to limitations in RethinkDB's current implementation and its transaction log-free architecture, RethinkDB is unable to provide performance figures on-par with MySQL for this workload. The next version will address this issue and provide significantly higher performance under strong durability guarantees.

The large values benchmark tests database performance for workloads involving values of sizes between 4KB and 64KB (evenly distributed across sizes in this range). Both Membase and RethinkDB provide a mostly stable throughput of around 10k queries per second. The throughput is limited due to network bandwidth saturation in our setup. However, this benchmark reveals a performance problem of MySQL for this workload, which only provides a steady throughput of around 4k qps.

This benchmark runs a canonical workload under high cache pressure with the backing store located on rotational hard drives. RethinkDB demonstrates a slightly higher throughput than MySQL, at the expense of higher latency.