STAC Report: kdb+ tick analytics at scale on GCP

First public cloud solution with publicly disclosed STAC-M3 results outperformed previously tested Lustre-based on-prem solution

25 October 2018

STAC recently performed STAC-M3 benchmarks on a stack consisting of Kx Systems’ kdb+ 3.6 database system distributed across 13 x Google Cloud Platform (GCP) custom instances (32vCPU, 160GB DRAM, Skylake requested), accessing Google Persistent SSD. The solution was subjected to both the baseline (Antuco) and scaling (Kanaga) suites of STAC-M3.

The report is available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database solutions that manage large time series of market data (tick data). STAC-M3 delivers dozens of test results, which are presented through a variety of tables and visualizations in this report. Google chose to highlight the following:

  • These are the first public STAC-M3 results based on a public cloud solution.
  • This solution, composed of "off the shelf" GCP offerings, outperformed a kdb+ solution involving a Lustre-based on-premise cluster (SUT ID KDB150528) in 14 of the 17 required baseline (Antuco) benchmarks (from 1.3x to 7.8x speedup)
  • It also outperformed the Lustre-based solution in 16 of 16 of the scale (Kanaga) benchmarks that were reported for that solution (from 1.6x to 12.6x speedup)*

* KDB150528 operated on only 4 years of data. For that dataset size, the Kanaga suite has 16 benchmarks. The GCP solution operated on 5 years of data, which results in 24 benchmarks.
This GCP solution outperformed a bare metal solution based on Broadwell EX and 6TB DRAM (SUT ID KDB160425) in 8 of the 15 required benchmarks.

Details are in the STAC Reports at the links above. Premium subscribers also have access to the code used in this project and the micro-detailed configuration information for the solution. (To learn about subscription options, please contact us.)

This project follows on the heels of an earlier project in which we tested kdb+ running in memory in a GCP "Ultramem" instance. Google has now disclosed results for all of the dozens of STAC-M3 use cases, on datasets ranging from 3TB in memory to 55TB on Persistent SSD.

About STAC News

Read the latest about research, events, and other important news from STAC.

Subscribe to notifications of research, events, and more.

(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.

Enter your email above, then click "Sign Up" to join the STAC mail list and (optionally) register to access materials on the site. Click for terms.