How does TPC-DS benchmark big data systems?

The TPC-DS benchmark evaluates big data systems by simulating decision support workloads, which involve complex queries and large-scale data processing. It models a retail company’s data ecosystem, including sales, inventory, and customer interactions, to test how well a system handles analytical tasks. The benchmark combines a realistic schema, diverse query patterns, and concurrent user simulations to measure performance metrics like query response time, throughput, and scalability. This makes it a comprehensive tool for assessing systems designed for data warehousing, business intelligence, or advanced analytics.

TPC-DS uses a star schema with fact tables (e.g., sales, returns) and dimension tables (e.g., customers, products) to replicate real-world data relationships. It includes 99 SQL queries covering operations like joins, aggregations, and window functions, designed to stress-test different aspects of a system. For example, Query 19 involves multi-table joins and large aggregations to analyze sales trends, while Query 72 uses subqueries and correlated aggregates to assess customer behavior. Additionally, the benchmark incorporates data maintenance tasks, such as loading new data or updating existing records, to evaluate how systems handle ETL (extract, transform, load) workflows alongside analytical workloads. Concurrency is tested by simulating multiple users submitting queries simultaneously, ensuring the system can scale under realistic demand.

Developers use TPC-DS to compare systems like Hadoop, Spark, or cloud-based data warehouses by running the benchmark and measuring metrics like total execution time or queries per hour. For instance, a team might test a Spark cluster’s performance by running all 99 queries against a 10 TB dataset and comparing results to a competing system. The benchmark’s standardized data generation tool ensures consistency, allowing fair comparisons across platforms. While TPC-DS doesn’t mandate specific hardware configurations, it provides guidelines for reproducible testing, helping teams optimize hardware, software, or query execution plans. By focusing on real-world scenarios, it helps identify bottlenecks—such as slow join operations or resource contention—and validates improvements in query optimizers or storage layers.