andygrove commented on issue #1596: URL: https://github.com/apache/datafusion-ballista/issues/1596#issuecomment-5004201967
Update: **TPC-H at 1 TB now runs end to end.** SF1000 completes all 22 queries under both planners, so the "unable to get this working" premise no longer holds. Measured on `main` @ `26b29391` — 2 executors x (8 cores / 56 GiB), `--memory-pool-size=48GB`, one executor per node, SF1000 Parquet on node-local disk, `target_partitions=32`, `prefer_hash_join=false`, dynamic-filter pushdown off, 1 iteration: | Engine | Total | Completed | |---|---:|---| | Comet 0.17.0 | 2738.5 s | 22/22 | | vanilla Spark 3.5.3 | 4687.9 s | 22/22 | | Ballista (AQE on) | 6320.7 s | 22/22 | | Ballista (AQE off) | 6378.2 s | 22/22 | Per-query figures, method, and the exact commit: https://datafusion.apache.org/ballista/contributors-guide/benchmarking.html **Leaving this open, because neither of the things it actually asks for is done:** 1. **Not EKS, and not S3.** These numbers come from a two-node homelab cluster reading node-local disk. That was deliberate — it removes object-store latency and cloud network variance so a slow query points at Ballista rather than the environment — but it is not the environment this issue is about. Moving to AWS with data in S3 is the stated next step on the benchmarking page. Object-store reads and higher inter-executor latency should be expected to change the picture. 2. **Ballista does not match Spark.** The bar set here was "Ballista should at least match Spark performance". At SF1000 Ballista is **1.36x slower than vanilla Spark** and **2.33x slower than Comet**. It wins the light queries (Q1 70.8 s vs Spark's 444.8 s) and loses the join-heavy ones (Q21 1083.0 vs 536.4, Q9 891.8 vs 570.2). So the useful reframing is that this is no longer "unable to run at scale" but "runs at scale, slower than Spark on join-heavy queries" -- which is a measurable target rather than a blocker. Related: #2024 (adaptive planner regresses join-heavy queries at SF1000), #2025 (Q18 hash-join build side exhausts the memory pool). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
