phillipleblanc commented on code in PR #2075: URL: https://github.com/apache/datafusion-ballista/pull/2075#discussion_r3599873918
########## docs/source/contributors-guide/benchmarking.md: ########## @@ -0,0 +1,328 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Benchmarking + +This page describes how Ballista is benchmarked at scale, and records the current +results. It is aimed at contributors who want to reproduce a number, understand why +a comparison is set up the way it is, or add a result of their own. + +The benchmark used here is derived from TPC-H. For running TPC-H locally at small +scale factors, see [`benchmarks/README.md`][bench-readme] in the repository; this +page covers the multi-node, large-scale-factor setup and the cross-engine +comparisons. + +[bench-readme]: https://github.com/apache/datafusion-ballista/blob/main/benchmarks/README.md + +## What we measure, and why + +Ballista's central performance question is **how evenly work is spread across +executor tasks**. A distributed query is only as fast as its slowest stage, and a +stage is only as fast as its slowest task, so a query whose work concentrates onto +a few partitions will underperform no matter how fast the underlying operators are. +Benchmarks are therefore run at a scale where that imbalance actually shows up. + +Two configurations are always measured, because they select **different planners** +with materially different join behaviour: + +- **AQE off** (`ballista.planner.adaptive.enabled=false`, the default) — the static + `DefaultDistributedPlanner`. +- **AQE on** (`ballista.planner.adaptive.enabled=true`, experimental) — the adaptive + planner, which can re-plan stages using runtime statistics. + +A change that helps one planner can be a no-op or a regression under the other, so +results are reported for both rather than for whichever looks better. + +## Environment + +The results on this page currently come from a **small homelab cluster**: two +bare-metal nodes running Kubernetes, with the TPC-H data staged on node-local disk. +This is a deliberate starting point rather than a final destination. A two-node +cluster with local disk removes as many confounds as possible — no object-store +latency, no cloud network variance, no noisy neighbours — so that when a query is +slow, the cause is Ballista and not the environment. + +The intent is to **move these benchmarks to AWS with data in S3** once the results +here are good. That environment is the one users actually run, and it exercises +things a homelab cannot: object-store reads instead of local disk, higher and more +variable network latency between executors, and larger executor counts. Some of +Ballista's behaviour is expected to change there — a shuffle that is cheap over a +local link is not cheap over a cloud network, and object-store reads make scan +parallelism matter differently. + +So results on this page should be read as **relative comparisons on controlled +hardware**, useful for "did this change help", not as absolute throughput numbers +for a cloud deployment. + +## Reference cluster + +All results on this page use the following shape. It is a reference point, not a +requirement — the commands below work on any cluster, but numbers are only +comparable when the shape matches. + +| | | +| ------------------- | ------------------------------------------ | +| Executors | 2, one per physical node | +| Per executor | 8 cores, 56 GiB, `--memory-pool-size=48GB` | +| Per task slot | 8 concurrent tasks → 6 GB pool each | +| Scheduler | 1 | +| Data | TPC-H SF1000 Parquet, node-local disk | Review Comment: May also be useful to specify how this data is generated (i.e. via the tpchgen-rs, partitioned, etc) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
