andygrove opened a new pull request, #2075:
URL: https://github.com/apache/datafusion-ballista/pull/2075
# Which issue does this PR close?
Closes #2074.
# Rationale for this change
Benchmark numbers currently circulate through issues and PR descriptions
with their conditions attached informally, which makes it hard to tell whether
any two numbers are comparable. Results from the static planner and the
adaptive (AQE) planner are especially easy to conflate, even though they come
from different planners with materially different join behaviour.
`benchmarks/README.md` explains how to generate TPC-H data and run the
benchmark locally, but not what cluster the published numbers came from, how to
reproduce one, how Spark and Comet were configured for a cross-engine
comparison, or which known issues to keep in mind when a query looks slow.
This adds one place that answers those questions and records the current
result set.
# What changes are included in this PR?
A new `docs/source/contributors-guide/benchmarking.md`, wired into the
Contributors Guide nav in `docs/source/index.rst`:
- **What we measure, and why** — work balance across executor tasks, and why
AQE on and AQE off are always reported together.
- **Environment** — the results currently come from a small homelab cluster
with node-local data; the intent is to move to AWS with data in S3 once the
numbers here are good. Stated explicitly so the numbers are read as relative
comparisons on controlled hardware rather than absolute cloud throughput.
- **Reference cluster** — the shape behind the numbers, and the reasoning
for the parts that are easy to get wrong: one executor per node, node-local
data rather than object storage, and headroom between the memory pool and the
container limit.
- **Queries** — all engines run the SQLBench-H query set from
`apache/datafusion-benchmarks`, which is what makes the comparison
apples-to-apples.
- **Running the benchmark** — self-contained commands for Ballista, vanilla
Spark, and Comet, so a result can be reproduced without repo-specific tooling.
- **Why compare against Comet** — Comet and Ballista execute DataFusion
physical plans using the same operators, so a gap points at distribution,
scheduling, shuffle, and memory rather than operator speed. Includes the
caveats that stop it being an operator-level A/B: Spark's optimizer and Spark's
AQE produce the plan Comet accelerates, Comet falls back to the JVM for
unsupported operators, and the two use different distribution models.
- **Results** — a single current result set pinned to the exact commit, with
Spark, Comet, Ballista AQE off, and Ballista AQE on side by side, plus
conventions for recording a result.
- **Known issues found by benchmarking** — brief orientation for reading a
result, linking #1359, #2025, and #2063.
Draft: the result table is being filled in as runs complete. Q18 at SF1000
is measured (Spark 458.0s, Comet 225.5s, Ballista AQE on 750.5s, all returning
100 rows); the full Spark and Comet suites are running now and the remaining
rows are marked TBD. The prose is ready for review in the meantime.
# Are there any user-facing changes?
No code changes. Adds a documentation page to the Contributors Guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
