andygrove opened a new pull request, #2855:
URL: https://github.com/apache/datafusion-ballista/pull/2855
# Which issue does this PR close?
N/A
# Rationale for this change
The benchmarking guide shows how fast Ballista is, but not how much memory
it needs, which is what a cluster is sized and paid for. This adds a page that
answers that for one memory-intensive query, against Spark and Spark + Comet,
and shows how each engine's time changes as executor memory shrinks.
# What changes are included in this PR?
**A Memory usage page** in the benchmarking guide
(`docs/source/contributors-guide/benchmarking/memory.md`). It covers TPC-H q18
at SF1000 with 2 executors of 16 CPUs. Each configuration was run at decreasing
executor memory (the pod's memory limit) until it failed, then bisected to
within 2 GiB:
| Engine | Smallest executor memory | q18 time there | q18 time at 64 GiB |
| --- | ---: | ---: | ---: |
| Ballista (default settings) | 8 GiB | 132 s | 133 s |
| Spark, 50% off-heap | 12 GiB | 387 s | 330 s |
| Spark + Comet, 75% off-heap | 20 GiB | 150 s | 145 s |
"Smallest" means the smallest size at which the query finished with no
executor killed for running out of memory. The page also covers:
- Seven configurations in all, including Spark on-heap, which needed 18 GiB,
and other heap/off-heap/overhead splits for Spark and Comet.
- What limited each engine:
- Spark's floor was set by `memoryOverhead`, not the heap.
- Every Comet failure was an executor going over its pod limit, never an
error from Comet's memory pool.
- That Ballista did not fail cleanly below its smallest size: executors were
OOM-killed and restarted, and at 5 and 4 GiB the query then stopped making
progress instead of erroring.
- How to reproduce any point with `run.py`.
**Results as data.** The sweep is in
`benchmarks/cluster/published/memory/tpch-q18-sf1000.json`. `publish.py` now
generates the page's summary table, its table of all 47 runs and two charts
(`tpch_q18_memory_smallest.svg` and `tpch_q18_memory_time.svg`) from it, the
same way it already generates the TPC-H results. The new charts use the
existing chart styles, Spark grey and Ballista blue as in the README charts,
with light and dark variants. `test_publish.py` covers the new functions, and
its existing up-to-date check now includes the new outputs.
These runs were not on the AWS cluster behind the published TPC-H results.
They were on a two-server Kubernetes cluster with the Parquet files on local
disk, so the page says its times are not comparable with that table. Every
point is a single run of `run.py` with `--executor-memory-gib` or
`--spark-conf` memory settings; reading local data was the only addition.
# Are there any user-facing changes?
Documentation only, plus new generated outputs in `publish.py`. No code, API
or wire-format changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
