andygrove opened a new pull request, #2697:
URL: https://github.com/apache/datafusion-ballista/pull/2697
# Which issue does this PR close?
None. #2564 notes on the benchmarking page that the harness behind the
Spark, Spark + Comet and Trino numbers isn't in the repository yet; this adds
it.
# Rationale for this change
Contributors can't reproduce the Spark, Spark + Comet and Trino numbers on
the benchmarking page. This adds `benchmarks/cluster/`, which runs any of the
four engines on your own Kubernetes cluster with the settings those pages
describe. You provide the cluster, the data in S3 and the images. Everything
specific to a cluster is in `cluster.toml`: namespace, service account,
credentials Secret, region, storage class, node selector, tolerations, pull
secrets, data paths and images.
# What changes are included in this PR?
- **`run.py`** (standard library and `kubectl` only) deploys a fresh cluster
for one engine. It runs the queries from a Job, reads the results from the
Job's log, writes `results/<run>.json` and deletes everything it created. By
default every engine gets 32 executor pods of 8 CPUs.
- Ballista: a scheduler, executors and the `tpch`/`tpcds` runner. The
executor flags are the ones on the Ballista page (`--vcores 8`, a pool of 70%
of 64 GiB, `--work-dir` on a volume of its own, 128 MiB gRPC messages).
`--scheduler-policy pull-staged` is passed explicitly because the default is
now push-staged.
- Spark and Comet: a driver Job runs `spark/tpcbench.py` in client mode
and Spark starts the executors. The settings are in `conf/spark.conf` and
`conf/comet.conf`. S3 credentials come only from the environment or the service
account's web identity token: the AWS SDK's default chain falls back to the
node's instance role when it can't use the token, and executors on nodes whose
role can't read the data then fail.
- Trino: a coordinator, workers and `trino/run_queries.py`, which creates
external Hive tables over the Parquet files in a file metastore in S3. TPC-H
only.
- **The same query files on every engine**: `benchmarks/queries` and
`benchmarks/queries-tpcds`. A file with several statements is timed as a whole,
as the Ballista runners time it. For Trino, q15's `create view revenue0 (cols)
as ...` is rewritten to name the columns on a derived table, because Trino's
`CREATE VIEW` takes no column list.
- **Row counts**: the Spark and Trino runners fail a TPC-H query whose row
count at SF1000 differs from `tpch_expected_rows.py`.
- **Dockerfiles**:
- Ballista: builds the scheduler, executor and both runners from one
commit, with its own `.dockerignore`, since the root one leaves out the
scheduler's declared test target.
- Spark: `apache/spark` plus hadoop-aws and the AWS SDK.
- Comet: the Spark image plus a Comet JAR.
- **`compare.py`** prints the Markdown table the benchmarking page uses from
several result files.
- **Unit tests** for `run.py`, the Spark runner and the Trino runner, which
need neither a cluster nor Spark.
The scripts don't generate data: they take a directory of Parquet files per
table, partitioned or not.
# Are there any user-facing changes?
Documentation only: the benchmarking pages now point at `benchmarks/cluster`
instead of saying the harness is missing. The published Spark, Spark + Comet
and Trino numbers predate the scripts, so the pages still say they came from a
harness with its own phrasing of the queries. The scripts run the queries in
this repository on every engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
