andygrove opened a new pull request, #2148:
URL: https://github.com/apache/datafusion-ballista/pull/2148
# Which issue does this PR close?
N/A
# Rationale for this change
The TPC-DS correctness gate has been running at SF1. At that size several
tables fit in a single batch, so the suite mostly exercises the planner and
barely touches the distributed execution paths that actually break: shuffle
writes/reads across multiple partitions, repartitioning, and spilling.
Running the same 99-query suite at SF10 keeps the exact same coverage in
terms
of queries but puts real data volume behind it, which is where distributed
result divergence tends to show up.
# What changes are included in this PR?
- `.github/workflows/tpcds.yml`: generate the data with `--scale-factor 10`
instead of `1`, and rename the workflow, job, and log artifact to match.
- `benchmarks/tpcds-gen.sh`: default `SCALE_FACTOR` to `10` so a local run
reproduces CI without an env override. It is still overridable.
- `benchmarks/README.md`: update the SF references in the data generation and
CI sections.
No change to the query set or the skip list.
# Are there any user-facing changes?
No. This only affects CI and the benchmark data generation script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
