NoahKusaba opened a new pull request, #2478: URL: https://github.com/apache/datafusion-ballista/pull/2478
# Rationale for this change Many pages had drifted from the code: wrong defaults, commands that fail as written, and missing endpoints and features. Every correction here was checked against the source tree. # What changes are included in this PR? **Defaults and behaviour** - AQE was documented as experimental and off by default in the tuning guide, the architecture guide and the chaos-testing README. `ballista.planner.adaptive.enabled` defaults to `true`. - The scheduler policy default is `push-staged`, not `pull-staged`. The event-loop buffer default is 1000, not 10000. - The AQE optimization lists now include build-side staging (#2434), shuffle-partition coalescing and parallel windows. Coalescing is dropped from the known gaps, since it is implemented (#1987). - New 55.0.0 upgrade entry for build-side staging, which changes query plans. **Commands that fail as written** - `ballista-cli` has no `spark-compat` feature, but two guides told users to install and build with it. - Every guide that starts a second executor moved only `--bind-port`, colliding with the first executor's gRPC and health ports. - The CLI usage blocks were clap 3 output, missing `--vcores`, `--color` and `--tui`. They are regenerated from the built binary. - The benchmarks README used `$ARROW_HOME` paths, a `--bin executor` that does not exist, and an opt-in `mimalloc` feature that is already on by default. **Reference tables** - The scheduler REST table was missing five endpoints and both health probes (`/healthz` and `/readyz`). `rest-api` is on by default. - The Prometheus metric list, and the `PrometheusMetricsCollector` rustdoc, named a `failed` metric that is not exported. It is registered as `job_failed_total`. - The feature tables now include `utoipa` (#2397) and the CLI's `cli` and `web` features. - A rustdoc comment in the scheduler config named a `consistent-hash` task distribution that does not exist. **Smaller fixes** - The license badge pointed at `datafusion-comet`, and the adopter rows referenced images that are not in the repo. - Version strings in log samples, the dev-container prerequisites, a reversed git range in the release README, `start-slave.sh` renamed to `start-worker.sh` in the Spark benchmark README, and assorted typos. # Are there any user-facing changes? Documentation only. Two `.rs` files change rustdoc comments only (`scheduler/src/config.rs`, `scheduler/src/metrics/prometheus.rs`). Verified: `sphinx-build -W` is clean when combined with #2476, and `cargo check -p ballista-scheduler --all-features` passes. Split out of #2438 to make it easier to review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
