andygrove opened a new issue, #2145: URL: https://github.com/apache/datafusion-ballista/issues/2145
`docs/source/user-guide/configs.md` is hand-maintained, so it drifts from the config keys the code actually reads. Auditing every `BALLISTA_*` constant in `ballista/core/src/config.rs` against `docs/source/` turns up nine keys documented nowhere: - `ballista.planner.coalesce.enabled` - `ballista.planner.coalesce.target_partition_bytes` - `ballista.planner.coalesce.small_partition_factor` - `ballista.planner.coalesce.merged_partition_factor` - `ballista.planner.propagate_empty.enabled` - `ballista.shuffle.compression.codec` - `ballista.shuffle.writer_channel_capacity` - `ballista.testing.chaos_execution.enabled`, `.probability`, `.fault_type`, `.seed` Two more are documented only incidentally, outside the config reference: `ballista.planner.adaptive_join.enabled` appears solely in the 54.0.0 upgrade notes, and `ballista.optimizer.hash_join_max_build_partition_bytes` solely in the benchmarking guide. The AQE settings illustrate the failure mode: they were documented in `tuning-guide.md` next to the prose explaining them, and `configs.md` was never updated or cross-linked. Fixing the current gaps by hand would leave the same problem in place. Since `ballista/core/src/config.rs` already builds a runtime registry (`CONFIG_ENTRIES`) carrying each key's name, description, type, and default, the reference tables can be generated from it instead, with CI failing when a registered key is documented nowhere. The sibling DataFusion repo does this already via `dev/update_config_docs.sh` and a CI check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
