Rich-T-kid opened a new pull request, #26973: URL: https://github.com/apache/datafusion/pull/26973
## Which issue does this PR close? - Closes #26972. ## Rationale for this change `enable_rle_to_dictionary` promotes a column when any row group has a dictionary page. If a writer later falls back to plain pages, schema adaptation rebuilds dictionaries per batch. ClickBench Q23 spends most of its projection time in that cast. ## What changes are included in this PR? Dictionary promotion now requires page encoding statistics showing that every data page is dictionary encoded. Missing evidence keeps the plain type. When files disagree, directory schema inference also keeps the plain type instead of promoting the other files. The option documentation is updated to describe the narrower behavior. ## Results A [controlled ClickBench run](https://github.com/apache/datafusion/pull/26964#issuecomment-6545437200) compared `55a2794b` with `4b73f22a`. Both use DataFusion #26816, Arrow #12008, dictionary reads, and a 32 GiB limit; this PR's policy is the only code difference. | Metric | Before | After | | --- | ---: | ---: | | Total query time | 40.69 s | 22.26 s | | Q23 | 9.23 s | 6.12 s | | Peak RSS | 15.8 GiB | 13.4 GiB | | Peak spill | 267.9 MiB | 0 B | Twenty-four queries improved, three regressed, and sixteen were unchanged in the primary summary. Q28 was the only material regression at 1.12x. ## What is the testing strategy for this PR? Unit tests cover dictionary-only, mixed, missing-statistics, and missing-dictionary-page metadata, plus mixed-file schema normalization. Validated with: - `cargo test -p datafusion-datasource-parquet --lib` (296 tests) - `cargo test --test sqllogictests -- information_schema` - `cargo clippy --workspace --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check` - `./dev/update_config_docs.sh` ## Are there any user-facing changes? With `enable_rle_to_dictionary=true`, columns containing plain fallback pages, or lacking page encoding statistics, remain their plain string or binary type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
