Abdallah-Afifi opened a new pull request, #11004: URL: https://github.com/apache/arrow-rs/pull/11004
# Which issue does this PR close? - Closes #10644. # Rationale for this change When a column chunk's first data page has no values, `build_preset` returns the fallback pair (exponent 0, factor 0). `flush_buffer` cached that for the whole chunk, and `select_params` short-circuits on a single-candidate preset, so every later page was stuck on an integer scale and every fractional value became an exception. It's reachable from the normal write path, not just the encoder API: the column writer counts levels rather than values, so a first page of all nulls flushes an empty buffer. Encoding 3000 values of `i * 0.01`: | | bytes | | --- | --- | | as the first page | 3,808 | | raw f64, no encoding | 24,000 | | after an empty first page | 31,258 | The chunk ends up bigger than leaving it unencoded. # What changes are included in this PR? `flush_buffer` only caches the preset when the page had values. `put` already branches on whether the preset is set, so the next page with values builds it. The issue asks only for regression coverage and @alamb suggested the fix as a follow-up — this has both. Happy to split them if you'd rather. # Are these changes tested? `test_empty_first_page_does_not_poison_preset`, which fails on main with the numbers above. Full `cargo test -p parquet` passes (1558 tests); fmt and clippy clean. One thing I'd like a second opinion on: with the preset left unset, an all-null first page no longer arms the streaming path, so the second page takes the buffered path. The tests agree that's correct, but it's the part of this I'm least certain about. --- Per the AI policy in CONTRIBUTING.md: I used Claude Code to trace the encoder and draft the fix and test. I've reviewed all of it and verified the behaviour myself — the numbers above are from running the test locally, before and after. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
