NikitaMatskevich opened a new pull request, #11892: URL: https://github.com/apache/arrow-rs/pull/11892
# Which issue does this PR close? Closes #11891. # Rationale for this change DataFusion Comet needs to reproduce selected parquet-java data structures when accelerating Iceberg writes. Automatic folding changes sparse Bloom filters' bytes and realized false-positive rates even when the initial sizing and inserted values agree. Let applications retain the initial allocation when matching the non-adaptive Java writer matters. # What changes are included in this PR? Add `BloomFilterPropertiesBuilder::with_folding_enabled(bool)` and a corresponding accessor, defaulting to true. Existing global and per-column property setters carry this setting to both the generic encoder and Arrow's byte-array encoder. Document the file-size/pruning tradeoff and the limits of byte-identity guarantees. # Are these changes tested? The new writer regression writes INT32 and UTF8 columns with dictionary encoding on and off. It verifies that the global default folds, a per-column override retains the original serialized filter exactly, and both retain every inserted value. Properties tests cover the default and column override. All 1,462 Parquet library tests pass with Arrow enabled. A separate probe compared all 2,048 INT32 filter-bitset bytes with parquet-java 1.16.0 and found them identical before folding. Formatting passes. Strict Clippy on installed Rust 1.97.1 is blocked by existing `from_iter_instead_of_collect` errors in arrow-data, outside this change. The repository pins Rust 1.99.1, which is not installed locally. Fork Actions have not been enabled; local validation results are reported here. # Are there any significant user-facing changes? This is an additive API. Existing defaults and folding behavior are unchanged. Disabling folding does not by itself guarantee byte identity: initial sizing, physical encoding, boolean insertion, and other writer settings also need to agree. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
