NikitaMatskevich opened a new issue, #11891: URL: https://github.com/apache/arrow-rs/issues/11891
### Problem Apache DataFusion Comet's native Iceberg writer needs selected Rust-produced Parquet data structures to be byte-identical to the corresponding Java-produced structures. In particular, Bloom-filter output should be reproducible against Iceberg's non-adaptive parquet-java writer when initial allocation, physical value encoding, and inserted values agree. parquet-rs automatically folds sparse split-block Bloom filters after insertion. parquet-java's non-adaptive path retains the initial allocation. For example, NDV=1000 and FPP=0.01 initially allocate 2048 bytes, but writing only 100 distinct INT32 values allows parquet-rs to fold the filter. Both filters preserve inserted-value membership, but their serialized bytes, realized false-positive rates, and pruning behavior differ. Sparse row groups occur with skew or a partially filled final row group even when table-level estimates are reasonable. Comet currently needs a separate opt-in for this behavior: https://github.com/apache/datafusion-comet/pull/5724 . ### Proposed solution Add an option to disable post-insert Bloom-filter folding, while retaining folding by default. The option should work globally and per column through `BloomFilterProperties`, including both the generic value encoder and Arrow's specialized byte-array encoder. More generally, it is desirable to retain compatibility toggles in parquet-rs that let applications select parquet-java behavior when reproducible output is required. This need does not imply changing parquet-rs defaults or removing useful optimizations. ### Scope and limits Disabling folding removes one source of divergence. It does not alone guarantee byte identity: initial sizing/caps, physical type encoding (including decimals), boolean insertion, dictionary-only filter suppression, and other writer settings also need to agree. The proposed toggle should document that distinction. Related API work: #9667. Related Java-compatibility option: #10962. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
