Weijun-H opened a new pull request, #12291:
URL: https://github.com/apache/arrow-rs/pull/12291

   # Which issue does this PR close?
   
   Closes #11384.
   
   # Rationale for this change
   
   Callers can sample Arrow batches before opening a writer and disable 
dictionaries when the sampled first page would not benefit from them, without 
changing writer defaults.
   
   # What changes are included in this PR?
   
   Adds `choose_dictionary_encoding` and `DictionaryEncodingOptions` under 
`parquet::arrow::arrow_writer`. The helper returns adjusted `WriterProperties`, 
preserves other properties, and never enables disabled dictionaries. The first 
page-size check is configurable. For nonempty samples, Parquet v1 fixed-length 
byte-array dictionaries are disabled to match parquet-java, even though the 
Rust writer supports them.
   
   Adapted from [Comet's 
DictionaryChooser](https://github.com/apache/datafusion-comet/blob/916103f21a2f959e9bb70a25896df6a46cbbd746/native/core/src/execution/operators/iceberg_dictionary.rs).
 This approximates parquet-java's first-page decision: page boundaries and 
mixed-NaN behavior can differ. It targets the inferred Parquet schema, not a 
custom physical schema. Sampling and buffering remain caller-controlled; small 
samples can reject dictionaries useful over a full row group.
   
   # Are these changes tested?
   
   The 23 unit tests and public example cover parquet-java decisions recorded 
through Iceberg (ported from Comet's tests), physical conversions, column 
limits, first-check configuration, nested row boundaries and slices, encoded 
Arrow arrays, floating-point keys, property preservation, and dictionary-page 
metadata with complete file readback.
   
   # Are there any significant user-facing changes?
   
   New opt-in API only. Existing writer behavior is unchanged.
   
   # AI usage disclosure
   
   AI tools adapted the reference implementation and generated the helper, 
tests, and documentation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to