Rich-T-kid opened a new pull request, #11780: URL: https://github.com/apache/arrow-rs/pull/11780
# Which issue does this PR close? - Closes #11748 - Part of apache/datafusion#24111 # Rationale for this change DataFusion's `binary_as_string` and RLE dictionary options need an unannotated Parquet `BYTE_ARRAY` to map directly to `Dictionary<K, Utf8>`. The reader currently chooses the dictionary value type from the Parquet annotation, so it still builds a `Binary` child when the supplied Arrow schema requests `Utf8`. DataFusion then has to expand and rebuild the dictionary (apache/datafusion#26530). # What changes are included in this PR? Decode dictionary values using the value type requested by the supplied Arrow schema. This produces `Dictionary<K, Utf8>` directly, preserves the Parquet keys and shared values across batches, and validates UTF-8 while decoding. # Are these changes tested? Yes. Tests cover binary dictionaries requested as UTF-8 dictionaries across multiple batches and invalid UTF-8 input. # Are there any significant user-facing changes? Supplied dictionary schemas now produce columns with the requested value type. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
