Rich-T-kid opened a new pull request, #11749: URL: https://github.com/apache/arrow-rs/pull/11749
# Which issue does this PR close? - Closes #11748 # Rationale for this change When a Parquet `BYTE_ARRAY` column has no string annotation, the dictionary reader decodes its values as `Binary`. If a supplied Arrow schema requests `Dictionary<K, Utf8>`, the reader currently advertises that schema while returning a `Dictionary<K, Binary>` column. DataFusion hits this when `binary_as_string` is combined with direct Parquet dictionary reads. It has to expand the dictionary to rows and build it again to obtain the requested string type. # What changes are included in this PR? The dictionary reader now builds the array using the physical dictionary value type, then casts only the distinct values to the requested Arrow type and reattaches the existing keys. Converted values are cached while output batches share the same Parquet dictionary page. Binary-to-string conversions continue to reject invalid UTF-8 rather than replacing it with nulls. `arrow-cast` is moved from a Parquet test dependency to an optional dependency of the `arrow` feature. # Are these changes tested? New tests cover reading an unannotated dictionary-encoded Binary column as `Dictionary<Int32, Utf8>` across multiple batches. They verify that the batch schema and actual column type agree, keys are preserved, converted values are shared across batches, and invalid UTF-8 still returns an error. # Are there any significant user-facing changes? Supplied dictionary schemas that require a string/binary value conversion now produce columns with the requested value type. There is no public API change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
