andygrove opened a new issue, #5096: URL: https://github.com/apache/datafusion-comet/issues/5096
### What is the problem the feature request solves? Two dictionary-handling paths hand-roll what arrow's cast kernel supports natively: - `native/spark-expr/src/conversion_funcs/cast.rs` `dict_from_values` (explicitly "copied from datafusion common scalar/mod.rs") manufactures a degenerate dictionary with one key per row and no deduplication, then the cast recurses into it. - `native/core/src/parquet/parquet_support.rs` `parquet_convert_array` has a branch that manually unpacks `Dictionary(Int32, Utf8/LargeUtf8)` via recursive value conversion plus `take`, or rebuilds a dictionary by hand. ### Describe the potential solution - In `cast.rs`, invert the order: Spark-cast the plain array to the dictionary's value type first, then `arrow::compute::cast` to `Dictionary(key, value)`. Arrow's `cast_to_dictionary` handles key creation and dedups values via dictionary builders. The Spark-owned semantics stay in the value cast. - In `parquet_support.rs`, the dictionary branch can likely be deleted: since the value type is restricted to Utf8/LargeUtf8, the recursion can only hit identity or the generic `can_cast_types` fallthrough, and arrow's `cast_with_options` natively supports both `Dictionary -> T` (unpack via take) and `Dictionary -> Dictionary` (cast values). ### Additional context - Output representation differs in `cast.rs` (deduplicated dictionary instead of one-entry-per-row), which is logically the same array; downstream consumers treat dictionaries generically. - Before deleting the parquet branch, verify `can_cast_types(Dictionary(..), to)` covers the same combinations, since unsupported combos currently fall through to returning the original array. Found during an audit of native code that replicates existing arrow-rs kernels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
