peterxcli opened a new pull request, #5234: URL: https://github.com/apache/datafusion-comet/pull/5234
## Which issue does this PR close? Closes #5096. ## Rationale for this change Comet has two dictionary conversion paths that duplicate Arrow's cast kernels. The expression cast path creates one dictionary value per row, so repeated values are not deduplicated. The Parquet conversion path manually casts dictionary values and applies `take`, and dictionary-to-dictionary conversion can produce a nested dictionary instead of the requested key and value types. Delegating dictionary construction and conversion to Arrow keeps Spark-specific value semantics in Comet while using Arrow's tested key creation, deduplication, unpacking, and dictionary-to-dictionary casts. ## What changes are included in this PR? - Spark-cast plain arrays to the target dictionary value type before using Arrow to build the dictionary. - Remove the manual Parquet dictionary branch and use the existing `can_cast_types` / `cast_with_options` fallback. - Add Rust regressions for post-cast value deduplication and Parquet dictionary key/value conversion with nulls. ## How are these changes tested? - `cargo test --manifest-path native/Cargo.toml -p datafusion-comet-spark-expr --lib` (595 passed) - `cargo test --manifest-path native/Cargo.toml -p datafusion-comet --lib` (136 passed, 4 HDFS tests ignored) - `cargo clippy --manifest-path native/Cargo.toml -p datafusion-comet-spark-expr -p datafusion-comet --lib --tests -- -D warnings` - `cargo fmt --manifest-path native/Cargo.toml --all -- --check` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
