viirya opened a new pull request, #6760: URL: https://github.com/apache/datafusion-comet/pull/6760
## Which issue does this PR close? Closes #6758. ## Rationale for this change With `spark.sql.caseSensitive=false`, Spark accepts `greatest`/`least` arguments whose struct field names differ only in case, and compares them field by field by position. Comet passed the arguments through DataFusion's coercion, whose struct coercion matches fields by name when both structs hold the same set of names. Arrow's struct cast then reorders the values by name, so Comet compared different pairs of fields and returned a different row from Spark. ## What changes are included in this PR? - Adds `coerce_to_common_type` next to `if_common_type` and `coerce_branch` in `conditional_funcs/case_when.rs`, built from them. It folds `if_common_type` over the arguments to get a positional common type that keeps the first argument's field names and merges nullability, then casts each argument with Comet's Spark `Cast`, whose struct cast is positional. - The planner calls it for `greatest` and `least` before DataFusion's coercion runs. The arguments then share one type, so DataFusion's coercion has nothing left to reorder. This reuses the same helpers that #6428 builds on for comparisons and `CASE`, rather than adding another copy of the positional logic. ## How are these changes tested? - New SQL file `expressions/math/greatest_least_struct_field_case.sql`, run with `spark.sql.caseSensitive=false`. It covers `greatest` and `least` with float and integer fields, rows where the positional and by-name answers differ, three arguments, structs nested in arrays, and arguments that also differ in nested nullability. All 8 queries fail without this change: 7 return different rows from Spark, and the three-argument query fails with `Cannot cast nullable struct field 'X' to non-nullable field`. - New Rust unit test `coerce_to_common_type_is_positional`, for structs and lists of structs. - The existing `greatest`, `least`, float, and conditional SQL files still pass. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
