martin-g commented on code in PR #2760: URL: https://github.com/apache/datafusion-comet/pull/2760#discussion_r2514330928
########## docs/source/user-guide/latest/compatibility.md: ########## @@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details. + +### Complex Type Casting + +Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules: + +#### Array Type Casting + +- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string. + - Recursively checks that the array element type supports casting to string + - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"` + +- **Array to Array**: Arrays can be cast to other array types if the element types are compatible. Review Comment: There are no tests for this. https://github.com/apache/datafusion-comet/blob/e77998ad2228b758d0a93588c57325e56bda83bc/spark/src/test/scala/org/apache/comet/CometCastSuite.scala#L1050 `cast ArrayType to StringType` seems to be the only `cast ArrayType to ...` test in the suite ########## docs/source/user-guide/latest/compatibility.md: ########## @@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details. + +### Complex Type Casting + +Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules: + +#### Array Type Casting + +- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string. + - Recursively checks that the array element type supports casting to string + - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"` + +- **Array to Array**: Arrays can be cast to other array types if the element types are compatible. + - Recursively validates that the source element type can be cast to the target element type + - Example: `CAST(array(1, 2, 3) AS ARRAY<BIGINT>)` → `array(1L, 2L, 3L)` + - The compatibility level (Compatible/Incompatible/Unsupported) is determined by the element type cast + +- **Special case**: Arrays with `NullType` elements (empty arrays) can be cast to any array type + +#### Struct Type Casting + +- **Struct to Struct**: Structs can be cast to other struct types if all field types are compatible. + - Requires the same number of fields in source and target structs Review Comment: This is not quite correct! * https://github.com/apache/datafusion-comet/blob/e77998ad2228b758d0a93588c57325e56bda83bc/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala#L157-L158 - the comparison uses `Iterator.zip()` * https://github.com/scala/scala/blob/v2.13.6/src/library/scala/collection/Iterator.scala#L794-L799 - but zip() does not mandate equal sizes. It just iterates over the underlying iterators until one of them has no more items. If the first struct (from) has more items than the second (to) then some of from's items will be discarded. ########## docs/source/user-guide/latest/compatibility.md: ########## @@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details. + +### Complex Type Casting + +Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules: + +#### Array Type Casting + +- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string. + - Recursively checks that the array element type supports casting to string + - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"` + +- **Array to Array**: Arrays can be cast to other array types if the element types are compatible. + - Recursively validates that the source element type can be cast to the target element type + - Example: `CAST(array(1, 2, 3) AS ARRAY<BIGINT>)` → `array(1L, 2L, 3L)` + - The compatibility level (Compatible/Incompatible/Unsupported) is determined by the element type cast + +- **Special case**: Arrays with `NullType` elements (empty arrays) can be cast to any array type Review Comment: Is it correct to say `empty arrays` here ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
