kazantsev-maksim opened a new pull request, #5688:
URL: https://github.com/apache/datafusion-comet/pull/5688
## Which issue does this PR close?
Closes #5583
## Rationale for this change
`arrays_zip` over an array whose element type is a map
(`array<map<string,int>>`) is not executed natively: `CometArraysZip`'s type
gate returns `Unsupported` for `MapType` element arrays, so the entire
projection falls back to Spark, even though Spark supports it.
The JVM codegen dispatcher already handles map types:
`CometBatchKernelCodegen.isSupportedDataType` admits `MapType` recursively, and
`ArraysZip` is a plain codegen expression, so dispatching it runs Spark's own
`doGenCode` and matches Spark exactly by construction.
## What changes are included in this PR?
- `CometArraysZip` now mixes in `CodegenDispatchFallback`: arrays of maps
are routed through the JVM codegen dispatcher, while scalar/array/struct element
arrays keep the existing native path.
- Updated `getUnsupportedReasons()`: it now surfaces only when the
dispatcher is disabled or rejects the tree.
- Carve-out (unchanged behavior, documented): `NullType` elements are not
dispatchable — `NullType` is absent from `isSupportedDataType` — so
`arrays_zip` over `array<null>` continues to fall back to Spark.
- Convert-side declines are also unchanged: when a child of `ArraysZip`
fails to
serialize, the serde still returns `None` and the projection falls back
(the dispatcher cannot see that state).
- Regenerated expression compatibility docs (`arrays_zip` is now hybrid: map
elements via dispatch, other elements natively).
## How are these changes tested?
New sql test cases added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
