contentforge-press opened a new pull request, #6533:
URL: https://github.com/apache/datafusion-comet/pull/6533
## Summary
Fix for issue #3646: `array_except` fails with type mismatch error when
input arrays have different element nullability.
## Problem
DataFusion's `array_except` requires both input arrays to have the same
element nullability. When one array has nullable elements and the other has
non-nullable elements, we get:
```
CometNativeException: array_except received incompatible types: List(Int32),
List(non-null Int32)
```
## Solution
Normalize element type nullability before passing arrays to DataFusion's
`array_except` function. The fix:
1. Checks the nullability of both input arrays
2. If one is nullable and the other is not, casts the non-nullable array to
nullable
3. Ensures both arrays have compatible types before calling the native
function
## Changes
Modified `CometArrayExcept.convert()` in
`spark/src/main/scala/org/apache/comet/serde/arrays.scala`:
- Added nullability normalization logic
- Added clear comments explaining the fix
- Preserved existing type checking and fallback behavior
## Testing
The fix should enable the previously ignored test case in `array_except.sql`:
```sql
-- This should now work without the ignore directive
SELECT array_except(array(1, 2, 3), array(2, 3, 4)),
array_except(array(1, 2), array()),
array_except(array(), array(1)),
array_except(cast(NULL as array<int>), array(1))
```
## Related Issues
Fixes #3646
## Checklist
- [x] Code compiles without errors
- [x] Fix addresses the root cause (type nullability mismatch)
- [x] Comments explain the reasoning
- [x] No breaking changes to existing behavior
- [ ] Tests pass (requires CI)
- [ ] Documentation updated if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
