andygrove commented on code in PR #6533:
URL: https://github.com/apache/datafusion-comet/pull/6533#discussion_r4387717678
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -376,15 +384,40 @@ object CometArrayExcept
return None
case None =>
}
- val leftArrayExprProto = exprToProtoInternal(expr.left, inputs, binding)
- val rightArrayExprProto = exprToProtoInternal(expr.right, inputs, binding)
+
+ // Fix for issue #3646: Normalize element type nullability recursively to
avoid DataFusion
+ // type mismatch. DataFusion's array_except requires both arrays to have
the same element
+ // nullability at all levels (including nested arrays).
+ //
+ // We recursively make both types fully nullable to match the existing
widen-nullability
+ // behavior of CometCreateArray. This ensures compatibility with
DataFusion's type system
+ // while preserving the ability to handle nested arrays.
+ val leftType = expr.left.dataType
+ val rightType = expr.right.dataType
+
+ val normalizedLeftType = makeRecursivelyNullable(leftType)
+ val normalizedRightType = makeRecursivelyNullable(rightType)
Review Comment:
can we avoid the recursion if both types are already the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
