dwsmith1983 commented on code in PR #5867:
URL: https://github.com/apache/datafusion-comet/pull/5867#discussion_r4073204667
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -51,7 +51,39 @@ object CometArrayRemove
}
}
-object CometArrayAppend extends CometExpressionSerde[ArrayAppend] with
ArraysBase {
+/**
+ * Shared gate for serdes whose native NULL guard (`CASE WHEN child IS NOT
NULL`) serializes the
+ * child twice: a stateful child drifts between the two copies, so it is
declined and runs through
+ * the JVM codegen dispatcher, where Spark evaluates it once. Nullability is
not consulted: a
+ * non-nullable stateful child only stays in step because DataFusion skips the
filter when the
+ * guard matches every row, which is not a contract to lean on.
+ */
+private[serde] object NullGuardSupport {
+
+ val nondeterministicReason: String =
Review Comment:
> Would something like "Comet has no native path for a nondeterministic
operand such as `rand()` or `monotonically_increasing_id()`, because the native
NULL guard would evaluate it twice" sit better there?
Yes. That is the reason now, word for word apart from the code formatting,
and since the string is shared it changes the bullet for all four serdes in
0d61cac1e.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
