sunchao commented on code in PR #5526:
URL: https://github.com/apache/datafusion-comet/pull/5526#discussion_r3921234624
##########
spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala:
##########
@@ -113,13 +142,19 @@ object CometBatchKernelCodegen extends Logging with
CometExprTraitShim with Come
* back cleanly rather than crashing the Janino compile at execute time.
*
* Checks every `BoundReference`'s data type and the root `expr.dataType`
against
- * [[isSupportedDataType]], rejects aggregates / generators / `Unevaluable`,
and gates total
- * nested-field count on `spark.sql.codegen.maxFields`.
+ * [[isSupportedDataType]] and [[duplicateStructFieldNames]], rejects
aggregates / generators /
+ * `Unevaluable`, and gates total nested-field count on
`spark.sql.codegen.maxFields`.
*/
def canHandle(boundExpr: Expression): Option[String] = {
- if (!isSupportedDataType(boundExpr.dataType)) {
+ if (!isSupportedDataType(boundExpr.dataType, allowNullType = true)) {
Review Comment:
[P2] Residual stateful guard case across different arguments
At `79fc84008a7dca16a4c24f28036649c0af2603b0`, the original nullable
`element_at` witness is fixed, but the shared guard still misses a cross-input
case. This is source-derived, not an executed reproduction. With a native LONG
input `t` containing ordered ids 0, 1, 2, 3 in one partition/batch and codegen
dispatch enabled, consider `SELECT id, arrays_zip(transform(array(id), x ->
named_struct('i', monotonically_increasing_id(), 'n', NULL)), IF(id % 2 = 0,
array(id), CAST(NULL AS ARRAY<BIGINT>))) AS z FROM t`. The first array is
non-nullable and nondeterministic, while the second is nullable and
deterministic, so neither matches this predicate. Spark evaluates the first
array even on odd rows where the zip is NULL. `CometArraysZip.convert` instead
puts the value-producing kernel inside a CASE whose THEN batch contains only
even rows, changing the stateful child's evaluation sequence. The base rejected
this NullType-bearing producer. This guard therefore misses a composition newly
ad
mitted by the PR. Could the guard preserve one evaluation before filtering or
retain fallback when a nullable sibling filters a stateful child, with coverage
for nullability and nondeterminism occurring in different arguments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
