cloud-fan commented on code in PR #58077:
URL: https://github.com/apache/spark/pull/58077#discussion_r3823269824
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:
##########
@@ -125,7 +127,17 @@ case class InSubqueryExec(
@transient private lazy val inSet = InSet(child, result.toSet)
- override def nullable: Boolean = child.nullable
+ // Mirror the logical InSubquery.nullable: nullable when any output column
is nullable
+ // (null in any column position produces UNKNOWN on a miss) or when the
child is nullable.
+ // Respects LEGACY_IN_SUBQUERY_NULLABILITY to stay in sync with the logical
node.
+ // See SPARK-58481.
+ override def nullable: Boolean = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_IN_SUBQUERY_NULLABILITY)) {
+ child.nullable || plan.output.exists(_.nullable)
Review Comment:
**Blocking:**
Row-valued `IN` must remain nullable when any left-hand field is nullable.
For multi-column inputs, `child` is the `CreateNamedStruct` from
`PlanSubqueries`, whose top-level `nullable` is always false; with a
non-nullable RHS, this declares the expression non-nullable even though
`evalMultiColumn` returns UNKNOWN for a null LHS field. Generated `NOT IN` can
then turn UNKNOWN into TRUE. Please derive LHS nullability from the struct
fields or underlying values and add a nullable-LHS/non-nullable-RHS codegen
regression.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:
##########
@@ -165,14 +177,81 @@ case class InSubqueryExec(
}
}
+ // Three-valued IN semantics for multi-column subqueries.
+ // Result rows are InternalRow objects; InSet uses TreeSet ordering which
treats null fields as
+ // non-equal and therefore cannot distinguish a definitively-false candidate
(a non-null field
+ // differs) from an indeterminate one (all non-null fields match but some
fields are null).
+ // We replicate In.eval's per-candidate logic: TRUE if any candidate matches
exactly,
+ // UNKNOWN if no TRUE and at least one candidate is indeterminate, FALSE
otherwise.
+ private def evalMultiColumn(inputRow: InternalRow): Any = {
+ val value = child.eval(inputRow)
+ if (value == null) return null
+ val inputStruct = value.asInstanceOf[InternalRow]
+ val output = plan.output
+ val numFields = output.length
+ // Hoist per-field orderings: invariant across all candidate rows.
+ val orderings = Array.tabulate(numFields)(i =>
+ TypeUtils.getInterpretedOrdering(output(i).dataType))
+ val fieldTypes = Array.tabulate(numFields)(i => output(i).dataType)
+ var hasUnknown = false
+ var i = 0
+ while (i < result.length) {
Review Comment:
**Blocking:**
Please avoid scanning the entire collected subquery result on every
predicate evaluation. The generated path calls this same loop, while `InSet`
already provides indexed struct membership; on a nested-loop join this adds
another factor of subquery cardinality to every candidate pair. Keep the
indexed fast path for provably non-null rows and precompute the schema,
orderings, and any null-aware candidate index outside `evalMultiColumn`.
##########
sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/in-joins.sql:
##########
@@ -389,6 +389,20 @@ ON s1.id = s2.id
AND s1.id NOT IN (SELECT id FROM s3);
+-- SPARK-58481: NOT IN subquery with NULL in result and non-nullable child on
FULL OUTER JOIN.
+-- 5 NOT IN (99, NULL) evaluates to UNKNOWN; a join condition that is not TRUE
matches no rows,
+-- so FULL OUTER JOIN must emit null-padded rows for every row from each side
(3+3=6 rows).
Review Comment:
**Nit:**
`s1` and `s2` contain five rows each, and the generated result contains ten
null-padded rows.
```suggestion
-- so FULL OUTER JOIN must emit null-padded rows for every row from each
side (5+5=10 rows).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
