LuciferYang opened a new pull request, #57308:
URL: https://github.com/apache/spark/pull/57308
### What changes were proposed in this pull request?
`SubqueryExpression` (in `sql/api`'s `internal/columnNodes.scala`) was the
only `ColumnNode` that returned `children = Seq.empty` even when its
`SubqueryType` was `IN(values)`, whose `values: Seq[ColumnNode]` can hold
nested column nodes -- including another `SubqueryExpression`. It also did not
override `normalize()`, unlike every sibling node.
This PR makes `SubqueryExpression` consistent with the other nodes:
- `children` now exposes `IN.values` (and stays empty for
`SCALAR`/`EXISTS`). The subquery plan `ds` is intentionally not a child -- it
is a DataFrame plan, not a column node, and the converters handle it separately.
- `normalize()` is overridden to strip `origin` and normalize the
`IN.values`, mirroring sibling nodes.
### Why are the changes needed?
A nested subquery is reachable through the public 4.1 API, e.g.
```scala
t1.select(max($"v")).scalar().isin(t2.select($"c"))
// WHERE (SELECT max(v) FROM t1) IN (SELECT c FROM t2)
```
On Spark Connect, `SparkSession.newDataset` collects subquery plan
references via `node.collect { case _: SubqueryExpression => ... }`, which
recurses through `children`. Because `children` was empty, the inner scalar
subquery's plan was never added to `WithRelations`, while the proto converter
still emitted a planId reference to it. The server then failed resolving that
planId:
```
[CONNECT_INVALID_PLAN.ASSERTION_FAILURE] Missing relation in WithRelations:
(N) not in (M)
```
Spark Classic is unaffected: its converter embeds the subquery's logical
plan directly and does not rely on `children`. The missing `normalize()`
override is a separate, lower-impact inconsistency: two Columns built from the
same `ds` at different call sites compared unequal purely due to retained
`origin`.
### Does this PR introduce _any_ user-facing change?
Yes. On Spark Connect, a nested subquery inside `isin(ds)` -- e.g.
`df1.scalar().isin(df2)` -- previously failed with an internal `Missing
relation in WithRelations` error. After this fix it works, matching Spark
Classic and the equivalent SQL. This is a bug fix relative to released 4.1.x.
### How was this patch tested?
New cases added to `DataFrameSubquerySuite` in both `sql/core` (Classic) and
`sql/connect/client/jvm` (Connect), exercising a scalar subquery nested inside
an IN subquery's values. Verified reproduce-then-fix: the Connect case fails
with `Missing relation in WithRelations` on the unfixed tree and passes after
the fix; both full suites pass with no regressions.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
