Yang Jie created SPARK-58165:
--------------------------------
Summary: SubqueryExpression drops nested subquery plans in IN
values, breaking IN-over-subquery on Spark Connect
Key: SPARK-58165
URL: https://issues.apache.org/jira/browse/SPARK-58165
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.2
Reporter: Yang Jie
{{SubqueryExpression}} (sql-api ColumnNode) returns {{children = Seq.empty}}
even when its
{{SubqueryType}} is {{IN(values)}}, whose {{values}} may hold nested
{{ColumnNode}}s --
including another {{SubqueryExpression}}. A nested subquery is reachable
through the public
4.1 API, e.g.
{code:scala}
t1.select(max($"v")).scalar().isin(t2.select($"c"))
// WHERE (SELECT max(v) FROM t1) IN (SELECT c FROM t2)
{code}
On Spark Connect, {{SparkSession.newDataset}} collects subquery plan references
via
{{node.collect \{ case _: SubqueryExpression => ... \}}}, which recurses through
{{children}}. Because {{children}} is empty, the inner scalar subquery's plan
is never added
to {{WithRelations}}, while the proto converter still emits a planId reference
to it. The
server then fails resolving that planId:
{noformat}
[CONNECT_INVALID_PLAN.ASSERTION_FAILURE] Missing relation in WithRelations: (N)
not in (M)
{noformat}
Spark Classic is unaffected: its converter embeds the subquery's logical plan
directly and
does not rely on {{children}}.
Fix: {{SubqueryExpression.children}} now exposes {{IN.values}}, and
{{normalize()}} is
overridden to mirror sibling nodes (strip {{origin}}, normalize {{values}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
