Yibo Dong created SPARK-59042:
---------------------------------
Summary: PushProjectionThroughUnion fails on a correlated scalar
subquery over UNION ALL
Key: SPARK-59042
URL: https://issues.apache.org/jira/browse/SPARK-59042
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark 5.0.0-SNAPSHOT, commit
9da9f8d673914d1648514f59d85e3adafb300d1a; macOS arm64; Java 17.0.17
Reporter: Yibo Dong
A legal correlated scalar subquery over UNION ALL fails with a
NoSuchElementException when PushProjectionThroughUnion is enabled.
Disabling only this optimizer rule makes the same query execute
successfully.
h3. Reproduction
{code:sql}
SELECT u.a,
(SELECT max(r.x)
FROM (VALUES (1), (2), (3), (NULL)) AS r(x)
WHERE r.x = u.a) AS m
FROM (
SELECT a FROM (VALUES (1), (2)) AS l(a)
UNION ALL
SELECT a FROM (VALUES (2), (3)) AS q(a)
) u
ORDER BY u.a, m;
{code}
With the default optimizer configuration, the query fails with:
{code}
java.util.NoSuchElementException: key not found: x#7
at scala.collection.MapOps.default(Map.scala:289)
at
org.apache.spark.sql.catalyst.expressions.AttributeMap.apply(AttributeMap.scala:41)
at
org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion$$anonfun$3.applyOrElse(Optimizer.scala:1124)
{code}
If only PushProjectionThroughUnion is excluded:
{code}
--conf
spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion
{code}
the query executes successfully and returns:
{code}
1 1
2 2
2 2
3 3
{code}
h3. Expected behavior
The query should execute successfully with the default optimizer configuration
and return the same four rows.
h3. Additional information
The failure appears specific to PushProjectionThroughUnion: excluding only this
rule avoids the exception.
The exception indicates that PushProjectionThroughUnion attempts to look up
the subquery-local attribute x in an AttributeMap while rewriting the
projection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
