Muhammad Ali Gulzar created SPARK-60751:
-------------------------------------------
Summary: Correlated EXISTS over dropDuplicates fails with
INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND
Key: SPARK-60751
URL: https://issues.apache.org/jira/browse/SPARK-60751
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.4, 4.0.0, 3.5.9, 4.2.1
Reporter: Muhammad Ali Gulzar
{code:scala}
val df = spark.range(20).toDF("k").withColumn("sold", $"k" %
7).dropDuplicates("k")
df.createOrReplaceTempView("v")
spark.sql("SELECT * FROM v o WHERE EXISTS (SELECT 1 FROM v i WHERE i.sold =
o.sold)").collect()
// expected: 20 rows
// actual: [INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND] Could not find sold#7L in
[k#1L,sold#11L]
{code}
The query passes analysis and fails while binding the join condition of
BroadcastNestedLoopJoinExec. The same happens with o.sold IN (SELECT i.sold
FROM v i WHERE i.sold = o.sold). Correct when the subquery correlates on the
key {{{}k{}}}, and when {{dropDuplicates("k")}} is replaced by
{{{}groupBy("k").agg(first("sold").alias("sold")){}}}.
Cause: PullupCorrelatedPredicates moves {{sold#7L = sold#2L}} out of the
subquery. OptimizeSubqueries then runs ReplaceDeduplicateWithAggregate on the
subquery, which replaces the non-key column with {{{}first(sold#7L) AS
sold#9L{}}}, a new expression ID. The pulled-up predicate still references
{{{}sold#7L{}}}, so RewritePredicateSubquery produces {{Join LeftSemi, (sold#7L
= sold#11L)}} over a right side that no longer outputs {{{}sold#7L{}}}.
Rewriting the pulled-up predicates with the new attribute is the likely fix.
Reproduces on 4.0.0, 4.0.5, 4.1.4, 4.2.0, 4.2.1 and master (5.0.0-SNAPSHOT,
2026-10-29). On 3.5.9, this query returns 20 rows, but it fails with the same
error when the outer query and the subquery read two separately built views, so
the bug is older than 4.0. The query above most likely started failing in 4.0
because of SPARK-43838, which changed how the analyzer handles a view that
appears in both the outer query and the subquery.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
