Nikolina Vraneš created SPARK-60035:
---------------------------------------
Summary: Single-pass analyzer drops hidden USING keys referenced
by parent expressions
Key: SPARK-60035
URL: https://issues.apache.org/jira/browse/SPARK-60035
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Nikolina Vraneš
h3. Problem
The single-pass analyzer can drop a qualified key hidden by a nested USING join
when a parent expression still references that key. The fixed-point analyzer
accepts the same query.
This affects ordinary JOIN. The ASOF branch added in
https://github.com/apache/spark/pull/59267 exposes the same problem for ASOF
JOIN.
h3. Reproducer
The ASOF case was reproduced on PR #59267 at commit
a579e4b79ac86e81ce139f10184cb33415270fc8, with single-pass analysis explicitly
enabled and fallback disabled:
{code:sql}
SET spark.sql.join.asofJoin.enabled = true;
SET spark.sql.analyzer.singlePassResolver.enableAsOfJoinResolution = true;
SET spark.sql.analyzer.singlePassResolver.enabledTentatively = false;
SET spark.sql.analyzer.singlePassResolver.dualRunWithLegacy = false;
SET spark.sql.analyzer.singlePassResolver.enabled = true;
SELECT d.k + 1 AS answer
FROM (VALUES (10)) AS l(lts)
ASOF JOIN (
(VALUES (7, 5)) AS r(k, rts)
JOIN (VALUES (7)) AS d(k) USING (k)
)
MATCH_CONDITION (l.lts >= d.k);
{code}
Expected result: one row with answer = 8.
Actual result: MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION. The
parent Project references d.k, but its child no longer outputs that attribute.
The following ordinary JOIN query also fails under single-pass analysis:
{code:sql}
SELECT d.k + 1 AS answer
FROM (VALUES (10)) AS l(lts)
JOIN (
(VALUES (7, 5)) AS r(k, rts)
JOIN (VALUES (7)) AS d(k) USING (k)
)
ON l.lts >= d.k;
{code}
h3. Verified cases
A temporary suite ran 10 cases. Six passed and four failed with the
missing-attribute error:
* Fixed-point analysis accepts ASOF with a bare hidden key, an alias, a
calculation, and an aggregate. It also accepts the ordinary JOIN calculation.
* Single-pass analysis accepts the bare ASOF SELECT d.k case.
* Single-pass analysis rejects the ASOF SELECT d.k AS dk, SELECT d.k + 1 AS
answer, and SELECT sum(d.k) AS answer cases.
* Single-pass analysis rejects the ordinary JOIN calculation.
h3. Cause and suggested fix
RetainsOriginalJoinOutput compares the parent's top-level output ExprIds when
deciding whether to restore the join's original visible output. An alias,
calculation, or aggregate has a new output ExprId, so that check misses
references to the hidden input key.
The cleanup Project(scopes.current.output, join) then removes the hidden key
even though a parent expression still needs it. The ASOF branch adds the
equivalent Project(scopes.current.output, asOfJoin).
Base the retention decision on the attributes referenced by parent expressions,
not only their output ExprIds. Retain required hidden attributes until the
parent consumes them. Fix the shared helper for both ordinary JOIN and ASOF
JOIN, and add regression coverage for aliases, calculations, and aggregates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
