HyukjinKwon commented on code in PR #59267:
URL: https://github.com/apache/spark/pull/59267#discussion_r4261554266
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AsOfJoinResolver.scala:
##########
@@ -124,11 +132,43 @@ class AsOfJoinResolver(
)
cteRegistry.currentScope.tryPutWithCTE(
unresolvedOperator = unresolvedAsOfJoin,
- resolvedOperator = resolvedJoin
+ resolvedOperator = resolvedJoinWithOriginalOutput
)
}
}
+ /**
+ * If the join uses a hidden column, wraps it in a [[Project]] that also
keeps the hidden columns.
+ */
+ private def retainOriginalOutputIfHiddenColumnIsReferenced(
+ unresolvedAsOfJoin: AsOfJoin,
+ resolvedJoin: AsOfJoin,
+ leftNameScope: NameScope,
+ rightNameScope: NameScope): LogicalPlan = {
+ val originalOutput = AsOfJoin.computeOutput(
+ joinType = resolvedJoin.joinType,
+ leftOutput = leftNameScope.output,
+ rightOutput = rightNameScope.output
+ )
+ val originalOutputSet = AttributeSet(originalOutput)
+
+ if (resolvedJoin.references.subsetOf(originalOutputSet)) {
+ resolvedJoin
+ } else {
+ val hiddenColumns =
+ if (unresolvedAsOfJoin.containsTag(ResolverTag.TOP_LEVEL_OPERATOR)) {
Review Comment:
**Nit (P3):** The `TOP_LEVEL_OPERATOR` branch here has no test. Every new
case in `join-asof-semantics.sql` and `AsOfJoinSQLSuite` puts a `SELECT` (or
another operator) above the ASOF JOIN, so only the non-top-level branch runs. A
bare `FROM` query reaches this branch, because `fromStatement` allows zero
bodies and the resolver tags the plan root:
```sql
FROM (VALUES (1, 10), (2, 20)) AS l(k, lts) ASOF JOIN
((VALUES (1, 5), (8, 15)) AS r(k, rts) LEFT JOIN (VALUES (1)) AS d(k)
USING (k))
MATCH_CONDITION (l.lts >= d.k)
```
If this branch regressed and appended the hidden columns, `d.k` would end up
in the single-pass result schema: at the root, `PruneMetadataColumns` returns
the project list unchanged because nothing above needs anything pruned. Neither
the dual-run plan comparison nor `checkAnswer` would catch it. Both analyzers
agree today, so this guards against future regressions. Adding the query above
to `join-asof-semantics.sql` and regenerating the goldens would cover the
branch.
**Verification:**
- **Regression:** A bare FROM ASOF JOIN with a hidden-key MATCH_CONDITION
operand resolves with identical single-pass and fixed-point plans and exposes
only the visible columns. The case would fail if the top-level branch started
appending hidden columns.
- **Compatibility:** Existing ASOF JOIN golden plans and results, including
FVT-ASOF-2-027..039, remain unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
