dejankrak-db opened a new pull request, #59843: URL: https://github.com/apache/spark/pull/59843
### What changes were proposed in this pull request? The single-pass resolver fails dual-run for a top-level join whose condition references the hidden key of a USING join below it, when nothing above the join projects its output: ```sql FROM nt1 JOIN nt2 USING (k) JOIN nt2 x ON nt2.k = x.k ``` ``` HYBRID_ANALYZER_EXCEPTION.OUTPUT_SCHEMA_COMPARISON_MISMATCH ``` To resolve `nt2.k`, both analyzers add the hidden key to the output of the USING join's Project. The fixed-point analyzer's `AddMetadataColumns` then wraps the outer join in a Project with the join's original output. The single-pass resolver restores the original output only from the resolver of the operator above (`RetainsOriginalJoinOutput`), so a top-level join kept the extra key in its output. This PR adds `retainTopLevelJoinOutput` to `JoinResolver`. It wraps a top-level regular join in a Project with the join's output when the join condition references an attribute found only in the hidden output, setting the join as the base operator as the USING join path does. The condition's referenced attributes are used only when the join has a condition. It also makes `ThriftServerQueryTestBase.isNeedSort` sort the results of queries that start with `FROM`, as `SQLQueryTestSuite` does for any query whose plan is not sorted. Without it, `ThriftServerQueryTestSuite` compares the new FROM-first query in server row order against the sorted golden output. The expected and the actual output go through the same sort, so the other goldens with FROM-first queries (`unpivot.sql`, `named-window.sql`) stay consistent. Joins with such a condition below a `Filter`, `Limit`, `Sort` or `SubqueryAlias` with no projecting operator above are out of scope here. ### Why are the changes needed? With dual-run enabled, which is the default in tests, the query above fails analysis instead of returning its rows. The single-pass resolver must produce the same plan as the fixed-point analyzer for the queries `ResolverGuard` accepts. ### Does this PR introduce _any_ user-facing change? No. The single-pass resolver is off by default, and dual-run is enabled only in tests. ### How was this patch tested? Added two queries to `using-join.sql`: the top-level join above, and the same join with a correlated scalar subquery in its condition, which still fails with `UNSUPPORTED_CORRELATED_SCALAR_SUBQUERY` as it does in the fixed-point analyzer. Without the fix the first query fails with `OUTPUT_SCHEMA_COMPARISON_MISMATCH`. The golden files were regenerated with `SPARK_GENERATE_GOLDEN_FILES=1`. Also ran `SQLQueryTestSuite` for the join inputs, `ThriftServerQueryTestSuite` for `using-join.sql`, `DataFrameJoinSuite`, `JoinSuite`, `SQLQuerySuite` and the catalyst suites under `org.apache.spark.sql.catalyst.analysis.resolver`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
