srielau opened a new pull request, #57277: URL: https://github.com/apache/spark/pull/57277
### What changes were proposed in this pull request? Third stacked PR for [SPARK-58092](https://issues.apache.org/jira/browse/SPARK-58092) / [SPARK-58124](https://issues.apache.org/jira/browse/SPARK-58124). **Depends on #57264** (analysis) and **#57251** (parser). Until those land, the diff vs `master` includes earlier stacked commits; the execution-only delta is `SPARK-58123..SPARK-58124` (1 commit). After upstream merges, this branch will be rebased onto `master` for a clean diff. **Physical operator** - Extend `SortMergeAsOfJoinExec` with multi-column `leftSortExprs` / `rightSortExprs` (replacing single `leftAsOfExpr` / `rightAsOfExpr`) - Detect backward scan direction from `asOfCondition` so composite `MATCH_CONDITION` sort keys work **Planner** - `AsOfJoinSelection` uses analysis-populated `leftSortExprs` / `rightSortExprs` when non-empty; DataFrame path falls back to single-column extraction from `orderExpression` **DataFrame API** - `AsOfJoin.apply()` populates sort expressions via `matchSortExpressions` (deferred from #57264) **Tests** - Add `AsOfJoinSortMergeSQLSuite` — 30 SQL execution/integration tests (timestamp, struct, array, forward/strict, `USING`, left outer, etc.) ### Why are the changes needed? Analysis (#57264) materializes per-side sort expressions for SQL `MATCH_CONDITION` operands (scalar, struct tuple, whole struct, array). The physical operator and planner must sort by those expressions — not a single as-of column — for correct sort-merge ASOF join semantics. ### Does this PR introduce _any_ user-facing change? Yes (behind `spark.sql.join.asofJoin.enabled` and `spark.sql.join.sortMergeAsOfJoin.enabled`, both default off). With both confs enabled, valid SQL ASOF JOIN queries now execute end-to-end via `SortMergeAsOfJoinExec`. ### How was this patch tested? Locally: ``` build/sbt "sql/testOnly org.apache.spark.sql.AsOfJoinSortMergeSQLSuite \ org.apache.spark.sql.SortMergeAsOfJoinSuite \ org.apache.spark.sql.DataFrameAsOfJoinSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Cursor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
