LukaZdravic opened a new pull request, #60029: URL: https://github.com/apache/spark/pull/60029
### What changes were proposed in this pull request? Two small changes to `SortMergeAsOfJoinScanner`: - It compiles the as-of and residual conditions with `Predicate.create`. Before, each check walked the bound expression tree and boxed every value it read. - It returns its output row without a cop. A consumer that keeps rows copies them. ### Why are the changes needed? The scan checks the as-of condition for every right row it reads, so these checks are its hot path. Local benchmark (best of 5, ms; 400K rows per side and 2000 keys, except the no-key cases with 10K rows per side): | Case | Before | After | |---|---|---| | backward | 970 | 613 | | forward | 990 | 600 | | left outer | 942 | 611 | | backward with tolerance | 2271 | 1159 | | nearest | 782 | 773 | | no key | 723 | 437 | | no key, with an extra ON condition | 2155 | 610 | ### Does this PR introduce _any_ user-facing change? No. ### How was this s PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests: `SortMergeAsOfJoinScannerSuite`, `AsOfJoinSortMergeSQLSuite`, `SortMergeAsOfJoinSuite`, `AsOfJoinSQLSuite`, `DataFrameAsOfJoinSuite`, the ASOF test in `SQLMetricsSuite`, `PlannerSuite`, and the `join-asof-*.sql` golden files. Tests such as "null as-of keys" check that a NULL condition result still counts as no match. To check that no row is lost without the copy, a local test fed the join output into operators that keep rows (sort, limit, window, shuffle, distinct, union, `collect_list`, a second sort-merge join, cache, `toLocalIterator`), with whole-stage codegen on and off. All 37 cases matched the correlated subquery path. All 37 failed when the scanner was changed to overwrite its row early. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
