yadavay-amzn commented on PR #56101: URL: https://github.com/apache/spark/pull/56101#issuecomment-4552972662
@dilipbiswal @cloud-fan Could you please take a look at this change when you get a chance? This PR adds a dedicated `BroadcastNearestByJoinExec` operator that avoids the N * M cross-product materialization in RewriteNearestByJoin (noted as an out of scope improvement for SPARK-56395). This improvement is inspired by @sarutak's SortMergeAsOfJoinExec for AS-OF join (#55912) and is similar in spirit to it by using a dedicated physical operator to replace an expensive rewrite for a specialized join type. The operator broadcasts the right side and maintains a bounded k-heap per left row to avoid the shuffling cost and full materialization when the right table fits within `autoBroadcastJoinThreshold`. Benchmark numbers have been promising (see description). I've also added several unit tests covering correctness and edge cases. Does this approach align with the planned evolution of the feature? Any concerns about adding a dedicated physical operator vs. optimizing the existing rewrite? Happy to collaborate and adjust the approach based on your feedback. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
