This is an automated email from the ASF dual-hosted git repository.

gengliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 54e9999  [SPARK-35604][SQL] Fix condition check for FULL OUTER sort 
merge join
54e9999 is described below

commit 54e9999d39823c4fc236f328fe55e46607515cd0
Author: Cheng Su <chen...@fb.com>
AuthorDate: Wed Jun 2 14:01:34 2021 +0800

    [SPARK-35604][SQL] Fix condition check for FULL OUTER sort merge join
    
    ### What changes were proposed in this pull request?
    
    The condition check for FULL OUTER sort merge join 
(https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala#L1368
 ) has unnecessary trip when `leftIndex == leftMatches.size` or `rightIndex == 
rightMatches.size`. Though this does not affect correctness 
(`scanNextInBuffered()` returns false anyway). But we can avoid it in the first 
place.
    
    ### Why are the changes needed?
    
    Better readability for developers and avoid unnecessary execution.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Existing unit tests, such as `OuterJoinSuite.scala`.
    
    Closes #32736 from c21/join-bug.
    
    Authored-by: Cheng Su <chen...@fb.com>
    Signed-off-by: Gengliang Wang <ltn...@gmail.com>
---
 .../scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
index c565f91..5873754 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
@@ -1365,7 +1365,7 @@ private class SortMergeFullOuterJoinScanner(
 
   def advanceNext(): Boolean = {
     // If we already buffered some matching rows, use them directly
-    if (leftIndex <= leftMatches.size || rightIndex <= rightMatches.size) {
+    if (leftIndex < leftMatches.size || rightIndex < rightMatches.size) {
       if (scanNextInBuffered()) {
         return true
       }

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

Reply via email to