HeartSaVioR commented on a change in pull request #28975:
URL: https://github.com/apache/spark/pull/28975#discussion_r449295416
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -295,6 +295,10 @@ case class StreamingSymmetricHashJoinExec(
postJoinFilter(joinedRow.withLeft(leftKeyValue.value).withRight(rightValue))
}
}
+
+ // NOTE: we need to make sure `outerOutputIter` is evaluated "after"
exhausting all of
+ // elements in `innerOutputIter`, because evaluation of
`innerOutputIter` may update
+ // the match flag which the logic for outer join is relying on.
Review comment:
Yes right.
TBH I suspected this first and crafted a patch including the part with new
iterator explicitly runs the logic after evaluating innerOutputIter, and later
realized current logic already dealt with this properly, because
removeOldState() doesn't materialize the candidates and evaluate lazily. This
patch contains minimal change.
Worth to mention how it works for someone who may need to touch here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
