[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215712#comment-17215712 ]
Apache Spark commented on SPARK-32862: -------------------------------------- User 'c21' has created a pull request for this issue: https://github.com/apache/spark/pull/30076 > Left semi stream-stream join > ---------------------------- > > Key: SPARK-32862 > URL: https://issues.apache.org/jira/browse/SPARK-32862 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming > Affects Versions: 3.1.0 > Reporter: Cheng Su > Priority: Minor > > Current stream-stream join supports inner, left outer and right outer join > ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L166] > ). We do see internally a lot of users are using left semi stream-stream > join (not spark structured streaming), e.g. I want to get the ad impression > (join left side) which has click (joint right side), but I don't care how > many clicks per ad (left semi semantics). > > Left semi stream-stream join will work as followed: > (1).for left side input row, check if there's a match on right side state > store > (1.1). if there's a match, output the left side row. > (1.2). if there's no match, put the row in left side state store (with > "matched" field to set to false in state store). > (2).for right side input row, check if there's a match on left side state > store. If there's a match, update left side row state with "matched" field to > set to true. Put the right side row in right side state store. > (3).for left side row needs to be evicted from state store, output the row if > "matched" field is true. > (4).for right side row needs to be evicted from state store, doing nothing. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org