[ https://issues.apache.org/jira/browse/SPARK-32863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim updated SPARK-32863: --------------------------------- Priority: Major (was: Trivial) > Full outer stream-stream join > ----------------------------- > > Key: SPARK-32863 > URL: https://issues.apache.org/jira/browse/SPARK-32863 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming > Affects Versions: 3.1.0 > Reporter: Cheng Su > Priority: Major > > Current stream-stream join supports inner, left outer and right outer join > ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L166] > ). With current design of stream-stream join (which marks whether the row is > matched or not in state store), it would be very easy to support full outer > join as well. > > Full outer stream-stream join will work as followed: > (1).for left side input row, check if there's a match on right side state > store. If there's a match, output all matched rows. Put the row in left side > state store. > (2).for right side input row, check if there's a match on left side state > store. If there's a match, output all matched rows and update left side rows > state with "matched" field to set to true. Put the right side row in right > side state store. > (3).for left side row needs to be evicted from state store, output the row if > "matched" field is false. > (4).for right side row needs to be evicted from state store, output the row > if "matched" field is false. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org