[
https://issues.apache.org/jira/browse/SPARK-61380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61380:
-----------------------------------
Labels: pull-request-available (was: )
> Stream-stream left semi join drops a match when both rows arrive in one
> micro-batch between the watermarks
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61380
> URL: https://issues.apache.org/jira/browse/SPARK-61380
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.2.1, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> A stream-stream left semi join can drop a left row whose matching right row
> arrives in the same micro-batch.
> Since SPARK-55973 (Spark 4.2.0), a left semi join processes the right input
> of a micro-batch before its left input, so that a left row that matches is
> emitted without being stored. The join does not store a right row whose event
> time is at or below the batch's eviction cutoff, because it assumes the left
> rows of the batch have already probed it. That holds for inner and outer
> joins, which process the left input first, but not for a left semi join.
> With the default {{spark.sql.streaming.statefulOperator.allowMultiple=true}},
> a micro-batch filters late rows with the previous batch's watermark and
> evicts state with the current one. A right row with an event time between
> those two watermarks therefore passes late filtering but is not stored. The
> left rows of the same batch then probe the right state, find nothing, and are
> never emitted.
> Reproducer (Scala, {{MemoryStream}}, both watermarks 0 seconds, no-data
> micro-batches disabled so that each {{AddData}} is one batch):
> {code:scala}
> val left = leftInput.toDF()
> .select($"_1".as("leftKey"), timestamp_seconds($"_2").as("leftTime"))
> .withWatermark("leftTime", "0 seconds")
> val right = rightInput.toDF()
> .select($"_1".as("rightKey"), timestamp_seconds($"_2").as("rightTime"))
> .withWatermark("rightTime", "0 seconds")
> val joined = left.join(right, expr("leftKey = rightKey AND leftTime =
> rightTime"), "left_semi")
> // batch 1: left (7, 100), right (8, 100) -> watermark becomes 100 s
> // batch 2: left (7, 105), right (8, 105) -> watermark becomes 105 s
> // batch 3: left (1, 102), right (1, 102) -> late-events watermark 100 s,
> eviction watermark 105 s
> {code}
> - Expected: batch 3 outputs the left row (1, 102).
> - Actual: batch 3 outputs nothing, and later batches do not output it either.
> The same happens for a key shifted by an interval, such as {{leftTime +
> INTERVAL 10 SECONDS = rightTime}}, when the watermark advances by more than
> the shift in one batch. For example: left (1, 120) and right (1, 130) in a
> batch whose watermarks are 100 s and 150 s.
> How the affected versions were checked:
> - The reproducer fails on master in state formats 2, 3 and 4.
> - The right-first order and the right-side skip are present in v4.2.0, v4.2.1
> and v4.3.0. That was checked by inspecting
> {{StreamingSymmetricHashJoinExec}}; the reproducer was not run on those tags.
> - v4.1.x processes the left input first and is not affected.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]