Jungtaek Lim created SPARK-61380:
------------------------------------
Summary: Stream-stream left semi join drops a match when both rows
arrive in one micro-batch between the watermarks
Key: SPARK-61380
URL: https://issues.apache.org/jira/browse/SPARK-61380
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
A stream-stream left semi join can drop a left row whose matching right row
arrives in the same micro-batch.
Since SPARK-55973 (Spark 4.2.0), a left semi join processes the right input of
a micro-batch before its left input, so that a left row that matches is emitted
without being stored. The join does not store a right row whose event time is
at or below the batch's eviction cutoff, because it assumes the left rows of
the batch have already probed it. That holds for inner and outer joins, which
process the left input first, but not for a left semi join.
With the default {{spark.sql.streaming.statefulOperator.allowMultiple=true}}, a
micro-batch filters late rows with the previous batch's watermark and evicts
state with the current one. A right row with an event time between those two
watermarks therefore passes late filtering but is not stored. The left rows of
the same batch then probe the right state, find nothing, and are never emitted.
Reproducer (Scala, {{MemoryStream}}, both watermarks 0 seconds, no-data
micro-batches disabled so that each {{AddData}} is one batch):
{code:scala}
val left = leftInput.toDF()
.select($"_1".as("leftKey"), timestamp_seconds($"_2").as("leftTime"))
.withWatermark("leftTime", "0 seconds")
val right = rightInput.toDF()
.select($"_1".as("rightKey"), timestamp_seconds($"_2").as("rightTime"))
.withWatermark("rightTime", "0 seconds")
val joined = left.join(right, expr("leftKey = rightKey AND leftTime =
rightTime"), "left_semi")
// batch 1: left (7, 100), right (8, 100) -> watermark becomes 100 s
// batch 2: left (7, 105), right (8, 105) -> watermark becomes 105 s
// batch 3: left (1, 102), right (1, 102) -> late-events watermark 100 s,
eviction watermark 105 s
{code}
- Expected: batch 3 outputs the left row (1, 102).
- Actual: batch 3 outputs nothing, and later batches do not output it either.
The same happens for a key shifted by an interval, such as {{leftTime +
INTERVAL 10 SECONDS = rightTime}}, when the watermark advances by more than the
shift in one batch. For example: left (1, 120) and right (1, 130) in a batch
whose watermarks are 100 s and 150 s.
How the affected versions were checked:
- The reproducer fails on master in state formats 2, 3 and 4.
- The right-first order and the right-side skip are present in v4.2.0, v4.2.1
and v4.3.0. That was checked by inspecting {{StreamingSymmetricHashJoinExec}};
the reproducer was not run on those tags.
- v4.1.x processes the left input first and is not affected.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]