Jungtaek Lim created SPARK-61380:
------------------------------------

             Summary: Stream-stream left semi join drops a match when both rows 
arrive in one micro-batch between the watermarks
                 Key: SPARK-61380
                 URL: https://issues.apache.org/jira/browse/SPARK-61380
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


A stream-stream left semi join can drop a left row whose matching right row 
arrives in the same micro-batch.

Since SPARK-55973 (Spark 4.2.0), a left semi join processes the right input of 
a micro-batch before its left input, so that a left row that matches is emitted 
without being stored. The join does not store a right row whose event time is 
at or below the batch's eviction cutoff, because it assumes the left rows of 
the batch have already probed it. That holds for inner and outer joins, which 
process the left input first, but not for a left semi join.

With the default {{spark.sql.streaming.statefulOperator.allowMultiple=true}}, a 
micro-batch filters late rows with the previous batch's watermark and evicts 
state with the current one. A right row with an event time between those two 
watermarks therefore passes late filtering but is not stored. The left rows of 
the same batch then probe the right state, find nothing, and are never emitted.

Reproducer (Scala, {{MemoryStream}}, both watermarks 0 seconds, no-data 
micro-batches disabled so that each {{AddData}} is one batch):

{code:scala}
val left = leftInput.toDF()
  .select($"_1".as("leftKey"), timestamp_seconds($"_2").as("leftTime"))
  .withWatermark("leftTime", "0 seconds")
val right = rightInput.toDF()
  .select($"_1".as("rightKey"), timestamp_seconds($"_2").as("rightTime"))
  .withWatermark("rightTime", "0 seconds")
val joined = left.join(right, expr("leftKey = rightKey AND leftTime = 
rightTime"), "left_semi")

// batch 1: left (7, 100), right (8, 100)  -> watermark becomes 100 s
// batch 2: left (7, 105), right (8, 105)  -> watermark becomes 105 s
// batch 3: left (1, 102), right (1, 102)  -> late-events watermark 100 s, 
eviction watermark 105 s
{code}

- Expected: batch 3 outputs the left row (1, 102).
- Actual: batch 3 outputs nothing, and later batches do not output it either.

The same happens for a key shifted by an interval, such as {{leftTime + 
INTERVAL 10 SECONDS = rightTime}}, when the watermark advances by more than the 
shift in one batch. For example: left (1, 120) and right (1, 130) in a batch 
whose watermarks are 100 s and 150 s.

How the affected versions were checked:

- The reproducer fails on master in state formats 2, 3 and 4.
- The right-first order and the right-side skip are present in v4.2.0, v4.2.1 
and v4.3.0. That was checked by inspecting {{StreamingSymmetricHashJoinExec}}; 
the reproducer was not run on those tags.
- v4.1.x processes the left input first and is not affected.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to