Jungtaek Lim created SPARK-61344:
------------------------------------

             Summary: Fix stream-stream join range eviction column and output 
watermark
                 Key: SPARK-61344
                 URL: https://issues.apache.org/jira/browse/SPARK-61344
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


Stream-stream joins can remove state too early and forward an output watermark 
that is too high.

1. Range eviction uses the wrong column. For {{leftKey = rightKey AND rightTime 
BETWEEN leftTime AND expiry}}, a left row can still match until {{expiry}} 
passes the right watermark. Spark applies that cutoff to the left event-time 
column {{leftTime}} instead. The row is removed while it can still match. A 
later valid match is lost, and a left outer join can emit a premature null row. 
This affects state formats V2, V3 and V4.

2. The output watermark ignores what the state can still emit. An inner join on 
a non-time key can keep a left row at event time 5 while the input watermark 
advances to 20, and then match a right row at time 100. The join forwards 
watermark 20, so a downstream deduplication or window aggregate drops the valid 
output at time 5. The same happens when the event-time equality keys are 
crossed, for example {{la = ra AND lb = rb}} with watermarks on {{la}} and 
{{rb}}.

Fixing this changes which rows the state keeps. A checkpoint written by an 
older version may not be safe to reuse for affected join shapes. Spark must 
detect that and reject the checkpoint with a clear error, instead of resuming 
with incorrect state.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to