Jungtaek Lim created SPARK-62038:
------------------------------------

             Summary: Stream-stream join state format V4 never evicts rows that 
an older version's cutoff left in state after a restart
                 Key: SPARK-62038
                 URL: https://issues.apache.org/jira/browse/SPARK-62038
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join 
restarted from an older checkpoint keeps state rows that the older version's 
cutoff never evicted, after a fix makes them evictable. On each batch V4 starts 
its eviction scan after the previous batch's cutoff, so rows at or below an old 
cutoff are never scanned again. State formats 2 and 3 evaluate every stored row 
each batch and evict them.

This shows after SPARK-61985, which removes a constant-only cap from the range 
cutoff. Reproducer (left outer join, 1 second watermarks):

{code}
leftKey = rightKey AND lt > rt - INTERVAL 5 SECONDS AND CAST(lt AS LONG) > 10
{code}

1. On a build without SPARK-61985, run L(1, 20 s) R(9, 20 s); L(8, 40 s) R(9, 
40 s); R(9, 41 s). Nothing is emitted, because the cutoff is capped at 10 s.
2. Restart from that checkpoint with SPARK-61985 and add L(8, 60 s) R(9, 60 s).

Expected: {{(1, 20, null)}} in the first batch after the restart. V2 and V3 
emit it. V4 emits nothing, and the row stays in state.

A possible fix: on the first batch after a restart, scan the V4 timestamp index 
from the start instead of after the previous batch's cutoff.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to