Jungtaek Lim created SPARK-61514:
------------------------------------

             Summary: Derive V4 stream-stream join scan windows from the 
incoming row
                 Key: SPARK-61514
                 URL: https://issues.apache.org/jira/browse/SPARK-61514
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


In state format version 4, a stream-stream join stores each side's rows ordered 
by a timestamp. A new row reads the other side's rows for its key. When the 
join condition bounds the two event times by constant offsets, it reads only 
the rows in a time window. In every other case it reads all rows for the key.

A common interval join bounds the time by another column, as in Nexmark q9: 
{{bid.dateTime BETWEEN auction.dateTime AND auction.expires}}. The auction side 
then evicts on {{expires}}, so its state is ordered by {{expires}}, not by its 
event time. Spark turns off the window for both sides, and every row reads all 
rows for its key. Such queries can be slower in V4 than in V2.

Proposal: each side bounds the column that the other side's state is ordered 
by. It uses the comparisons of the join condition on that column, evaluated on 
the new row.
* An auction row reads bids with {{dateTime}} between its {{dateTime}} and its 
{{expires}}.
* A bid row reads auctions with {{expires}} at or after its {{dateTime}}. This 
needs a range scan with no upper end.

A bound must follow from the condition, so it never skips a row that the 
condition matches. Shifts by whole days or months, casts other than to BIGINT, 
and other functions of the stored column give no bound. The constant-offset 
window also missed matches for whole-day intervals across a daylight saving 
time change, because it counts a day as 24 hours; the new windows fix that too.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to