[ 
https://issues.apache.org/jira/browse/SPARK-61514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-61514:
---------------------------------
    Affects Version/s: 4.4.0

> Derive V4 stream-stream join scan windows from the incoming row
> ---------------------------------------------------------------
>
>                 Key: SPARK-61514
>                 URL: https://issues.apache.org/jira/browse/SPARK-61514
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> In state format version 4, a stream-stream join stores each side's rows 
> ordered by a timestamp. A new row reads the other side's rows for its key. 
> When the join condition bounds the two event times by constant offsets, it 
> reads only the rows in a time window. In every other case it reads all rows 
> for the key.
> A common interval join bounds the time by another column, as in Nexmark q9: 
> {{bid.dateTime BETWEEN auction.dateTime AND auction.expires}}. The auction 
> side then evicts on {{expires}}, so its state is ordered by {{expires}}, not 
> by its event time. Spark turns off the window for both sides, and every row 
> reads all rows for its key. Such queries can be slower in V4 than in V2.
> Proposal: each side bounds the column that the other side's state is ordered 
> by. It uses the comparisons of the join condition on that column, evaluated 
> on the new row.
> * An auction row reads bids with {{dateTime}} between its {{dateTime}} and 
> its {{expires}}.
> * A bid row reads auctions with {{expires}} at or after its {{dateTime}}. 
> This needs a range scan with no upper end.
> A bound must follow from the condition, so it never skips a row that the 
> condition matches. Shifts by whole days or months, casts other than to 
> BIGINT, and other functions of the stored column give no bound. The 
> constant-offset window also missed matches for whole-day intervals across a 
> daylight saving time change, because it counts a day as 24 hours; the new 
> windows fix that too.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to