[
https://issues.apache.org/jira/browse/SPARK-61514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61514:
-----------------------------------
Labels: pull-request-available (was: )
> Derive V4 stream-stream join scan windows from the incoming row
> ---------------------------------------------------------------
>
> Key: SPARK-61514
> URL: https://issues.apache.org/jira/browse/SPARK-61514
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> In state format version 4, a stream-stream join stores each side's rows
> ordered by a timestamp. A new row reads the other side's rows for its key.
> When the join condition bounds the two event times by constant offsets, it
> reads only the rows in a time window. In every other case it reads all rows
> for the key.
> A common interval join bounds the time by another column, as in Nexmark q9:
> {{bid.dateTime BETWEEN auction.dateTime AND auction.expires}}. The auction
> side then evicts on {{expires}}, so its state is ordered by {{expires}}, not
> by its event time. Spark turns off the window for both sides, and every row
> reads all rows for its key. Such queries can be slower in V4 than in V2.
> Proposal: each side bounds the column that the other side's state is ordered
> by. It uses the comparisons of the join condition on that column, evaluated
> on the new row.
> * An auction row reads bids with {{dateTime}} between its {{dateTime}} and
> its {{expires}}.
> * A bid row reads auctions with {{expires}} at or after its {{dateTime}}.
> This needs a range scan with no upper end.
> A bound must follow from the condition, so it never skips a row that the
> condition matches. Shifts by whole days or months, casts other than to
> BIGINT, and other functions of the stored column give no bound. The
> constant-offset window also missed matches for whole-day intervals across a
> daylight saving time change, because it counts a day as 24 hours; the new
> windows fix that too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]