[
https://issues.apache.org/jira/browse/SPARK-62044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-62044:
---------------------------------
Affects Version/s: 4.3.0
> Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the
> event time, losing matches
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-62044
> URL: https://issues.apache.org/jira/browse/SPARK-62044
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> A stream-stream join can bound one input's state through a column that is
> cast to TIMESTAMP,
> such as a DATE, TIMESTAMP_NTZ or numeric column. Spark solves the cutoff for
> the cast column, but
> then evicts rows by the input's event time with that cutoff. So it drops rows
> that later rows of
> the other input can still match. An inner join loses those matches. A left
> outer join emits the
> row with nulls too early and also loses the match.
> Example, with 1-hour watermarks on {{leftTime}} and {{rightTime}} and session
> time zone UTC:
> {code:sql}
> SELECT * FROM l JOIN r
> ON leftKey = rightKey
> AND rightTime BETWEEN leftTime AND CAST(leftDate AS TIMESTAMP)
> {code}
> ||Step||Input||Spark today||Expected||
> |1|left (1, leftTime = day 1, leftDate = day 5); right (9, day 1)|-|-|
> |2|left (8, day 4 01:00, day 9); right (9, day 4 01:00)|-|-|
> |3|right (9, day 4 01:00) (watermark now day 4)|left row 1 is evicted; a left
> outer join emits {{(1, day 1, null)}}|- (the row can still match until day 5)|
> |4|right (1, day 4 12:00)|-|{{(1, day 1, day 4 12:00)}}|
> The eviction predicate in step 3 is {{leftTime <= day 4}}, a cutoff solved
> from {{leftDate}} but
> applied to {{leftTime}}. Same result with state format versions 2 and 4,
> inner and left outer.
> A second symptom: an outer join whose preserved input has no watermark and is
> bounded only through
> such a cast column never emits that input's unmatched rows. Example: a left
> outer join with a
> watermark only on {{rightTime}} and
> {{leftKey = rightKey AND CAST(leftDate AS TIMESTAMP) BETWEEN rightTime -
> INTERVAL 1 DAY AND rightTime}}.
> Left {{(1, day 1)}} is never emitted, and its row stays in state, however far
> the watermark moves.
> h2. Fix outline
> Evict by the cast expression that the condition bounds, not by the event
> time. The cutoff is
> already in the cast's units (microseconds of a TIMESTAMP), so the eviction
> predicate becomes
> {{CAST(leftDate AS TIMESTAMP) <= cutoff}}, and a NULL cast value is evicted
> too. State format 4
> would order its index by the cast value. With this, the input without a
> watermark in the second
> example is also evicted, and its unmatched rows are emitted. Note that DATE
> and TIMESTAMP_NTZ
> casts depend on the session time zone.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]