[ 
https://issues.apache.org/jira/browse/SPARK-62002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-62002:
---------------------------------
    Affects Version/s: 4.3.0
                       4.4.0

> Stream-stream join loses matches when an equality key on event time is 
> shifted backward or truncated on one side
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62002
>                 URL: https://issues.apache.org/jira/browse/SPARK-62002
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> When a join key on one side carries a watermark, Spark evicts both inputs' 
> state when that key position falls to the watermark or below. If the other 
> input's key is its event time shifted backward ({{lt = rt - INTERVAL 10 
> SECONDS}}) or truncated ({{lt = CAST(CAST(rt AS DATE) AS TIMESTAMP)}}, or 
> {{date_trunc}}), a row of that input can be on time but still have a key 
> below the watermark. Its match on the first input has already been evicted. 
> Inner joins lose the match. Outer joins emit an early null row and then lose 
> the match.
> Reproducer (10-second watermarks on {{lt}} and {{rt}}):
> {code:scala}
> left.join(right, expr("lk = rk AND lt = rt - INTERVAL 10 SECONDS"))
> {code}
> Left (1, 95 s). Rows at 110 s move the watermark to 100 s, and Spark evicts 
> the left row. Right (1, 105 s) is on time and should match it (95 = 105 - 
> 10), but Spark emits nothing. A left outer join emits (1, 95, null) instead.
> With {{lt = CAST(CAST(rt AS DATE) AS TIMESTAMP)}}, left (1, 1970-01-02 00:00) 
> is evicted at watermark 00:00:05, and right (1, 02:00 the same day) finds no 
> match.
> Affects state format versions 2 to 4. Related to SPARK-61381 (inexact bounds) 
> and SPARK-61989 (equality-key eviction).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to