[ 
https://issues.apache.org/jira/browse/SPARK-61345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61345:
-----------------------------------
    Labels: pull-request-available  (was: )

> Fix stream-stream join watermark direction and equality-key eviction
> --------------------------------------------------------------------
>
>                 Key: SPARK-61345
>                 URL: https://issues.apache.org/jira/browse/SPARK-61345
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 5.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> Stream-stream joins check the wrong watermark direction and evict 
> equality-key state with the wrong side's watermark.
> 1. Admission. A left outer or left semi join finalizes left rows, so it needs 
> a watermark on the right input and a condition that limits how late a 
> matching right row can arrive. The analyzer accepts such joins with a 
> watermark on the left input only. They keep unmatched state indefinitely and 
> never emit outer rows. Full outer joins need a bound in both directions.
> 2. Equality-key eviction. The key-eviction predicate and the V4 timestamp 
> index use one equality-key ordinal for both sides. When the event-time keys 
> sit at different ordinals, Spark can remove a row whose match is still to 
> come and emit a premature outer null row. A key that is the opposite event 
> time shifted by a constant interval, such as {{lt = rt + INTERVAL 10 
> SECONDS}}, is also not handled. A truncating key such as {{CAST(CAST(rt AS 
> BIGINT) AS TIMESTAMP)}} must not get a bound, because a later row can 
> truncate below the watermark.
> 3. Null keys in V4. Reading a null selected window as a struct throws a 
> NullPointerException. Indexing a null key under another timestamp can remove 
> the row or emit an outer row too early.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to