Jungtaek Lim created SPARK-61345:
------------------------------------
Summary: Fix stream-stream join watermark direction and
equality-key eviction
Key: SPARK-61345
URL: https://issues.apache.org/jira/browse/SPARK-61345
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
Stream-stream joins check the wrong watermark direction and evict equality-key
state with the wrong side's watermark.
1. Admission. A left outer or left semi join finalizes left rows, so it needs a
watermark on the right input and a condition that limits how late a matching
right row can arrive. The analyzer accepts such joins with a watermark on the
left input only. They keep unmatched state indefinitely and never emit outer
rows. Full outer joins need a bound in both directions.
2. Equality-key eviction. The key-eviction predicate and the V4 timestamp index
use one equality-key ordinal for both sides. When the event-time keys sit at
different ordinals, Spark can remove a row whose match is still to come and
emit a premature outer null row. A key that is the opposite event time shifted
by a constant interval, such as {{lt = rt + INTERVAL 10 SECONDS}}, is also not
handled. A truncating key such as {{CAST(CAST(rt AS BIGINT) AS TIMESTAMP)}}
must not get a bound, because a later row can truncate below the watermark.
3. Null keys in V4. Reading a null selected window as a struct throws a
NullPointerException. Indexing a null key under another timestamp can remove
the row or emit an outer row too early.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]