Jungtaek Lim created SPARK-61513:
------------------------------------
Summary: Stream-stream join in state format version 4 misses
matches because its range scan window is too narrow
Key: SPARK-61513
URL: https://issues.apache.org/jira/browse/SPARK-61513
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
In state format version 4, a stream-stream join row reads only the other side's
rows in a time window. Spark derives the window's offsets from the join
condition with the watermark solver, at a watermark of 0. Three things make the
window narrower than the condition, so the join does not emit a match, in
either arrival order. Versions 1 to 3 emit it.
# *Casts that drop part of a time.* The solver reads {{CAST(ts AS BIGINT)}} as
{{ts}}, although the cast drops the fraction of a second.
#* Example: {{CAST(rightTime AS BIGINT) >= CAST(leftTime AS BIGINT) + 60 AND
CAST(rightTime AS BIGINT) <= CAST(leftTime AS BIGINT) + 120}}.
#* A left row at 100.6 seconds matches a right row at 160.2 seconds (160 >= 100
+ 60), but the left row's window starts near 160.6 seconds.
#* The same happens for a cast to any integral type, to FLOAT, or to DECIMAL
with fewer than 6 fractional digits.
# *Whole days and months.* The solver counts a day as 24 hours, but Spark adds
days and months in the session time zone, where a day can have 23 or 25 hours.
#* Example: {{rightTime BETWEEN leftTime AND leftTime + INTERVAL 1 DAY}} with
the session time zone America/Los_Angeles.
#* A right row 24.5 hours after a left row at noon on 2026-10-31 matches,
because that day has 25 hours, but the left row's window ends after 24 hours.
# *Sub-millisecond offsets.* The solver returns whole milliseconds, truncated
toward zero.
#* Example: {{rightTime > leftTime - INTERVAL 500 MICROSECONDS AND rightTime <
leftTime + INTERVAL 1 SECOND}}.
#* The lower offset of -0.5 ms becomes 0, so the join misses a right row 0.3 ms
before the left row.
State eviction is not affected by these issues in the cases above.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]