Jungtaek Lim created SPARK-60375:
------------------------------------
Summary: Base left semi join output watermarks on emitted event
time
Key: SPARK-60375
URL: https://issues.apache.org/jira/browse/SPARK-60375
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.3.0
Reporter: Jungtaek Lim
A stream-stream left semi join emits only left-side columns. Its output
watermark should therefore describe the event time carried by those columns.
The current calculation includes both state watermarks. Including the
right-state watermark can unnecessarily delay downstream window completion and
state cleanup, even though right-side timestamps are never emitted.
Update propagation as follows: * When the output contains a watermark-bearing
column, use the minimum of the left state watermark and input watermark.
* When no output column carries watermark metadata, report no output watermark.
For example, with {{{}L > R - INTERVAL 5 SECONDS AND L < R + INTERVAL 30
SECONDS{}}}, an input watermark of 120 seconds yields left/right state bounds
of 115/90 seconds. Using 115 seconds allows a downstream aggregation to
complete window {{{}[100, 110){}}}; using 90 seconds unnecessarily leaves it
buffered.
Join matching, state eviction, and checkpoint layout remain unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
