[
https://issues.apache.org/jira/browse/SPARK-60375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-60375:
---------------------------------
Affects Version/s: 4.4.0
(was: 4.3.0)
> Base left semi join output watermarks on emitted event time
> -----------------------------------------------------------
>
> Key: SPARK-60375
> URL: https://issues.apache.org/jira/browse/SPARK-60375
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> A stream-stream left semi join emits only left-side columns. Its output
> watermark should therefore describe the event time carried by those columns.
>
> The current calculation includes both state watermarks. Including the
> right-state watermark can unnecessarily delay downstream window completion
> and state cleanup, even though right-side timestamps are never emitted.
>
> Update propagation as follows: * When the output contains a watermark-bearing
> column, use the minimum of the left state watermark and input watermark.
> * When no output column carries watermark metadata, report no output
> watermark.
>
> For example, with {{{}L > R - INTERVAL 5 SECONDS AND L < R + INTERVAL 30
> SECONDS{}}}, an input watermark of 120 seconds yields left/right state bounds
> of 115/90 seconds. Using 115 seconds allows a downstream aggregation to
> complete window {{{}[100, 110){}}}; using 90 seconds unnecessarily leaves it
> buffered.
>
> Join matching, state eviction, and checkpoint layout remain unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
