[
https://issues.apache.org/jira/browse/SPARK-60862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-60862:
---------------------------------
Affects Version/s: 4.4.0
> Watermark propagation through reused exchanges uses zero watermark
> ------------------------------------------------------------------
>
> Key: SPARK-60862
> URL: https://issues.apache.org/jira/browse/SPARK-60862
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Major
> Labels: pull-request-available
>
> ReusedExchangeExec is a leaf physical plan node whose underlying exchange is
> stored in its
> child field rather than in the normal children sequence.
> PropagateWatermarkSimulator does not
> currently unwrap this node when traversing the plan or resolving an input
> watermark.
> As a result, when equivalent streaming aggregation branches reuse an
> exchange, the direct
> branch receives the correct event-time watermark while the reused branch
> remains pinned at
> zero. Downstream stateful operators also receive zero. In append mode, the
> affected branch may
> never emit closed windows and its state can grow without bound, causing wrong
> results and
> unbounded state growth.
> The fix treats ReusedExchangeExec as a transparent wrapper in both watermark
> traversal and
> input-watermark lookup, with a guard against re-traversing a hidden child
> already visited
> through the canonical exchange. A streaming SQL regression test covers a
> union of two windowed
> aggregation branches and verifies that both branches emit their closed
> windows after the
> watermark advances.
> Pull request: https://github.com/apache/spark/pull/60073
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
