Siying Dong created SPARK-60850:
-----------------------------------
Summary: Preserve watermark columns when pruning stream-stream
joins
Key: SPARK-60850
URL: https://issues.apache.org/jira/browse/SPARK-60850
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Siying Dong
When both sides of a stream-stream join have watermarks and the downstream
projection only
selects the join key, ColumnPruning can remove the event-time attributes from
both join inputs.
The physical stream-stream join then cannot identify and discard late input
rows, allowing a
late matching pair to be emitted.
The join should retain watermarked input attributes as implicit references
whenever it is
stateful, even when those attributes are not used by the join condition or
selected downstream.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
