Jungtaek Lim created SPARK-61346:
------------------------------------
Summary: Derive stream-stream join range eviction from the
late-filtering event time
Key: SPARK-61346
URL: https://issues.apache.org/jira/browse/SPARK-61346
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
In the legacy mode
{{spark.sql.streaming.statefulOperator.allowMultiple=false}}, a stream-stream
join input can carry several watermarked columns, for example the output of an
upstream join. Late-input filtering only checks the first watermarked column of
the opposite input, but the range-eviction bound can be derived from any of
them.
Example: an upstream join emits {{(u = 100, v = 5)}} after the watermark
reaches 50. A downstream join filters late input on {{u}} but has the condition
{{l > v}}. Spark substitutes watermark 50 for {{v}} and evicts {{l = 10}},
although the row {{v = 5}} still passes late filtering and matches it. The
valid match is lost.
The range bound must be derived only from the opposite timestamp that
late-input filtering actually uses. V4 and the default mode already reject
multiple distinct event-time columns.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]