Jungtaek Lim created SPARK-61346:
------------------------------------

             Summary: Derive stream-stream join range eviction from the 
late-filtering event time
                 Key: SPARK-61346
                 URL: https://issues.apache.org/jira/browse/SPARK-61346
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


In the legacy mode 
{{spark.sql.streaming.statefulOperator.allowMultiple=false}}, a stream-stream 
join input can carry several watermarked columns, for example the output of an 
upstream join. Late-input filtering only checks the first watermarked column of 
the opposite input, but the range-eviction bound can be derived from any of 
them.

Example: an upstream join emits {{(u = 100, v = 5)}} after the watermark 
reaches 50. A downstream join filters late input on {{u}} but has the condition 
{{l > v}}. Spark substitutes watermark 50 for {{v}} and evicts {{l = 10}}, 
although the row {{v = 5}} still passes late filtering and matches it. The 
valid match is lost.

The range bound must be derived only from the opposite timestamp that 
late-input filtering actually uses. V4 and the default mode already reject 
multiple distinct event-time columns.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to