Jungtaek Lim created SPARK-61350:
------------------------------------

             Summary: Reject negative watermark terms in stream-stream join 
eviction bounds
                 Key: SPARK-61350
                 URL: https://issues.apache.org/jira/browse/SPARK-61350
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


The eviction-bound inference for stream-stream joins accepts a condition in 
which the opposite watermark appears with a negative sign after normalization.

Example: for {{CAST(leftTime AS BIGINT) + CAST(rightTime AS BIGINT) > 100}}, a 
later right row with a larger {{rightTime}} can match an arbitrarily old left 
row. Spark treats the right watermark as an upper bound and removes left rows 
that can still match. Valid matches are lost in V2, V3 and V4.

A normalized negative opposite-watermark term must not produce an eviction 
bound. Outer and semi joins without any other valid bound must then fail 
analysis.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to