Jungtaek Lim created SPARK-61347:
------------------------------------

             Summary: Evict stream-stream join range state without a watermark 
on the stored side
                 Key: SPARK-61347
                 URL: https://issues.apache.org/jira/browse/SPARK-61347
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


A stream-stream join only infers a range-eviction bound for one side when that 
side has its own watermark.

Example: a left outer join of impressions and clicks with a watermark on 
{{clickTime}} only and the condition {{clickTime BETWEEN impressionTime AND 
impressionTime + INTERVAL 1 HOUR}}. The right watermark and the time range are 
enough to know when a left row can no longer match. Because the left side has 
no watermark, Spark never removes left rows and never emits their unmatched 
outer rows. The right outer case is symmetric.

The same happens when the stored side's event-time column has been pruned from 
the physical join because no downstream operator uses it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to