Jungtaek Lim created SPARK-61348:
------------------------------------
Summary: Reject numeric state bounds in stream-stream outer joins
Key: SPARK-61348
URL: https://issues.apache.org/jira/browse/SPARK-61348
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
The analyzer accepts a stream-stream outer join whose only bound is on a
numeric state column, for example {{n > CAST(rightTime AS BIGINT)}}. Runtime
range eviction only handles timestamp state columns, so the join can never
finalize its unmatched rows. It keeps state indefinitely and never emits them.
The analyzer should require a timestamp state column when it validates a range
bound, and reject such outer joins.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]