Tathagata Das created SPARK-24730:
-------------------------------------
Summary: Add policy to choose max as global watermark when
streaming query has multiple watermarks
Key: SPARK-24730
URL: https://issues.apache.org/jira/browse/SPARK-24730
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 2.3.1
Reporter: Tathagata Das
Assignee: Tathagata Das
Currently, when a streaming query has multiple watermark, the policy is to
choose the min of them as the global watermark. This is safe to do as the
global watermark moves with the slowest stream, and is therefore is safe as it
does not unexpectedly drop some data as late, etc. While this is indeed the
safe thing to do, in some cases, you may want the watermark to advance with the
fastest stream, that is, take the max of multiple watermarks. This JIRA is to
add that configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
