Arun Mahadevan created SPARK-26655:
--------------------------------------
Summary: Support multiple aggregates in Structured Streaming
append mode
Key: SPARK-26655
URL: https://issues.apache.org/jira/browse/SPARK-26655
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.0.0
Reporter: Arun Mahadevan
Right now multiple aggregates are not supported in structured streaming.
However, in append mode, the aggregates are emitted only after the watermark
passes the threshold (e.g. the window boundary) and the emitted value is not
affected by further late data. So it possible to chain multiple aggregates in
'Append' output mode without worrying about retractions.
However the current event time watermarks in structured streaming are tracked
at a global level and this does not work when aggregates are chained.
We need to track the watermarks at individual operator level so that each
operator can make progress independently and not rely on global min or max
value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
