[
https://issues.apache.org/jira/browse/SPARK-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das resolved SPARK-18124.
-----------------------------------
Resolution: Fixed
Fix Version/s: 2.1.0
Issue resolved by pull request 15702
[https://github.com/apache/spark/pull/15702]
> Observed delay based event time watermarks
> ------------------------------------------
>
> Key: SPARK-18124
> URL: https://issues.apache.org/jira/browse/SPARK-18124
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Reporter: Tathagata Das
> Assignee: Michael Armbrust
> Fix For: 2.1.0
>
>
> Whenever we aggregate data by event time, we want to consider data is late
> and out-of-order in terms of its event time. Since we keep aggregate keyed by
> the time as state, the state will grow unbounded if we keep around all old
> aggregates in an attempt consider arbitrarily late data. Since the state is a
> store in-memory, we have to prevent building up of this unbounded state.
> Hence, we need a watermarking mechanism by which we will mark data that is
> older beyond a threshold as “too late”, and stop updating the aggregates with
> them. This would allow us to remove old aggregates that are never going to be
> updated, thus bounding the size of the state.
> Here is the design doc -
> https://docs.google.com/document/d/1z-Pazs5v4rA31azvmYhu4I5xwqaNQl6ZLIS03xhkfCQ/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
