n email]
> <http:///user/SendEmail.jtp?type=node=19600=0>]
> *Sent:* Thursday, October 27, 2016 10:17 AM
> *To:* Mendelson, Assaf
> *Subject:* Re: Watermarking in Structured Streaming to drop late data
>
>
>
> Hi all
>
> I would highly recommend to all users-dev
papageorgopoylos [via Apache Spark Developers List]
[mailto:ml-node+s1001551n19592...@n3.nabble.com]
Sent: Thursday, October 27, 2016 10:17 AM
To: Mendelson, Assaf
Subject: Re: Watermarking in Structured Streaming to drop late data
Hi all
I would highly recommend to all users-devs interested
;> To enable the user to specify details like lateness threshold, we are
>> considering adding a new method to Dataset. We would like to get more
>> feedback on this API. Here is the design doc
>>
>>
>>
>> https://docs.google.com/document/d/1z-Pazs5v4rA31azvmYhu4I5x
>
s the entire
> aggregation solve this?
>
> Am I missing something here?
>
>
>
> *From:* Michael Armbrust [via Apache Spark Developers List] [mailto:
> ml-node+[hidden email]
> <http:///user/SendEmail.jtp?type=node=19591=0>]
> *Sent:* Thursday, October 27, 2016 3
this?
>
> Am I missing something here?
>
>
>
> *From:* Michael Armbrust [via Apache Spark Developers List] [mailto:
> ml-node+[hidden email]
> <http:///user/SendEmail.jtp?type=node=19591=0>]
> *Sent:* Thursday, October 27, 2016 3:04 AM
> *To:* Mendelson, Assaf
Apache Spark Developers List]
[mailto:ml-node+s1001551n19590...@n3.nabble.com]
Sent: Thursday, October 27, 2016 3:04 AM
To: Mendelson, Assaf
Subject: Re: Watermarking in Structured Streaming to drop late data
And the JIRA: https://issues.apache.org/jira/browse/SPARK-18124
On Wed, Oct 26, 2016 at 4
And the JIRA: https://issues.apache.org/jira/browse/SPARK-18124
On Wed, Oct 26, 2016 at 4:56 PM, Tathagata Das wrote:
> Hey all,
>
> We are planning implement watermarking in Structured Streaming that would
> allow us handle late, out-of-order data better. Specially, when
Hey all,
We are planning implement watermarking in Structured Streaming that would
allow us handle late, out-of-order data better. Specially, when we are
aggregating over windows on event-time, we currently can end up keeping
unbounded amount data as state. We want to define watermarks on the