Re: How does Spark Structured Streaming determine an event has arrived late?

I see! I get the logic now! On Tue, Feb 27, 2018 at 5:55 PM, naresh Goud wrote: > Hi Kant, > > TD's explanation makes a lot sense. Refer this stackoverflow, where its > was explained with program output. Hope this helps. > >

Re: How does Spark Structured Streaming determine an event has arrived late?

Hi Kant, TD's explanation makes a lot sense. Refer this stackoverflow, where its was explained with program output. Hope this helps. https://stackoverflow.com/questions/45579100/structured-streaming-watermark-vs-exactly-once-semantics Thanks, Naresh www.linkedin.com/in/naresh-dulam

Re: How does Spark Structured Streaming determine an event has arrived late?

Let me answer the original question directly, that is, how do we determine that an event is late. We simply track the maximum event time the engine has seen in the data it has processed till now. And any data that has event time less than the max is basically "late" (as it is out-of-order). Now,