Re: few basic questions on structured streaming

2016-12-08 Thread Michael Armbrust
> > 1. what happens if an event arrives few days late? Looks like we have an > unbound table with sorted time intervals as keys but I assume spark doesn't > keep several days worth of data in memory but rather it would checkpoint > parts of the unbound table to a storage at a specified interval

RE: few basic questions on structured streaming

2016-12-08 Thread Mendelson, Assaf
questions on structured streaming Hi All, I read the documentation on Structured Streaming based on event time and I have the following questions. 1. what happens if an event arrives few days late? Looks like we have an unbound table with sorted time intervals as keys but I assume spark doesn't keep

few basic questions on structured streaming

2016-12-08 Thread kant kodali
Hi All, I read the documentation on Structured Streaming based on event time and I have the following questions. 1. what happens if an event arrives few days late? Looks like we have an unbound table with sorted time intervals as keys but I assume spark doesn't keep several days worth of data in