Re: few basic questions on structured streaming

2016-12-08 Thread Michael Armbrust
> > 1. what happens if an event arrives few days late? Looks like we have an > unbound table with sorted time intervals as keys but I assume spark doesn't > keep several days worth of data in memory but rather it would checkpoint > parts of the unbound table to a storage at a specified interval

RE: few basic questions on structured streaming

2016-12-08 Thread Mendelson, Assaf
For watermarking you can read this excellent article: part 1: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101, part2: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102. It explains more than just watermarking but it helped me understand a lot of the concepts