Re: [Structured Streaming] Trying to use Spark structured streaming

2017-09-11 Thread Eduardo D'Avila
ark 2.2 for this IIRC), with a window duration of 5 minutes and no > sliding interval, and a processing time trigger of 1 minute. Note that this > still doesn't guarantee 1 output row every trigger as late data may arrive > (unless you set the watermark accordingly). > > > Bes

[Structured Streaming] Trying to use Spark structured streaming

2017-09-11 Thread Eduardo D'Avila
Hi, I'm trying to use Spark 2.1.1 structured streaming to *count the number of records* from Kafka *for each time window* with the code in this GitHub gist . I expected that, *once each minute* (the slide duration), it would *outp