Re: [Structured Streaming] Trying to use Spark structured streaming

2017-09-11 Thread Eduardo D'Avila
est, > Burak > > > On Mon, Sep 11, 2017 at 8:04 AM, Eduardo D'Avila < > eduardo.dav...@corp.globo.com> wrote: > >> Hi, >> >> I'm trying to use Spark 2.1.1 structured streaming to *count the number >> of records* from Kafka *for each time window* with

Re: [Structured Streaming] Trying to use Spark structured streaming

2017-09-11 Thread Burak Yavuz
rive (unless you set the watermark accordingly). Best, Burak On Mon, Sep 11, 2017 at 8:04 AM, Eduardo D'Avila < eduardo.dav...@corp.globo.com> wrote: > Hi, > > I'm trying to use Spark 2.1.1 structured streaming to *count the number > of records* from Kafka *for each time windo

[Structured Streaming] Trying to use Spark structured streaming

2017-09-11 Thread Eduardo D'Avila
Hi, I'm trying to use Spark 2.1.1 structured streaming to *count the number of records* from Kafka *for each time window* with the code in this GitHub gist <https://gist.github.com/erdavila/b6ab0c216e82ae77fa8192c48cb816e4>. I expected that, *once each minute* (the slide duration), it