Re: Is the trigger interval the same as batch interval in structured streaming?

Perfect! Thanks a lot. On Mon, Apr 10, 2017 at 1:39 PM, Tathagata Das wrote: > The trigger interval is optionally specified in the writeStream option > before start. > > val windowedCounts = words.groupBy( > window($"timestamp", "24 hours", "24 hours"), >

Re: Is the trigger interval the same as batch interval in structured streaming?

The trigger interval is optionally specified in the writeStream option before start. val windowedCounts = words.groupBy( window($"timestamp", "24 hours", "24 hours"), $"word" ).count() .writeStream .trigger(ProcessingTime("10 seconds")) // optional .format("memory") .queryName("tableName")

Re: Is the trigger interval the same as batch interval in structured streaming?

Thanks again! Looks like the update mode is not available in 2.1 (which seems to be the latest version as of today) and I am assuming there will be a way to specify trigger interval with the next release because with the following code I don't see a way to specify trigger interval. val

Re: Is the trigger interval the same as batch interval in structured streaming?

It sounds like you want a tumbling window (where the slide and duration are the same). This is the default if you give only one interval. You should set the output mode to "update" (i.e. output only the rows that have been updated since the last trigger) and the trigger to "1 second". Try

Re: Is the trigger interval the same as batch interval in structured streaming?

Hi Michael, Thanks for the response. I guess I was thinking more in terms of the regular streaming model. so In this case I am little confused what my window interval and slide interval be for the following case? I need to hold a state (say a count) for 24 hours while capturing all its updates

Re: Is the trigger interval the same as batch interval in structured streaming?

Nope, structured streaming eliminates the limitation that micro-batching should affect the results of your streaming query. Trigger is just an indication of how often you want to produce results (and if you leave it blank we just run as quickly as possible). To control how tuples are grouped

Is the trigger interval the same as batch interval in structured streaming?

Hi All, Is the trigger interval mentioned in this doc the same as batch interval in structured streaming? For example I have a long running receiver(not kafka) which sends me a real time stream I want to use window