Re: [Structured Streaming]Data processing and output trigger should be decoupled

I think something like state store can be used to keep the intermediate data. For aggregations the engines keeps processing batches of data and update the results in state store(or sth like this), and when a trigger begins the engines just fetch the current result from state store and output it to

Re: [Structured Streaming]Data processing and output trigger should be decoupled

I don't think that's a good idea. If the engine keeps on processing data but doesn't output anything, where to keep the intermediate data? On Wed, Aug 30, 2017 at 9:26 AM, KevinZwx wrote: > Hi, > > I'm working with structured streaming, and I'm wondering whether there >

[Structured Streaming]Data processing and output trigger should be decoupled

Hi, I'm working with structured streaming, and I'm wondering whether there should be some improvements about trigger. Currently, when I specify a trigger, i.e. tigger(Trigger.ProcessingTime("10 minutes")), the engine will begin processing data at the time the trigger begins, like 10:00:00,