I think something like state store can be used to keep the intermediate
data. For aggregations the engines keeps processing batches of data and
update the results in state store(or sth like this), and when a trigger
begins the engines just fetch the current result from state store and
output it to
I don't think that's a good idea. If the engine keeps on processing data
but doesn't output anything, where to keep the intermediate data?
On Wed, Aug 30, 2017 at 9:26 AM, KevinZwx wrote:
> Hi,
>
> I'm working with structured streaming, and I'm wondering whether there
> should be some improvemen
Hi,
I'm working with structured streaming, and I'm wondering whether there
should be some improvements about trigger.
Currently, when I specify a trigger, i.e. tigger(Trigger.ProcessingTime("10
minutes")), the engine will begin processing data at the time the trigger
begins, like 10:00:00, 10:10:
