Re: clearing an aggregation?

2017-03-23 Thread Michael Noll
> Since apparently there isn't a way to iterate through Windowed KTables Im > guessing that this sort of 'aggregate and clear' approach still requires an > external datastore (like Redis). Please correct me if Im wrong. You don't need an external datastore. You can use state stores for that: http

Re: clearing an aggregation?

2017-03-22 Thread Jon Yeargers
I get that the windows are aligned along seconds but this doesn't really help with true clock alignment (IE top of the hour, midnight, etc). I can imagine a strategy using overlapping windows. One would (hypothetically) walk through the list until a window that spanned the desired time was found.

Re: clearing an aggregation?

2017-03-20 Thread Michael Noll
Jon, the windowing operation of Kafka's Streams API (in its DSL) aligns time-based windows to the epoch [1]: Quoting from e.g. hopping windows (sometimes called sliding windows in other technologies): > Hopping time windows are aligned to the epoch, with the lower interval bound > being inclusiv

clearing an aggregation?

2017-03-20 Thread Jon Yeargers
Is this possible? Im wondering about gathering data from a stream into a series of windowed aggregators: minute, hour and day. A separate process would start at fixed intervals, query the appropriate state store for available values and then hopefully clear / zero / reset everything for the next in