> Since apparently there isn't a way to iterate through Windowed KTables Im > guessing that this sort of 'aggregate and clear' approach still requires an > external datastore (like Redis). Please correct me if Im wrong.
You don't need an external datastore. You can use state stores for that: http://docs.confluent.io/current/streams/developer-guide.html#state-stores FWIW, a newer question of yours was asking how to access data in state stores "from the outside", which (for the record) I answered by pointing to Kafka's interactive queries feature: http://docs.confluent.io/current/streams/developer-guide.html#interactive-queries -Michael On Wed, Mar 22, 2017 at 10:00 PM, Jon Yeargers <jon.yearg...@cedexis.com> wrote: > I get that the windows are aligned along seconds but this doesn't really > help with true clock alignment (IE top of the hour, midnight, etc). > > I can imagine a strategy using overlapping windows. One would > (hypothetically) walk through the list until a window that spanned the > desired time was found. > > Since apparently there isn't a way to iterate through Windowed KTables Im > guessing that this sort of 'aggregate and clear' approach still requires an > external datastore (like Redis). Please correct me if Im wrong. > > On Mon, Mar 20, 2017 at 9:29 AM, Michael Noll <mich...@confluent.io> > wrote: > > > Jon, > > > > the windowing operation of Kafka's Streams API (in its DSL) aligns > > time-based windows to the epoch [1]: > > > > Quoting from e.g. hopping windows (sometimes called sliding windows in > > other technologies): > > > > > Hopping time windows are aligned to the epoch, with the lower interval > > bound > > > being inclusive and the upper bound being exclusive. “Aligned to the > > epoch” > > > means that the first window starts at timestamp zero. > > > For example, hopping windows with a size of 5000ms and an advance > > interval > > > (“hop”) of 3000ms have predictable window boundaries > > `[0;5000),[3000;8000),...` > > > — and not `[1000;6000),[4000;9000),...` or even something “random” like > > > `[1452;6452),[4452;9452),...`. > > > > Would that help you? > > > > -Michael > > > > > > > > [1] http://docs.confluent.io/current/streams/developer-guide.html > > > > > > On Mon, Mar 20, 2017 at 12:51 PM, Jon Yeargers <jon.yearg...@cedexis.com > > > > wrote: > > > > > Is this possible? Im wondering about gathering data from a stream into > a > > > series of windowed aggregators: minute, hour and day. A separate > process > > > would start at fixed intervals, query the appropriate state store for > > > available values and then hopefully clear / zero / reset everything for > > the > > > next interval. > > > > > > I could use the retention period setting but I would (somehow) need to > > > guarantee that the windows would reset on clock boundaries and not > based > > on > > > start time for the app. > > > > > >
