You could do this using custom triggers and evictors in Flink. That way you can control when the windows fire and what elements are fired with it. And lastly the custom evictors know when to remove elements from the window.
Yes Flink can support it. Sent from my iPhone > On Aug 7, 2019, at 4:19 PM, Yoandy Rodríguez <mr.domi...@gmail.com> wrote: > > Hello everybody, > > We have the following situation: > > 1) A data stream which collects all system events (near 1/2 a mil per day). > > 2) A database storing some aggregation of the data. > > We want to split the data into different "time slices" and be able to > "tag it" accordingly. > > Example: > > the events in the first hour will be tagged as such: > > Time of arrival (slice) Tag > > 0:00:00 - 0:59:59 Last Hour > > 0:30:00 - 0:59:59 Last 1/2 Hour > > 0:50:00 - 0:59:59 Last 10 minutes > > Now, when we reach 1:09:59 the "last ten minutes" tags, moves to that > slice, and so do the other ones. > > Mi initial idea was to have multiple windows operating over the same > stream, but in that case I would have > > to keep a longer window just to remove the tag for events after the 1 > hour period. Is there any way to avoid this? > > > PD. > > This is part of my first Flink project so alternative > solutions/literature are very much welcome > >
