Hi Navneeth, You could also apply filtering on the incoming records before windowing. This might save you some development effort but I do not know full details of your requirement whether filtering is sufficient. In general, you can use timers as you suggested as the windowing itself works in a similar way.
Best, Andrey On Thu, Oct 17, 2019 at 11:10 PM Navneeth Krishnan <reachnavnee...@gmail.com> wrote: > Hi All, > > I'm currently using a tumbling window of 5 seconds using > TumblingTimeWindow but due to change in requirements I would not have to > window every incoming data. With that said I'm planning to use process > function to achieve this selective windowing. > > I looked at the example provided in the documentation and I'm not clear on > how I can implement the windowing. > > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html > > Basically what I want is keep collecting data until it reaches 5 seconds > from the time the first data came in for the key and then forward it to the > next operator. I will be using ListState to add the entries and then > register a timer when the list is empty. When the timer runs then collect > all entries and forward it, also remove entries from the list. Do you guys > think this will suffice or anything else has to be done? > > Also I will have about 1M keys, then would there be any performance impact > in creating these many timers? I believe the timers are automatically > removed after they are fired or should I do anything extra to remove these > timers? > > Thanks >
