Hi Dominik, I see, thanks for explaining the diagram.
This is expected because the 1 minute window in your case is aligned with the beginning of every minute. For example, if the first element element comes at 12:10:45, then the element will be put in the window of 12:10:00 to 12:10:59. Therefore, it will fire after 14 seconds instead of 1 minute. Does that explain what you are experiencing? Cheers, Gordon On March 31, 2017 at 3:06:56 AM, Dominik Safaric (dominiksafa...@gmail.com) wrote: First, some remarks here - sources (in your case the Kafka consumer) will not stop fetching / producing data when the windows haven’t fired yet. This is for sure true. However, the plot shows the number of records produced per second, where each record was assigned a created at timestamp while being created and before being pushed back to Kafka. Sorry I did not clarify this before. Anyway, because of this I would expect to have a certain lag. Of course, messages will not only be produced into Kafka exactly at window expiry and then the produced shutdown - however, what concerns me is that messages were produced to Kafka before the first window expired - hence the questions. If you’re writing the outputs of the window operation to Kafka (by adding a Kafka sink after the windowing), then yes it should only write to Kafka when the window has fired. Hence, I this behaviour that you’ve described and we’ve expected did not occur. If it would help, I can share the source code and a detail Flink configuration. Cheers, Dominik On 30 Mar 2017, at 13:09, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: Hi, Thanks for the clarification. What are the reasons behind consuming/producing messages from/to Kafka while the window has not expired yet? First, some remarks here - sources (in your case the Kafka consumer) will not stop fetching / producing data when the windows haven’t fired yet. Does this explain what you have plotted in the diagram you attached (sorry, I can’t really reason about the diagram because I’m not so sure what the values of the x-y axes represent)? If you’re writing the outputs of the window operation to Kafka (by adding a Kafka sink after the windowing), then yes it should only write to Kafka when the window has fired. The characteristics will also differ for different types of windows, so you should definitely take a look at the Windowing docs [1] about them. Cheers, Gordon [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#window-assigners On March 30, 2017 at 2:37:41 PM, Dominik Safaric (dominiksafa...@gmail.com) wrote: What are the reasons behind consuming/producing messages from/to Kafka while the window has not expired yet?
