Hi Bowen, (very nice graphics :) )
I don't think you can do anything about the windows itself (unless you are able to build the windows yourself using the ProcessFunction, playing some tricks because you know your data), so I should focus on reducing the pain in "burning down your sink". Are there any issues with the Sink by the spikes? (What's the downstream system?) Does it make sense for you to artificially throttle down the stream before the sink, so that the records per second get limited to a certain rate. Since you are using Event time, the window results will always be correct & consistent. From a business perspective, this will of course introduce additional latency (= results come in later). On Fri, Aug 25, 2017 at 6:23 AM, Bowen Li <bowen...@offerupnow.com> wrote: > Hi guys, > > I do have a question for how Flink generates windows. > > We are using a 1-day sized sliding window with 1-hour slide to count some > features of items based on event time. We have about 20million items. We > observed that Flink only emit results on a fixed time in an hour (e.g. 1am, > 2am, 3am, or 1:15am, 2:15am, 3:15am with a 15min offset). That's means > 20million windows/records are generated at the same time every hour, which > burns down our sink. But nothing is generated in the rest of that hour. The > pattern is like this: > > # generated windows > | > | /\ /\ > | / \ / \ > |_/__\_______/__\_ > time > > Is there any way to even out the number of generated windows/records in an > hour? Can we have evenly distributed generated load like this? > > # generated windows > | > | > | ------------------------ > |_______________ > time > > Thanks, > Bowen > >
