Hi Eyal, It’s unclear what warmup phase does in your use cases. Usually we see Flink start consume at high rate and drop to a point downstream can handle.
Thanks Chen > On May 10, 2020, at 12:25 AM, Eyal Pe'er <eyal.p...@startapp.com> wrote: > > Hi all, > Lately I've added more resources to my Flink cluster which required a restart > of all apps. > From the cluster side, the only change I made, is to add more task slots. > On the cluster I have a streaming app that consumes from Kafka and sinks to > files. > I noticed that since the restart, the applications "warmup" has impacted > dramatically. > Before, the change it took few minutes for the app to start and consume > normally (from my point of view, normally is a stable rate) - from 0 to 16K > events per second in 4 minutes. > Now, after the change, it takes hours till it stabilizes on the normal > processing rate- from 0 to 12K events per second in 3 hours. > The data source behavior hasn’t changed (same incoming rate, partitions, > servers etc.). > I am aware to the backpressure mechanism in Flink, but it seems like it works > too slow here. > Is there a way to speed or control it? > > Thanks a lot > Eyal Peer
