I forgot to mention, we are consuming said records from AWS kinesis and writing out to S3.
From: Senthil Kumar <senthi...@vmware.com> Date: Tuesday, May 12, 2020 at 10:47 AM To: "user@flink.apache.org" <user@flink.apache.org> Subject: Flink Streaming Job Tuning help Hello Flink Community! We have a fairly intensive flink streaming application, processing 8-9 million records a minute, with each record being 10k. One of our steps is a keyBy operation. We are finding that flink lags seriously behind when we introduce the keyBy (presumably because of shuffle across the network). We are trying to tune it ourselves (size of nodes, memory, network buffers etc), but before we spend way too much time on this; would it be better to hire some “flink tuning expert” to get us through? If so what resources are recommended on this list? Cheers Kumar
