Hi Sigalit,
In your settings, I guess each job will only have one slot (parallelism). So is it too many input for your jobs with parallelism only one? One easy way to confirm is that you increase your slots and job parallelism twice and then see whether the QPS is increased. Hope this would help you. Yuan > On 7 Apr 2022, at 7:12 PM, Sigalit Eliazov <e.siga...@gmail.com> wrote: > > hi all > > I would appreciate some help to understand the pipeline behaviour... > > We deployed a standalone flink cluster. The pipelines are deployed via the jm > rest api. > We have 5 task managers with 1 slot each. > > In total i am deploying 5 pipelines which mainly read from kafka, a simple > object conversion and either write back to kafka or GCP pub/sub or save in > the DB. > > These jobs run "forever" and basically each task manager runs a specific job > (this is how flink handled it). > > We have a test that sends to kafka 10k messages per second. but according to > the metrics exposed by flink i see that the relevant job handles only 500 > messages per second. > I would expect all the 10K to be handled. I guess the setup is not correct. > > The messages are in avro format > Currently we are not using checkpoints at all. > Any suggestions are welcome. > > Thanks alot > Sigalit > > >
