Hi Sigalit,
First of all, did you ensure different source operator consumes different
consumer id for the kafka source? Did each flink job share the same data or
consumed the data independently?
Moreover, was your job behaves back pressured? It might need to break the
chained operator to see whether the sink back-pressured the source to impact
the throughput of source.
Last but not least, did your source already have 100% CPU usage, which means
your source operator has already reached to its highest throughput.
Best
Yun Tang
From: Sigalit Eliazov
Sent: Thursday, April 7, 2022 19:12
To: user
Subject: flink pipeline handles very small amount of messages in a second (only
500)
hi all
I would appreciate some help to understand the pipeline behaviour...
We deployed a standalone flink cluster. The pipelines are deployed via the jm
rest api.
We have 5 task managers with 1 slot each.
In total i am deploying 5 pipelines which mainly read from kafka, a simple
object conversion and either write back to kafka or GCP pub/sub or save in the
DB.
These jobs run "forever" and basically each task manager runs a specific job
(this is how flink handled it).
We have a test that sends to kafka 10k messages per second. but according to
the metrics exposed by flink i see that the relevant job handles only 500
messages per second.
I would expect all the 10K to be handled. I guess the setup is not correct.
The messages are in avro format
Currently we are not using checkpoints at all.
Any suggestions are welcome.
Thanks alot
Sigalit