Consumers don't look too busy. Couple java apps and a spark cluster.
Java boxes are not busy, and spark is hard to judge as it does shuffling,
so network is high, but CPU is low.
But thank you for the hint, will check!
Alex
On Thu, Nov 29, 2018 at 10:31 PM Johan Rask wrote:
> What about CPU an
What about CPU and load on your consumers? We had a similar problem with
high cpu and high network traffic on our Kafka cluster and high cpu and
load and traffic on our consumer machines. Thiis was caused by la to ow
setting on kafka consumer fetch.max.wait.ms which was only 5 ms. Setting
this to 1
+1
We too face similar issue sometime. I am interested to know more about it.
On Thursday, November 29, 2018, Alexander Filipchik
wrote:
> Hello!
>
> I inherited Kafka cluster which runs on AWS (I3.4xl instances).
> Each node ingests around 8k messages per second.
> Instance stats:
> Network in
Hello!
I inherited Kafka cluster which runs on AWS (I3.4xl instances).
Each node ingests around 8k messages per second.
Instance stats:
Network in is around: 13 MB/s
Network out is around: 50 MB/s
Kafka stats show slightly different picture:
counter-bytes-in: 2.7 MB/s
counter-bytes-out: 2.9 MB/s
