Hi, I have setup a Ganglia host to get all the stats from my 5 node kafka cluster. I run jmxtrans on Kafka nodes which export to Ganglia. I have exposed all of the available stats to Ganglia and have grouped them as under:-
Kafka Cluster Stats metrics (18)Kafka Consumer Stats metrics (208)Kafka Controller Stats metrics (23)Kafka Log Stats metrics (72)Kafka Network Stats metrics (1091)Kafka Server Stats metrics (144)cpu metrics (7)disk metrics (3)jvmGC metrics (4)jvmheapmemory metrics (8)load metrics (3)memory metrics (5)network metrics (4)process metrics (2) This kind of setup helps me debug any issues very clearly. Recently, I saw an issue on my cluster where I saw very slow ISR catchup on the last replica. I was able to see that the bytes/sec input to the last replica was around 10% of the other machines - including leader and the other followers. I believe having the current JMX is much better as people can handle these stats in whichever way they want. They could plot any stat vs any stat, but if we limit that to a kafka topic, it will become more complex On the other hand, like Cassandra exposes its stats in Cassandra as well, if one Cassandra node is down, the other will still be able to report. But, the reports will be limited to what Cassandra provides. Having a generic way like JMX reporting allows greater flexibility I would suggest to encourage people to contribute to easily setting up the jmx monitoring pipelines, either using tools like Ganglia or Graphite or Cacti. Something like Kafka clients, which are not maintained by kafka committers. I would love to hear from others on this Regards, Prabhjot On Wed, Sep 23, 2015 at 9:22 AM, James Cheng (JIRA) <j...@apache.org> wrote: > > [ > https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903904#comment-14903904 > ] > > James Cheng commented on KAFKA-2569: > ------------------------------------ > > Yes, although I didn't realize that was a rare thing. I'm pretty new to > the operations-side of Kafka. > > How do people typically monitor things with JMX metrics (like Kafka and > Zookeeper)? > > > Kafka should write its metrics to a Kafka topic > > ----------------------------------------------- > > > > Key: KAFKA-2569 > > URL: https://issues.apache.org/jira/browse/KAFKA-2569 > > Project: Kafka > > Issue Type: New Feature > > Reporter: James Cheng > > > > Kafka is often used to hold and transport monitoring data. > > In order to monitor Kafka itself, Kafka currently exposes many metrics > via JMX, which require using a tool to pull the JMX metrics, and then write > them to the monitoring system. > > It would be convenient if Kafka could simply send its metrics to a Kafka > topic. This would make most sense if the Kafka topic was in a different > Kafka cluster, but could still be useful even if it was sent to a topic in > the same Kafka cluster. > > Of course, if sent to the same cluster, it would not be accessible if > the cluster itself was down. > > This would allow monitoring of Kafka itself without requiring people to > set up their own JMX-to-monitoring-system pipelines. > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"