[jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[UTKARSH BHATNAGAR (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906189#comment-14906189
 ] 

UTKARSH BHATNAGAR commented on KAFKA-2569:
--

[~otis] & [~wushujames] - This is the use case which I faced a couple of months 
ago. I implemented JMXTrans KafkaWriter to send JMX Metrics to Kafka. Here is 
the link:

https://github.com/jmxtrans/jmxtrans/tree/master/jmxtrans-output/jmxtrans-output-kafka

So, just install JMXTrans on Kafka instances and send Kafka metrics to any 
Kafka(itself or another one). Hope this helps. Please let me know if there are 
questions.

> Kafka should write its metrics to a Kafka topic
> ---
>
> Key: KAFKA-2569
> URL: https://issues.apache.org/jira/browse/KAFKA-2569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: James Cheng
>
> Kafka is often used to hold and transport monitoring data.
> In order to monitor Kafka itself, Kafka currently exposes many metrics via 
> JMX, which require using a tool to pull the JMX metrics, and then write them 
> to the monitoring system.
> It would be convenient if Kafka could simply send its metrics to a Kafka 
> topic. This would make most sense if the Kafka topic was in a different Kafka 
> cluster, but could still be useful even if it was sent to a topic in the same 
> Kafka cluster.
> Of course, if sent to the same cluster, it would not be accessible if the 
> cluster itself was down.
> This would allow monitoring of Kafka itself without requiring people to set 
> up their own JMX-to-monitoring-system pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[Otis Gospodnetic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905760#comment-14905760
 ] 

Otis Gospodnetic commented on KAFKA-2569:
-

bq. How do people typically monitor things with JMX metrics (like Kafka and 
Zookeeper)?

They use tools like SPM for Kafka / ZK / etc., or home-grown stuff.  See 
http://sematext.com/spm/integrations/kafka-monitoring.html for example.


> Kafka should write its metrics to a Kafka topic
> ---
>
> Key: KAFKA-2569
> URL: https://issues.apache.org/jira/browse/KAFKA-2569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: James Cheng
>
> Kafka is often used to hold and transport monitoring data.
> In order to monitor Kafka itself, Kafka currently exposes many metrics via 
> JMX, which require using a tool to pull the JMX metrics, and then write them 
> to the monitoring system.
> It would be convenient if Kafka could simply send its metrics to a Kafka 
> topic. This would make most sense if the Kafka topic was in a different Kafka 
> cluster, but could still be useful even if it was sent to a topic in the same 
> Kafka cluster.
> Of course, if sent to the same cluster, it would not be accessible if the 
> cluster itself was down.
> This would allow monitoring of Kafka itself without requiring people to set 
> up their own JMX-to-monitoring-system pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903904#comment-14903904
 ] 

James Cheng commented on KAFKA-2569:


Yes, although I didn't realize that was a rare thing. I'm pretty new to the 
operations-side of Kafka.

How do people typically monitor things with JMX metrics (like Kafka and 
Zookeeper)? 

> Kafka should write its metrics to a Kafka topic
> ---
>
> Key: KAFKA-2569
> URL: https://issues.apache.org/jira/browse/KAFKA-2569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: James Cheng
>
> Kafka is often used to hold and transport monitoring data.
> In order to monitor Kafka itself, Kafka currently exposes many metrics via 
> JMX, which require using a tool to pull the JMX metrics, and then write them 
> to the monitoring system.
> It would be convenient if Kafka could simply send its metrics to a Kafka 
> topic. This would make most sense if the Kafka topic was in a different Kafka 
> cluster, but could still be useful even if it was sent to a topic in the same 
> Kafka cluster.
> Of course, if sent to the same cluster, it would not be accessible if the 
> cluster itself was down.
> This would allow monitoring of Kafka itself without requiring people to set 
> up their own JMX-to-monitoring-system pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[Prabhjot Bharaj]

Hi,

I have setup a Ganglia host to get all the stats from my 5 node kafka
cluster. I run jmxtrans on Kafka nodes which export to Ganglia.
I have exposed all of the available stats to Ganglia and have grouped them
as under:-

Kafka Cluster Stats metrics (18)Kafka Consumer Stats metrics (208)Kafka
Controller Stats metrics (23)Kafka Log Stats metrics (72)Kafka Network
Stats metrics (1091)Kafka Server Stats metrics (144)cpu metrics (7)disk
metrics (3)jvmGC metrics (4)jvmheapmemory metrics (8)load metrics (3)memory
metrics (5)network metrics (4)process metrics (2)


This kind of setup helps me debug any issues very clearly.
Recently, I saw an issue on my cluster where I saw very slow ISR catchup on
the last replica. I was able to see that the bytes/sec input to the last
replica was around 10% of the other machines - including leader and the
other followers.

I believe having the current JMX is much better as people can handle these
stats in whichever way they want.
They could plot any stat vs any stat, but if we limit that to a kafka
topic, it will become more complex

On the other hand, like Cassandra exposes its stats in Cassandra as well,
if one Cassandra node is down, the other will still be able to report. But,
the reports will be limited to what Cassandra provides.
Having a generic way like JMX reporting allows greater flexibility

I would suggest to encourage people to contribute to easily setting up the
jmx monitoring pipelines, either using tools like Ganglia or Graphite or
Cacti. Something like Kafka clients, which are not maintained by kafka
committers.

I would love to hear from others on this

Regards,
Prabhjot

On Wed, Sep 23, 2015 at 9:22 AM, James Cheng (JIRA)  wrote:

>
> [
> https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903904#comment-14903904
> ]
>
> James Cheng commented on KAFKA-2569:
> 
>
> Yes, although I didn't realize that was a rare thing. I'm pretty new to
> the operations-side of Kafka.
>
> How do people typically monitor things with JMX metrics (like Kafka and
> Zookeeper)?
>
> > Kafka should write its metrics to a Kafka topic
> > ---
> >
> > Key: KAFKA-2569
> > URL: https://issues.apache.org/jira/browse/KAFKA-2569
> > Project: Kafka
> >  Issue Type: New Feature
> >Reporter: James Cheng
> >
> > Kafka is often used to hold and transport monitoring data.
> > In order to monitor Kafka itself, Kafka currently exposes many metrics
> via JMX, which require using a tool to pull the JMX metrics, and then write
> them to the monitoring system.
> > It would be convenient if Kafka could simply send its metrics to a Kafka
> topic. This would make most sense if the Kafka topic was in a different
> Kafka cluster, but could still be useful even if it was sent to a topic in
> the same Kafka cluster.
> > Of course, if sent to the same cluster, it would not be accessible if
> the cluster itself was down.
> > This would allow monitoring of Kafka itself without requiring people to
> set up their own JMX-to-monitoring-system pipelines.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>



-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

[jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[Otis Gospodnetic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903876#comment-14903876
 ] 

Otis Gospodnetic commented on KAFKA-2569:
-

bq. This would allow monitoring of Kafka itself without requiring people to set 
up their own JMX-to-monitoring-system pipelines.

But wouldn't that then require people to set up their own "Kafka topic to 
monitoring system" pipelines, which relatively few people and monitoring tools 
have?


> Kafka should write its metrics to a Kafka topic
> ---
>
> Key: KAFKA-2569
> URL: https://issues.apache.org/jira/browse/KAFKA-2569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: James Cheng
>
> Kafka is often used to hold and transport monitoring data.
> In order to monitor Kafka itself, Kafka currently exposes many metrics via 
> JMX, which require using a tool to pull the JMX metrics, and then write them 
> to the monitoring system.
> It would be convenient if Kafka could simply send its metrics to a Kafka 
> topic. This would make most sense if the Kafka topic was in a different Kafka 
> cluster, but could still be useful even if it was sent to a topic in the same 
> Kafka cluster.
> Of course, if sent to the same cluster, it would not be accessible if the 
> cluster itself was down.
> This would allow monitoring of Kafka itself without requiring people to set 
> up their own JMX-to-monitoring-system pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)