Re: Re: How to push metrics to graphite - jmxtrans does not work
Hi,
Thanks for the help. I found the issue.I was appending to the bottom
when I should have placed the below line at the top of the file.
echo 'KAFKA_JMX_OPTS="-Dcom.sun.
management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false"' | tee -a
/var/kafka/bin/kafka-run-class.sh
On Wed, Dec 3, 2014 at 1:27 PM, Jason Rosenberg wrote:
> fwiw, we wrap the kafka server in our java service container framework.
> This allows us to use the default GraphiteReporter class that is part of
> the yammer metrics library (which is used by kafka directly). So it works
> seemlessly. (We've since changed our use of GraphiteReporter to instead
> send all our metrics via kafka :))
>
> Jason
>
> On Tue, Dec 2, 2014 at 11:00 PM, Otis Gospodnetic <
> otis.gospodne...@gmail.com> wrote:
>
> > Hi,
> >
> > You can make use of this documentation aimed at JMX and monitoring:
> >
> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
> >
> > There is a section about Kafka and the information is not SPM-specific.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Tue, Dec 2, 2014 at 9:34 PM, YuanJia Li wrote:
> >
> > > Hi David，
> > > Just edit "kafka-server-start.sh", and add "export
> JMX_PORT=",it
> > > will work.
> > >
> > >
> > >
> > >
> > > Yuanjia
> > >
> > > From: David Montgomery
> > > Date: 2014-12-03 04:47
> > > To:users
> > > Subject: Re: How to push metrics to graphite - jmxtrans does not work
> > > Hi,
> > >
> > > I am seeing this in the logs and wondering what "jmx_port":-1 means?
> > >
> > > INFO conflict in /brokers/ids/29136 data: { "host":"104.111.111.111.",
> > > "jmx_port":-1, "port":9092, "timestamp":"1417552817875", "version":1 }
> > > stored data: { "host":"104.111.111", "jmx_port":-1, "port":9092,
> > > "timestamp":"1417552738253", "version":1
> > >
> > > despite having these added
> > >
> > > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=
> > > -Dcom.sun.management.jmxremote=true
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> > > /var/kafka/bin/kafka-run-class.sh
> > > echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> > > /var/kafka/bin/kafka-server-start.sh
> > >
> > > Thanks
> > >
> > > On Tue, Dec 2, 2014 at 9:58 PM, Andrew Otto
> wrote:
> > >
> > > > Maybe also set:
> > > >
> > > > -Dcom.sun.management.jmxremote.port=
> > > >
> > > > ?
> > > >
> > > >
> > > > > On Dec 2, 2014, at 02:59, David Montgomery <
> > davidmontgom...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I am having a very difficult time trying to report kafka 8 metrics
> to
> > > > > Graphite. Nothing is listening on and and no data in
> graphite.
> > > If
> > > > > this method of graphite reporting is know to not work is there an
> > > > > alternative to jmxtrans to get data to graphite?
> > > > >
> > > > > I am using the deb file to install jmxtrans on ubuntu 12.04
> > > > >
> > > > > And I use the below to modify kafka scripts
> > > > >
> > > > > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> > > > > /var/kafka/bin/kafka-run-class.sh
> > > > > echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> > > > > /var/kafka/bin/kafka-server-start.sh
> > > > >
> > > > > {
> > > > > "servers" : [ {
> > > > >"host" : "127.0.0.1",
> > > > >"port" : "",
> > > > >"alias" : "<%=node.name%>",
> > > > >"queries" : [
> > > > > {
> > > > > "obj" : "kafka:type=kafka.SocketServerStats",
> > > > > "resultAlias": "kafka.socketServerStats",
> > > > > "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs",
> > > > > "BytesReadPerSecond", "BytesWrittenPerSecond",
> > > "FetchRequestsPerSecond",
> > > > > "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" ,
> > > > > "NumProduceRequests" , "ProduceRequestsPerSecond",
> "TotalBytesRead",
> > > > > "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs"
> > ],
> > > > > "outputWriters" : [ {
> > > > > "@class" :
> > > > "com.googlecode.jmxtrans.model.output.GraphiteWriter",
> > > > > "settings" : {
> > > > >"host" : "<%=@monitor_host%>",
> > > > >"port" : "2003"
> > > > > }
> > > > >} ]
> > > > > }
> > > > >],
> > > > >"numQueryThreads": "2"
> > > > > } ]
> > > > > }
> > > >
> > > >
> > >
> >
>
Re: Re: How to push metrics to graphite - jmxtrans does not work
fwiw, we wrap the kafka server in our java service container framework.
This allows us to use the default GraphiteReporter class that is part of
the yammer metrics library (which is used by kafka directly). So it works
seemlessly. (We've since changed our use of GraphiteReporter to instead
send all our metrics via kafka :))
Jason
On Tue, Dec 2, 2014 at 11:00 PM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:
> Hi,
>
> You can make use of this documentation aimed at JMX and monitoring:
> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
>
> There is a section about Kafka and the information is not SPM-specific.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Dec 2, 2014 at 9:34 PM, YuanJia Li wrote:
>
> > Hi David，
> > Just edit "kafka-server-start.sh", and add "export JMX_PORT=",it
> > will work.
> >
> >
> >
> >
> > Yuanjia
> >
> > From: David Montgomery
> > Date: 2014-12-03 04:47
> > To:users
> > Subject: Re: How to push metrics to graphite - jmxtrans does not work
> > Hi,
> >
> > I am seeing this in the logs and wondering what "jmx_port":-1 means?
> >
> > INFO conflict in /brokers/ids/29136 data: { "host":"104.111.111.111.",
> > "jmx_port":-1, "port":9092, "timestamp":"1417552817875", "version":1 }
> > stored data: { "host":"104.111.111", "jmx_port":-1, "port":9092,
> > "timestamp":"1417552738253", "version":1
> >
> > despite having these added
> >
> > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=
> > -Dcom.sun.management.jmxremote=true
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> > /var/kafka/bin/kafka-run-class.sh
> > echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> > /var/kafka/bin/kafka-server-start.sh
> >
> > Thanks
> >
> > On Tue, Dec 2, 2014 at 9:58 PM, Andrew Otto wrote:
> >
> > > Maybe also set:
> > >
> > > -Dcom.sun.management.jmxremote.port=
> > >
> > > ?
> > >
> > >
> > > > On Dec 2, 2014, at 02:59, David Montgomery <
> davidmontgom...@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I am having a very difficult time trying to report kafka 8 metrics to
> > > > Graphite. Nothing is listening on and and no data in graphite.
> > If
> > > > this method of graphite reporting is know to not work is there an
> > > > alternative to jmxtrans to get data to graphite?
> > > >
> > > > I am using the deb file to install jmxtrans on ubuntu 12.04
> > > >
> > > > And I use the below to modify kafka scripts
> > > >
> > > > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> > > > /var/kafka/bin/kafka-run-class.sh
> > > > echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> > > > /var/kafka/bin/kafka-server-start.sh
> > > >
> > > > {
> > > > "servers" : [ {
> > > >"host" : "127.0.0.1",
> > > >"port" : "",
> > > >"alias" : "<%=node.name%>",
> > > >"queries" : [
> > > > {
> > > > "obj" : "kafka:type=kafka.SocketServerStats",
> > > > "resultAlias": "kafka.socketServerStats",
> > > > "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs",
> > > > "BytesReadPerSecond", "BytesWrittenPerSecond",
> > "FetchRequestsPerSecond",
> > > > "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" ,
> > > > "NumProduceRequests" , "ProduceRequestsPerSecond", "TotalBytesRead",
> > > > "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs"
> ],
> > > > "outputWriters" : [ {
> > > > "@class" :
> > > "com.googlecode.jmxtrans.model.output.GraphiteWriter",
> > > > "settings" : {
> > > >"host" : "<%=@monitor_host%>",
> > > >"port" : "2003"
> > > > }
> > > >} ]
> > > > }
> > > >],
> > > >"numQueryThreads": "2"
> > > > } ]
> > > > }
> > >
> > >
> >
>
Re: Re: How to push metrics to graphite - jmxtrans does not work
Hi,
You can make use of this documentation aimed at JMX and monitoring:
https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
There is a section about Kafka and the information is not SPM-specific.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Dec 2, 2014 at 9:34 PM, YuanJia Li wrote:
> Hi David，
> Just edit "kafka-server-start.sh", and add "export JMX_PORT=",it
> will work.
>
>
>
>
> Yuanjia
>
> From: David Montgomery
> Date: 2014-12-03 04:47
> To:users
> Subject: Re: How to push metrics to graphite - jmxtrans does not work
> Hi,
>
> I am seeing this in the logs and wondering what "jmx_port":-1 means?
>
> INFO conflict in /brokers/ids/29136 data: { "host":"104.111.111.111.",
> "jmx_port":-1, "port":9092, "timestamp":"1417552817875", "version":1 }
> stored data: { "host":"104.111.111", "jmx_port":-1, "port":9092,
> "timestamp":"1417552738253", "version":1
>
> despite having these added
>
> echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=
> -Dcom.sun.management.jmxremote=true
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> /var/kafka/bin/kafka-run-class.sh
> echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> /var/kafka/bin/kafka-server-start.sh
>
> Thanks
>
> On Tue, Dec 2, 2014 at 9:58 PM, Andrew Otto wrote:
>
> > Maybe also set:
> >
> > -Dcom.sun.management.jmxremote.port=
> >
> > ?
> >
> >
> > > On Dec 2, 2014, at 02:59, David Montgomery
> > wrote:
> > >
> > > Hi,
> > >
> > > I am having a very difficult time trying to report kafka 8 metrics to
> > > Graphite. Nothing is listening on and and no data in graphite.
> If
> > > this method of graphite reporting is know to not work is there an
> > > alternative to jmxtrans to get data to graphite?
> > >
> > > I am using the deb file to install jmxtrans on ubuntu 12.04
> > >
> > > And I use the below to modify kafka scripts
> > >
> > > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> > > /var/kafka/bin/kafka-run-class.sh
> > > echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> > > /var/kafka/bin/kafka-server-start.sh
> > >
> > > {
> > > "servers" : [ {
> > >"host" : "127.0.0.1",
> > >"port" : "",
> > >"alias" : "<%=node.name%>",
> > >"queries" : [
> > > {
> > > "obj" : "kafka:type=kafka.SocketServerStats",
> > > "resultAlias": "kafka.socketServerStats",
> > > "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs",
> > > "BytesReadPerSecond", "BytesWrittenPerSecond",
> "FetchRequestsPerSecond",
> > > "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" ,
> > > "NumProduceRequests" , "ProduceRequestsPerSecond", "TotalBytesRead",
> > > "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs" ],
> > > "outputWriters" : [ {
> > > "@class" :
> > "com.googlecode.jmxtrans.model.output.GraphiteWriter",
> > > "settings" : {
> > >"host" : "<%=@monitor_host%>",
> > >"port" : "2003"
> > > }
> > >} ]
> > > }
> > >],
> > >"numQueryThreads": "2"
> > > } ]
> > > }
> >
> >
>
Re: Re: How to push metrics to graphite - jmxtrans does not work
Hi David，
Just edit "kafka-server-start.sh", and add "export JMX_PORT=",it will
work.
Yuanjia
From: David Montgomery
Date: 2014-12-03 04:47
To:users
Subject: Re: How to push metrics to graphite - jmxtrans does not work
Hi,
I am seeing this in the logs and wondering what "jmx_port":-1 means?
INFO conflict in /brokers/ids/29136 data: { "host":"104.111.111.111.",
"jmx_port":-1, "port":9092, "timestamp":"1417552817875", "version":1 }
stored data: { "host":"104.111.111", "jmx_port":-1, "port":9092,
"timestamp":"1417552738253", "version":1
despite having these added
echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false"' | tee -a
/var/kafka/bin/kafka-run-class.sh
echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
/var/kafka/bin/kafka-server-start.sh
Thanks
On Tue, Dec 2, 2014 at 9:58 PM, Andrew Otto wrote:
> Maybe also set:
>
> -Dcom.sun.management.jmxremote.port=
>
> ?
>
>
> > On Dec 2, 2014, at 02:59, David Montgomery
> wrote:
> >
> > Hi,
> >
> > I am having a very difficult time trying to report kafka 8 metrics to
> > Graphite. Nothing is listening on and and no data in graphite. If
> > this method of graphite reporting is know to not work is there an
> > alternative to jmxtrans to get data to graphite?
> >
> > I am using the deb file to install jmxtrans on ubuntu 12.04
> >
> > And I use the below to modify kafka scripts
> >
> > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> > /var/kafka/bin/kafka-run-class.sh
> > echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> > /var/kafka/bin/kafka-server-start.sh
> >
> > {
> > "servers" : [ {
> >"host" : "127.0.0.1",
> >"port" : "",
> >"alias" : "<%=node.name%>",
> >"queries" : [
> > {
> > "obj" : "kafka:type=kafka.SocketServerStats",
> > "resultAlias": "kafka.socketServerStats",
> > "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs",
> > "BytesReadPerSecond", "BytesWrittenPerSecond", "FetchRequestsPerSecond",
> > "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" ,
> > "NumProduceRequests" , "ProduceRequestsPerSecond", "TotalBytesRead",
> > "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs" ],
> > "outputWriters" : [ {
> > "@class" :
> "com.googlecode.jmxtrans.model.output.GraphiteWriter",
> > "settings" : {
> >"host" : "<%=@monitor_host%>",
> >"port" : "2003"
> > }
> >} ]
> > }
> >],
> >"numQueryThreads": "2"
> > } ]
> > }
>
>
Re: How to push metrics to graphite - jmxtrans does not work
Hi,
I am seeing this in the logs and wondering what "jmx_port":-1 means?
INFO conflict in /brokers/ids/29136 data: { "host":"104.111.111.111.",
"jmx_port":-1, "port":9092, "timestamp":"1417552817875", "version":1 }
stored data: { "host":"104.111.111", "jmx_port":-1, "port":9092,
"timestamp":"1417552738253", "version":1
despite having these added
echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false"' | tee -a
/var/kafka/bin/kafka-run-class.sh
echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
/var/kafka/bin/kafka-server-start.sh
Thanks
On Tue, Dec 2, 2014 at 9:58 PM, Andrew Otto wrote:
> Maybe also set:
>
> -Dcom.sun.management.jmxremote.port=
>
> ?
>
>
> > On Dec 2, 2014, at 02:59, David Montgomery
> wrote:
> >
> > Hi,
> >
> > I am having a very difficult time trying to report kafka 8 metrics to
> > Graphite. Nothing is listening on and and no data in graphite. If
> > this method of graphite reporting is know to not work is there an
> > alternative to jmxtrans to get data to graphite?
> >
> > I am using the deb file to install jmxtrans on ubuntu 12.04
> >
> > And I use the below to modify kafka scripts
> >
> > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> > /var/kafka/bin/kafka-run-class.sh
> > echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> > /var/kafka/bin/kafka-server-start.sh
> >
> > {
> > "servers" : [ {
> >"host" : "127.0.0.1",
> >"port" : "",
> >"alias" : "<%=node.name%>",
> >"queries" : [
> > {
> > "obj" : "kafka:type=kafka.SocketServerStats",
> > "resultAlias": "kafka.socketServerStats",
> > "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs",
> > "BytesReadPerSecond", "BytesWrittenPerSecond", "FetchRequestsPerSecond",
> > "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" ,
> > "NumProduceRequests" , "ProduceRequestsPerSecond", "TotalBytesRead",
> > "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs" ],
> > "outputWriters" : [ {
> > "@class" :
> "com.googlecode.jmxtrans.model.output.GraphiteWriter",
> > "settings" : {
> >"host" : "<%=@monitor_host%>",
> >"port" : "2003"
> > }
> >} ]
> > }
> >],
> >"numQueryThreads": "2"
> > } ]
> > }
>
>
Re: How to push metrics to graphite - jmxtrans does not work
Maybe also set:
-Dcom.sun.management.jmxremote.port=
?
> On Dec 2, 2014, at 02:59, David Montgomery wrote:
>
> Hi,
>
> I am having a very difficult time trying to report kafka 8 metrics to
> Graphite. Nothing is listening on and and no data in graphite. If
> this method of graphite reporting is know to not work is there an
> alternative to jmxtrans to get data to graphite?
>
> I am using the deb file to install jmxtrans on ubuntu 12.04
>
> And I use the below to modify kafka scripts
>
> echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false"' | tee -a
> /var/kafka/bin/kafka-run-class.sh
> echo 'export JMX_PORT=${JMX_PORT:-}' | tee -a
> /var/kafka/bin/kafka-server-start.sh
>
> {
> "servers" : [ {
>"host" : "127.0.0.1",
>"port" : "",
>"alias" : "<%=node.name%>",
>"queries" : [
> {
> "obj" : "kafka:type=kafka.SocketServerStats",
> "resultAlias": "kafka.socketServerStats",
> "attr" : [ "AvgFetchRequestMs", "AvgProduceRequestMs",
> "BytesReadPerSecond", "BytesWrittenPerSecond", "FetchRequestsPerSecond",
> "MaxFetchRequestMs", "MaxProduceRequestMs" , "NumFetchRequests" ,
> "NumProduceRequests" , "ProduceRequestsPerSecond", "TotalBytesRead",
> "TotalBytesWritten", "TotalFetchRequestMs", "TotalProduceRequestMs" ],
> "outputWriters" : [ {
> "@class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
> "settings" : {
>"host" : "<%=@monitor_host%>",
>"port" : "2003"
> }
>} ]
> }
>],
>"numQueryThreads": "2"
> } ]
> }
Re: Re: How to push metrics to graphite - jmxtrans does not work
Jmxtrans should connect to the jmxremote port. Try to run "ps -aux |grep kafka", and find the process contain -Dcom.sun.management.jmxremote.port or not. If not, try to edit "kafka-server-start.sh", add "export JMX_PORT=".
Re: How to push metrics to graphite - jmxtrans does not work
I am using kafka 8. I will try your suggestion. But if I run lsof -i : should I not see a proccess running on that port? I am not seeing anything. On Tue, Dec 2, 2014 at 5:37 PM, yuanjia8947 wrote: > hi David, > which version do you use kafka? > when I use kafka 0.8.0, I write jmxtrans "obj" like this "obj": > "\"kafka.server\":type=\"BrokerTopicMetrics\",name=\"AllTopicsBytesOutPerSec\"" > . > Hope it useful for you. > > liyuanjia > > > > liyuanjia
