Thanks Liam... went through the book and got many of doubts cleared.. I think apart from BrokerTopicsMetrics available at broker, there are many other per topic metrics available at consumer and producers. Are you also fetching those client app metrics through jmx exporter?
On Thu, Jul 16, 2020 at 11:16 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hi Pushkar, > > Best way to see what metrics are available is to connect to a broker via > JConsole to see the exposed mbeans. > > You can iterate over them programmatically by using the MBean API. > > Also recommend chapter 10 of Kafka: The Definitive Guide, it covers the > metrics really well. > > Cheers, > > Liam Clarke-Hutchinson > > On Thu, 16 Jul. 2020, 5:05 pm Pushkar Deole, <pdeole2...@gmail.com> wrote: > > > Thanks Liam... > > Few questions: in your pattern the topic parameter is appended pattern: > > 'kafka.server<type=BrokerTopicMetrics, name=(.+), > > topic=(.+)><>OneMinuteRate' > > however the kafka docs doesn't mention that > > kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec > > > > does the topic parameter available in all BrokerTopicMetrics and can > the > > broker provide that parameter dynamically by iterating through all topics > > or selective topics? Where will the logic reside to iterate through > > available topics and export metrics for all of them? > > > > On Thu, Jul 16, 2020 at 10:04 AM Liam Clarke-Hutchinson < > > liam.cla...@adscale.co.nz> wrote: > > > > > Whoops, just spotted a typo - the second $1 in the above snippet should > > of > > > course be $2. > > > > > > On Thu, Jul 16, 2020 at 4:33 PM Liam Clarke-Hutchinson < > > > liam.cla...@adscale.co.nz> wrote: > > > > > > > Hi Pushkar, > > > > > > > > There are broker side metrics for messages in / bytes in / bytes out > > per > > > > topic per second. I use this jmx_exporter rule to export them: > > > > - pattern: 'kafka.server<type=BrokerTopicMetrics, name=(.+), > > > > topic=(.+)><>OneMinuteRate' > > > > name: kafka_broker_per_topic_$1_one_minute_rate > > > > labels: > > > > topic: $1 > > > > type: GAUGE > > > > > > > > You can't get the number of messages out per topic from the broker > > > > because... I think it's somehow related to batching, or similar, it > > > doesn't > > > > count messages out, only bytes out. You can, however, get that metric > > > from > > > > the consumer if you're using the Java Kafka client, there's a > per-topic > > > > messages consumed per second metric exposed as an MBean. > > > > > > > > You could either use jmx_exporter to also export that from the client > > > app, > > > > or if possible, add some code that connects to the mbean inside the > JVM > > > and > > > > then exports it via any pre-existing Prometheus registry. You might > > want > > > to > > > > then use a Prometheus aggregating rule to collate all the > per-consumer > > > apps > > > > into a per-consumer-group metric, unless the per consumer granularity > > is > > > of > > > > interest to you: https://prometheus.io/docs/practices/rules/ > > > > > > > > Hope that helps, > > > > > > > > Kind regards, > > > > > > > > Liam Clarke-Hutchinsons > > > > > > > > On Thu, 16 Jul. 2020, 3:46 pm Pushkar Deole, <pdeole2...@gmail.com> > > > wrote: > > > > > > > >> Thanks Claudia! For broker level metrics, we are also using same jmx > > > >> exporter to export those metrics to prometheus. > > > >> Are you fetching any per topic metrics from broker? e.g. messages > > > produced > > > >> on a certain topic or messages consumed from a certain topic. I am > > > mainly > > > >> interested in these metrics. > > > >> > > > >> I read in kafka docs that they are present at producer/consumer, > > > however I > > > >> am not sure how to fetch them from consumer/producer. > > > >> > > > >> On Wed, Jul 15, 2020 at 8:32 PM Claudia Kesslau < > c.kess...@kasasi.de> > > > >> wrote: > > > >> > > > >> > Hi, > > > >> > > > > >> > I use https://github.com/prometheus/jmx_exporter for collecting > > > broker > > > >> > metrics and integrating them into prometheus. > > > >> > > > > >> > Hope this helps. > > > >> > Greetings, > > > >> > Claudia > > > >> > > > > >> > -----Ursprüngliche Nachricht----- > > > >> > Von: Pushkar Deole <pdeole2...@gmail.com> > > > >> > Gesendet: Mittwoch, 15. Juli 2020 09:07 > > > >> > An: users@kafka.apache.org > > > >> > Betreff: Re: kafka per topic metrics > > > >> > > > > >> > We are using prometheus as metrics collection and storage system > and > > > >> > Grafana for displaying those metrics, so integration with them is > > > >> required > > > >> > > > > >> > On Wed, Jul 15, 2020 at 11:11 AM rohit garg < > rohit.gar...@gmail.com > > > > > > >> > wrote: > > > >> > > > > >> > > You can try using kafka manager and check it will fullfill most > of > > > >> > > requirement of yours. > > > >> > > > > > >> > > Thanks and Regards, > > > >> > > Rohit > > > >> > > > > > >> > > On Wed, Jul 15, 2020, 10:33 Pushkar Deole <pdeole2...@gmail.com > > > > > >> wrote: > > > >> > > > > > >> > > > Hi All, > > > >> > > > > > > >> > > > Any inputs as to how the kafka consumer and producer metrics > can > > > be > > > >> > > hooked > > > >> > > > up to a monitoring system such as prometheus ? > > > >> > > > > > > >> > > > On Tue, Jul 14, 2020 at 4:22 PM Pushkar Deole < > > > pdeole2...@gmail.com > > > >> > > > > >> > > > wrote: > > > >> > > > > > > >> > > > > i did find these metrics from confluent docs: however how > can > > i > > > >> > > > > get > > > >> > > this > > > >> > > > > metric ? is it available at kafka broker? > > > >> > > > > > > > >> > > > > Per-Topic Metrics > > > >> > > > > > > > >> > > > > MBean: > > > >> > > > > > > > >> > > > > > > >> > > > > > kafka.producer:type=producer-topic-metrics,client-id=([-.w]+),topic=([ > > > >> > > -.w]+) > > > >> > > > > record-send-rateThe average number of records sent per > second > > > for > > > >> > > > > a > > > >> > > > topic. > > > >> > > > > > > > >> > > > > On Tue, Jul 14, 2020 at 3:27 PM Pushkar Deole > > > >> > > > > <pdeole2...@gmail.com> > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > >> Hi All, > > > >> > > > >> > > > >> > > > >> Need some help on kafka metrics, i am interested in certain > > > >> > > > >> metrics > > > >> > > e.g. > > > >> > > > >> i need to know the number of records published on a > > particular > > > >> > > > >> topic > > > >> > > and > > > >> > > > >> number of records consumed from that topic by a specific > > > consumer > > > >> > > > group, i > > > >> > > > >> would need a total of these 2 and also average per second > for > > > >> them. > > > >> > > > >> > > > >> > > > >> Are those metrics available on kafka brokers and if yes > then > > > >> > > > >> which are those metrics that would give me above counts? > > > >> > > > >> > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
