Hi Pushkar,
There are broker side metrics for messages in / bytes in / bytes out per
topic per second. I use this jmx_exporter rule to export them:
- pattern: 'kafka.server<type=BrokerTopicMetrics, name=(.+),
topic=(.+)><>OneMinuteRate'
name: kafka_broker_per_topic_$1_one_minute_rate
labels:
topic: $1
type: GAUGE
You can't get the number of messages out per topic from the broker
because... I think it's somehow related to batching, or similar, it doesn't
count messages out, only bytes out. You can, however, get that metric from
the consumer if you're using the Java Kafka client, there's a per-topic
messages consumed per second metric exposed as an MBean.
You could either use jmx_exporter to also export that from the client app,
or if possible, add some code that connects to the mbean inside the JVM and
then exports it via any pre-existing Prometheus registry. You might want to
then use a Prometheus aggregating rule to collate all the per-consumer apps
into a per-consumer-group metric, unless the per consumer granularity is of
interest to you: https://prometheus.io/docs/practices/rules/
Hope that helps,
Kind regards,
Liam Clarke-Hutchinsons
On Thu, 16 Jul. 2020, 3:46 pm Pushkar Deole, <pdeole2...@gmail.com> wrote:
> Thanks Claudia! For broker level metrics, we are also using same jmx
> exporter to export those metrics to prometheus.
> Are you fetching any per topic metrics from broker? e.g. messages produced
> on a certain topic or messages consumed from a certain topic. I am mainly
> interested in these metrics.
>
> I read in kafka docs that they are present at producer/consumer, however I
> am not sure how to fetch them from consumer/producer.
>
> On Wed, Jul 15, 2020 at 8:32 PM Claudia Kesslau <c.kess...@kasasi.de>
> wrote:
>
> > Hi,
> >
> > I use https://github.com/prometheus/jmx_exporter for collecting broker
> > metrics and integrating them into prometheus.
> >
> > Hope this helps.
> > Greetings,
> > Claudia
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Pushkar Deole <pdeole2...@gmail.com>
> > Gesendet: Mittwoch, 15. Juli 2020 09:07
> > An: users@kafka.apache.org
> > Betreff: Re: kafka per topic metrics
> >
> > We are using prometheus as metrics collection and storage system and
> > Grafana for displaying those metrics, so integration with them is
> required
> >
> > On Wed, Jul 15, 2020 at 11:11 AM rohit garg <rohit.gar...@gmail.com>
> > wrote:
> >
> > > You can try using kafka manager and check it will fullfill most of
> > > requirement of yours.
> > >
> > > Thanks and Regards,
> > > Rohit
> > >
> > > On Wed, Jul 15, 2020, 10:33 Pushkar Deole <pdeole2...@gmail.com>
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Any inputs as to how the kafka consumer and producer metrics can be
> > > hooked
> > > > up to a monitoring system such as prometheus ?
> > > >
> > > > On Tue, Jul 14, 2020 at 4:22 PM Pushkar Deole <pdeole2...@gmail.com>
> > > > wrote:
> > > >
> > > > > i did find these metrics from confluent docs: however how can i
> > > > > get
> > > this
> > > > > metric ? is it available at kafka broker?
> > > > >
> > > > > Per-Topic Metrics
> > > > >
> > > > > MBean:
> > > > >
> > > >
> > > kafka.producer:type=producer-topic-metrics,client-id=([-.w]+),topic=([
> > > -.w]+)
> > > > > record-send-rateThe average number of records sent per second for
> > > > > a
> > > > topic.
> > > > >
> > > > > On Tue, Jul 14, 2020 at 3:27 PM Pushkar Deole
> > > > > <pdeole2...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi All,
> > > > >>
> > > > >> Need some help on kafka metrics, i am interested in certain
> > > > >> metrics
> > > e.g.
> > > > >> i need to know the number of records published on a particular
> > > > >> topic
> > > and
> > > > >> number of records consumed from that topic by a specific consumer
> > > > group, i
> > > > >> would need a total of these 2 and also average per second for
> them.
> > > > >>
> > > > >> Are those metrics available on kafka brokers and if yes then
> > > > >> which are those metrics that would give me above counts?
> > > > >>
> > > > >
> > > >
> > >
> >
>
