Thanks Gabriele, it turned out I didn't have that pattern deployed (facepalm). After deploying it , worked right away. Now I'm struggling with understanding the size metric. Do you know if it's reporting the size (in bytes) of all segments for that broker/topic/partition?
I'm trying to compare those values with the actual Log file, but doesn't seems to match exactly. Sorry for those questions, but can't find a good documentation and I'm lost reading core/src/main/scala/kafka/log/Log.scala Thanks for your help.- On Wed, Feb 26, 2020 at 6:02 AM Gabriele Paggi <gabriele.pa...@gmail.com> wrote: > > Hi Richard, > > The beans path is: > kafka.log:name=Size,partition=<partition>,topic=<topic>,type=Log > I don't have a jmx_exporter at hand to test it at the moment but I > don't see anything obviously wrong in your config, other than type: > GAUGE missing. > Did you try browsing the beans with jmxterm before configuring the exporter? > > Gabriele > > On Mon, 24 Feb 2020 at 23:01, Richard Rossel <henhis...@gmail.com> wrote: > > > > Hi Gabriele, > > I'm using Kafka 5.3.1, which is apache kafka 2.3 and I'm using JMX to > > retrieve metrics from brokers. > > The only metric I saw from "kafka.log" is LogFlushStats, but nothing > > about Log.partition > > This is the pattern I have deployed for kafka.log metrics, maybe I > > need a different for partitions sizes? > > > > - pattern : kafka.log<type=Log, name=(.+), topic=(.+), > > partition=(.+)><>Value > > name: kafka_log_$1 > > labels: > > topic: "$2" > > partition: "$3" > > > > what do you think? > > > > On Mon, Feb 24, 2020 at 4:05 PM Gabriele Paggi <gabriele.pa...@gmail.com> > > wrote: > > > > > > Hi Richard, > > > > > > If you are running Kafka > 1.0.0 the information you are looking for > > > is exposed by the Admin API, describeLogDirs method, or via CLI with > > > kafka-log-dirs.sh. > > > The metrics are also exposed by the dropwizard metrics reporter, e.g. > > > for Graphite: > > > kafkalog001.kafka.log.Log.partition.49.topic.__consumer_offsets.Size.value > > > 1315862 1582578155 > > > > > > Gabriele > > > > > > On Mon, 24 Feb 2020 at 19:54, Richard Rossel <henhis...@gmail.com> wrote: > > > > > > > > Hi List, > > > > I'm trying to find the way to keep track of topics' size (or > > > > partitions) across brokers, but > > > > I didn't found any Kafka metrics for that. I start wonder if that is > > > > possible. > > > > > > > > I have ideas for a work around (querying log files) but I wonder what > > > > will be the right way or how do you keep monitoring this type of > > > > metric? > > > > > > > > Thanks.- > > > > > > > > -- > > > > Richard Rossel > > > > Atlanta - GA > > > > > > > > > > > > -- > > > Gabriele > > > > > > > > -- > > Richard Rossel > > Atlanta - GA > > > > -- > Gabriele -- Richard Rossel Atlanta - GA
