We are also collecting consumer group metrics from Kafka - we didn't want to add extra unnecessary dependencies (such as burrow, which is also overkill for what we need), so we just run a script every minute on the brokers that parses the output of kafka-consumer-groups.sh and uploads it to an http-listening logstash, which then indexes it in our central ElasticSearch cluster where we keep all of our application metrics. Then, it's easy to visualize the data with Kibana.
Unfortunately, as Jens pointed out, the output format seems to change a lot (it changed between 0.10.0.1 to 0.10.1.0 if I'm not mistaken, and has changed again in 0.11.0.0) and each time we upgrade we have to adjust our script to account for the format difference. I agree it would be great if Kafka either didn't change the format so much or better yet exposed it via JMX. Dan On 12 August 2017 at 15:55, Jens Rantil <jens.ran...@gmail.com> wrote: > Hi, > > I am one of the maintainers of prometheus-kafka-consumer- > group-exporter[1], > which exports consumer group offsets and lag to Prometheus. The way we > currently scrape this information is by periodically executing > `kafka-consumer-groups.sh --describe` for each group and parse the output. > > Recently the output from `kafka-consumer-groups.sh --describe` was > changed[2]. While I am working on a patch[3] to accomodate for the new > output format I was wondering if there is an easier, possibly more > stable[4] and more future proof, way for our project to extract the > information we are interested in. Does anyone know of a Go library that > could extract the metrics we need? Or would it make sense to refactor > `kafka-consumer-groups.sh` to support a more structured output? I'd love to > hear your input. > > Also, if Kafka exported the same metrics through JMX our project would not > exist, but maybe that's another story... > > Cheers, > Jens > > [1] https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter > [2] > https://github.com/kawamuray/prometheus-kafka-consumer- > group-exporter/issues/24 > [3] > https://github.com/kawamuray/prometheus-kafka-consumer- > group-exporter/pull/29 > [4] We've also encountered `kafka-consumer-groups.sh` hanging a few times > in production. There's a race condition somewhere in the script, most > likely when a topic is rebalancing. Currently we kill the process if it > doesn't finish within a timeout. See > https://github.com/kawamuray/prometheus-kafka-consumer- > group-exporter/blob/e4cdc3b1245f636d89d7e227066f02 > 578d732165/kafka/collector.go#L44 > . > > -- > Want to communicate with me securely? You can find my PGP key here > <https://pgp.mit.edu/pks/lookup?op=get&search=0xBA632DB691DA03B8>. >
