Hi Jens, We did something similar to this at Zalando.
https://github.com/zalando-incubator/remora It effectively supplies the kafka consumer group supply command as a http endpoint. On 12 August 2017 at 16:42, Subhash Sriram <subhash.sri...@gmail.com> wrote: > Hi Jens, > > Have you looked at Burrow? > > https://github.com/linkedin/Burrow/blob/master/README.md > > Thanks, > Subhash > > Sent from my iPhone > > > On Aug 12, 2017, at 8:55 AM, Jens Rantil <jens.ran...@gmail.com> wrote: > > > > Hi, > > > > I am one of the maintainers of prometheus-kafka-consumer- > group-exporter[1], > > which exports consumer group offsets and lag to Prometheus. The way we > > currently scrape this information is by periodically executing > > `kafka-consumer-groups.sh --describe` for each group and parse the > output. > > > > Recently the output from `kafka-consumer-groups.sh --describe` was > > changed[2]. While I am working on a patch[3] to accomodate for the new > > output format I was wondering if there is an easier, possibly more > > stable[4] and more future proof, way for our project to extract the > > information we are interested in. Does anyone know of a Go library that > > could extract the metrics we need? Or would it make sense to refactor > > `kafka-consumer-groups.sh` to support a more structured output? I'd love > to > > hear your input. > > > > Also, if Kafka exported the same metrics through JMX our project would > not > > exist, but maybe that's another story... > > > > Cheers, > > Jens > > > > [1] https://github.com/kawamuray/prometheus-kafka-consumer- > group-exporter > > [2] > > https://github.com/kawamuray/prometheus-kafka-consumer- > group-exporter/issues/24 > > [3] > > https://github.com/kawamuray/prometheus-kafka-consumer- > group-exporter/pull/29 > > [4] We've also encountered `kafka-consumer-groups.sh` hanging a few times > > in production. There's a race condition somewhere in the script, most > > likely when a topic is rebalancing. Currently we kill the process if it > > doesn't finish within a timeout. See > > https://github.com/kawamuray/prometheus-kafka-consumer- > group-exporter/blob/e4cdc3b1245f636d89d7e227066f02 > 578d732165/kafka/collector.go#L44 > > . > > > > -- > > Want to communicate with me securely? You can find my PGP key here > > <https://pgp.mit.edu/pks/lookup?op=get&search=0xBA632DB691DA03B8>. >
