Querying consumer groups programmatically (from Golang)

[Jens Rantil]

Hi,

I am one of the maintainers of prometheus-kafka-consumer-group-exporter[1],
which exports consumer group offsets and lag to Prometheus. The way we
currently scrape this information is by periodically executing
`kafka-consumer-groups.sh --describe` for each group and parse the output.

Recently the output from `kafka-consumer-groups.sh --describe` was
changed[2]. While I am working on a patch[3] to accomodate for the new
output format I was wondering if there is an easier, possibly more
stable[4] and more future proof, way for our project to extract the
information we are interested in. Does anyone know of a Go library that
could extract the metrics we need? Or would it make sense to refactor
`kafka-consumer-groups.sh` to support a more structured output? I'd love to
hear your input.

Also, if Kafka exported the same metrics through JMX our project would not
exist, but maybe that's another story...

Cheers,
Jens

[1] https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter
[2]
https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/issues/24
[3]
https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/pull/29
[4] We've also encountered `kafka-consumer-groups.sh` hanging a few times
in production. There's a race condition somewhere in the script, most
likely when a topic is rebalancing. Currently we kill the process if it
doesn't finish within a timeout. See
https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/blob/e4cdc3b1245f636d89d7e227066f02578d732165/kafka/collector.go#L44
.

-- 
Want to communicate with me securely? You can find my PGP key here
.

Re: Querying consumer groups programmatically (from Golang)

[Subhash Sriram]

Hi Jens,

Have you looked at Burrow?

https://github.com/linkedin/Burrow/blob/master/README.md

Thanks,
Subhash

Sent from my iPhone

> On Aug 12, 2017, at 8:55 AM, Jens Rantil  wrote:
> 
> Hi,
> 
> I am one of the maintainers of prometheus-kafka-consumer-group-exporter[1],
> which exports consumer group offsets and lag to Prometheus. The way we
> currently scrape this information is by periodically executing
> `kafka-consumer-groups.sh --describe` for each group and parse the output.
> 
> Recently the output from `kafka-consumer-groups.sh --describe` was
> changed[2]. While I am working on a patch[3] to accomodate for the new
> output format I was wondering if there is an easier, possibly more
> stable[4] and more future proof, way for our project to extract the
> information we are interested in. Does anyone know of a Go library that
> could extract the metrics we need? Or would it make sense to refactor
> `kafka-consumer-groups.sh` to support a more structured output? I'd love to
> hear your input.
> 
> Also, if Kafka exported the same metrics through JMX our project would not
> exist, but maybe that's another story...
> 
> Cheers,
> Jens
> 
> [1] https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter
> [2]
> https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/issues/24
> [3]
> https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/pull/29
> [4] We've also encountered `kafka-consumer-groups.sh` hanging a few times
> in production. There's a race condition somewhere in the script, most
> likely when a topic is rebalancing. Currently we kill the process if it
> doesn't finish within a timeout. See
> https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/blob/e4cdc3b1245f636d89d7e227066f02578d732165/kafka/collector.go#L44
> .
> 
> -- 
> Want to communicate with me securely? You can find my PGP key here
> .

Re: Querying consumer groups programmatically (from Golang)

[Ian Duffy]

Hi Jens,

We did something similar to this at Zalando.

https://github.com/zalando-incubator/remora

It effectively supplies the kafka consumer group supply command as a http
endpoint.

On 12 August 2017 at 16:42, Subhash Sriram  wrote:

> Hi Jens,
>
> Have you looked at Burrow?
>
> https://github.com/linkedin/Burrow/blob/master/README.md
>
> Thanks,
> Subhash
>
> Sent from my iPhone
>
> > On Aug 12, 2017, at 8:55 AM, Jens Rantil  wrote:
> >
> > Hi,
> >
> > I am one of the maintainers of prometheus-kafka-consumer-
> group-exporter[1],
> > which exports consumer group offsets and lag to Prometheus. The way we
> > currently scrape this information is by periodically executing
> > `kafka-consumer-groups.sh --describe` for each group and parse the
> output.
> >
> > Recently the output from `kafka-consumer-groups.sh --describe` was
> > changed[2]. While I am working on a patch[3] to accomodate for the new
> > output format I was wondering if there is an easier, possibly more
> > stable[4] and more future proof, way for our project to extract the
> > information we are interested in. Does anyone know of a Go library that
> > could extract the metrics we need? Or would it make sense to refactor
> > `kafka-consumer-groups.sh` to support a more structured output? I'd love
> to
> > hear your input.
> >
> > Also, if Kafka exported the same metrics through JMX our project would
> not
> > exist, but maybe that's another story...
> >
> > Cheers,
> > Jens
> >
> > [1] https://github.com/kawamuray/prometheus-kafka-consumer-
> group-exporter
> > [2]
> > https://github.com/kawamuray/prometheus-kafka-consumer-
> group-exporter/issues/24
> > [3]
> > https://github.com/kawamuray/prometheus-kafka-consumer-
> group-exporter/pull/29
> > [4] We've also encountered `kafka-consumer-groups.sh` hanging a few times
> > in production. There's a race condition somewhere in the script, most
> > likely when a topic is rebalancing. Currently we kill the process if it
> > doesn't finish within a timeout. See
> > https://github.com/kawamuray/prometheus-kafka-consumer-
> group-exporter/blob/e4cdc3b1245f636d89d7e227066f02
> 578d732165/kafka/collector.go#L44
> > .
> >
> > --
> > Want to communicate with me securely? You can find my PGP key here
> > .
>

Re: Querying consumer groups programmatically (from Golang)

[Gabriel Machado]

Hi Jens and Ian,

Very usefuls projects :).
What's the difference between the 2 softwares ?
Do they support kafka ssl clusters ?

Thanks,
Gabriel.

2017-08-13 3:29 GMT+02:00 Ian Duffy :

> Hi Jens,
>
> We did something similar to this at Zalando.
>
> https://github.com/zalando-incubator/remora
>
> It effectively supplies the kafka consumer group supply command as a http
> endpoint.
>
> On 12 August 2017 at 16:42, Subhash Sriram 
> wrote:
>
> > Hi Jens,
> >
> > Have you looked at Burrow?
> >
> > https://github.com/linkedin/Burrow/blob/master/README.md
> >
> > Thanks,
> > Subhash
> >
> > Sent from my iPhone
> >
> > > On Aug 12, 2017, at 8:55 AM, Jens Rantil 
> wrote:
> > >
> > > Hi,
> > >
> > > I am one of the maintainers of prometheus-kafka-consumer-
> > group-exporter[1],
> > > which exports consumer group offsets and lag to Prometheus. The way we
> > > currently scrape this information is by periodically executing
> > > `kafka-consumer-groups.sh --describe` for each group and parse the
> > output.
> > >
> > > Recently the output from `kafka-consumer-groups.sh --describe` was
> > > changed[2]. While I am working on a patch[3] to accomodate for the new
> > > output format I was wondering if there is an easier, possibly more
> > > stable[4] and more future proof, way for our project to extract the
> > > information we are interested in. Does anyone know of a Go library that
> > > could extract the metrics we need? Or would it make sense to refactor
> > > `kafka-consumer-groups.sh` to support a more structured output? I'd
> love
> > to
> > > hear your input.
> > >
> > > Also, if Kafka exported the same metrics through JMX our project would
> > not
> > > exist, but maybe that's another story...
> > >
> > > Cheers,
> > > Jens
> > >
> > > [1] https://github.com/kawamuray/prometheus-kafka-consumer-
> > group-exporter
> > > [2]
> > > https://github.com/kawamuray/prometheus-kafka-consumer-
> > group-exporter/issues/24
> > > [3]
> > > https://github.com/kawamuray/prometheus-kafka-consumer-
> > group-exporter/pull/29
> > > [4] We've also encountered `kafka-consumer-groups.sh` hanging a few
> times
> > > in production. There's a race condition somewhere in the script, most
> > > likely when a topic is rebalancing. Currently we kill the process if it
> > > doesn't finish within a timeout. See
> > > https://github.com/kawamuray/prometheus-kafka-consumer-
> > group-exporter/blob/e4cdc3b1245f636d89d7e227066f02
> > 578d732165/kafka/collector.go#L44
> > > .
> > >
> > > --
> > > Want to communicate with me securely? You can find my PGP key here
> > > .
> >
>

Re: Querying consumer groups programmatically (from Golang)

[Gabriel Machado]

Hello,

Could you tell me if burrow or remora is compatible with ssl kafka clusters
?


Gabriel.

2017-08-16 15:39 GMT+02:00 Gabriel Machado :

> Hi Jens and Ian,
>
> Very usefuls projects :).
> What's the difference between the 2 softwares ?
> Do they support kafka ssl clusters ?
>
> Thanks,
> Gabriel.
>
> 2017-08-13 3:29 GMT+02:00 Ian Duffy :
>
>> Hi Jens,
>>
>> We did something similar to this at Zalando.
>>
>> https://github.com/zalando-incubator/remora
>>
>> It effectively supplies the kafka consumer group supply command as a http
>> endpoint.
>>
>> On 12 August 2017 at 16:42, Subhash Sriram 
>> wrote:
>>
>> > Hi Jens,
>> >
>> > Have you looked at Burrow?
>> >
>> > https://github.com/linkedin/Burrow/blob/master/README.md
>> >
>> > Thanks,
>> > Subhash
>> >
>> > Sent from my iPhone
>> >
>> > > On Aug 12, 2017, at 8:55 AM, Jens Rantil 
>> wrote:
>> > >
>> > > Hi,
>> > >
>> > > I am one of the maintainers of prometheus-kafka-consumer-
>> > group-exporter[1],
>> > > which exports consumer group offsets and lag to Prometheus. The way we
>> > > currently scrape this information is by periodically executing
>> > > `kafka-consumer-groups.sh --describe` for each group and parse the
>> > output.
>> > >
>> > > Recently the output from `kafka-consumer-groups.sh --describe` was
>> > > changed[2]. While I am working on a patch[3] to accomodate for the new
>> > > output format I was wondering if there is an easier, possibly more
>> > > stable[4] and more future proof, way for our project to extract the
>> > > information we are interested in. Does anyone know of a Go library
>> that
>> > > could extract the metrics we need? Or would it make sense to refactor
>> > > `kafka-consumer-groups.sh` to support a more structured output? I'd
>> love
>> > to
>> > > hear your input.
>> > >
>> > > Also, if Kafka exported the same metrics through JMX our project would
>> > not
>> > > exist, but maybe that's another story...
>> > >
>> > > Cheers,
>> > > Jens
>> > >
>> > > [1] https://github.com/kawamuray/prometheus-kafka-consumer-
>> > group-exporter
>> > > [2]
>> > > https://github.com/kawamuray/prometheus-kafka-consumer-
>> > group-exporter/issues/24
>> > > [3]
>> > > https://github.com/kawamuray/prometheus-kafka-consumer-
>> > group-exporter/pull/29
>> > > [4] We've also encountered `kafka-consumer-groups.sh` hanging a few
>> times
>> > > in production. There's a race condition somewhere in the script, most
>> > > likely when a topic is rebalancing. Currently we kill the process if
>> it
>> > > doesn't finish within a timeout. See
>> > > https://github.com/kawamuray/prometheus-kafka-consumer-
>> > group-exporter/blob/e4cdc3b1245f636d89d7e227066f02
>> > 578d732165/kafka/collector.go#L44
>> > > .
>> > >
>> > > --
>> > > Want to communicate with me securely? You can find my PGP key here
>> > > .
>> >
>>
>
>

Re: Querying consumer groups programmatically (from Golang)

[Dan Markhasin]

We are also collecting consumer group metrics from Kafka - we didn't want
to add extra unnecessary dependencies (such as burrow, which is also
overkill for what we need), so we just run a script every minute on the
brokers that parses the output of kafka-consumer-groups.sh and uploads it
to an http-listening logstash, which then indexes it in our central
ElasticSearch cluster where we keep all of our application metrics. Then,
it's easy to visualize the data with Kibana.

Unfortunately, as Jens pointed out, the output format seems to change a lot
(it changed between 0.10.0.1 to 0.10.1.0 if I'm not mistaken, and has
changed again in 0.11.0.0) and each time we upgrade we have to adjust our
script to account for the format difference.
I agree it would be great if Kafka either didn't change the format so much
or better yet exposed it via JMX.

Dan

On 12 August 2017 at 15:55, Jens Rantil  wrote:

> Hi,
>
> I am one of the maintainers of prometheus-kafka-consumer-
> group-exporter[1],
> which exports consumer group offsets and lag to Prometheus. The way we
> currently scrape this information is by periodically executing
> `kafka-consumer-groups.sh --describe` for each group and parse the output.
>
> Recently the output from `kafka-consumer-groups.sh --describe` was
> changed[2]. While I am working on a patch[3] to accomodate for the new
> output format I was wondering if there is an easier, possibly more
> stable[4] and more future proof, way for our project to extract the
> information we are interested in. Does anyone know of a Go library that
> could extract the metrics we need? Or would it make sense to refactor
> `kafka-consumer-groups.sh` to support a more structured output? I'd love to
> hear your input.
>
> Also, if Kafka exported the same metrics through JMX our project would not
> exist, but maybe that's another story...
>
> Cheers,
> Jens
>
> [1] https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter
> [2]
> https://github.com/kawamuray/prometheus-kafka-consumer-
> group-exporter/issues/24
> [3]
> https://github.com/kawamuray/prometheus-kafka-consumer-
> group-exporter/pull/29
> [4] We've also encountered `kafka-consumer-groups.sh` hanging a few times
> in production. There's a race condition somewhere in the script, most
> likely when a topic is rebalancing. Currently we kill the process if it
> doesn't finish within a timeout. See
> https://github.com/kawamuray/prometheus-kafka-consumer-
> group-exporter/blob/e4cdc3b1245f636d89d7e227066f02
> 578d732165/kafka/collector.go#L44
> .
>
> --
> Want to communicate with me securely? You can find my PGP key here
> .
>