Re: Reading consumer offsets in Kafka 0.9.0 and later

[Dana Powers]

Hi Marcos,

ConsumerMetadata* was renamed to GroupCoordinator* in 0.9 . the api
protocol is unchanged.

However, the new Java clients use non-blocking network channels. It looks
like the example code may reference the deprecated, or
soon-to-be-deprecated, Scala client.

Rather than roll your own monitoring, you might take a look at some of the
options already out there:

https://github.com/linkedin/Burrow
https://github.com/quantifind/KafkaOffsetMonitor

You might also consider monitoring JMX data (though for consumer lag this
will be limited to Java clients), described well here:

http://docs.confluent.io/2.0.1/kafka/monitoring.html

Hope this helps,

-Dana
We're building an application to monitor our Kafka consumers, and since the
offsets are now not stored in Zookeeper as before, I was trying to use the
following example in the Kafka website:

https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka

However, it seems either the example is stale, or it doesn't apply for the
latest 0.9+ Kafka releases.  Specifically, the ConsumerMetadataRequest
doesn't seem to exist in the latest releases, so I can't even instantiate
the class.

Can you point us to a current example of how we can get consumer offsets
from Kafka?

Thanks,

Marcos Juarez

Re: Reading consumer offsets in Kafka 0.9.0 and later

[Jason Gustafson]

Hi Marcos,

We should really update that wiki! We renamed ConsumerMetadataRequest to
GroupCoordinatorRequest in 0.9.0 because of the coordinator's more general
role. You might also want to take a look at consumer-groups.sh, which is
shipped with the release.

-Jason

On Thu, Mar 31, 2016 at 3:32 PM, Marcos Juarez  wrote:

> We're building an application to monitor our Kafka consumers, and since the
> offsets are now not stored in Zookeeper as before, I was trying to use the
> following example in the Kafka website:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
>
> However, it seems either the example is stale, or it doesn't apply for the
> latest 0.9+ Kafka releases.  Specifically, the ConsumerMetadataRequest
> doesn't seem to exist in the latest releases, so I can't even instantiate
> the class.
>
> Can you point us to a current example of how we can get consumer offsets
> from Kafka?
>
> Thanks,
>
> Marcos Juarez
>

Reading consumer offsets in Kafka 0.9.0 and later

[Marcos Juarez]

We're building an application to monitor our Kafka consumers, and since the
offsets are now not stored in Zookeeper as before, I was trying to use the
following example in the Kafka website:

https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka

However, it seems either the example is stale, or it doesn't apply for the
latest 0.9+ Kafka releases.  Specifically, the ConsumerMetadataRequest
doesn't seem to exist in the latest releases, so I can't even instantiate
the class.

Can you point us to a current example of how we can get consumer offsets
from Kafka?

Thanks,

Marcos Juarez