Re: Offset Tools

[Jörg Wagner]

Hello Jan,

Thanks for the information and also offer. I might come back to that 
some day, but for now we seem to have located the issue:
the consumer was noch consuming as expected. Hence the consumer group is 
not yet known to kafka and can not be seen.


Cheers
Jörg

On 11.07.2016 14:07, Jan Omar wrote:

Hi Joerg,

We recently built a tool for fetching current offsets (per partition) for a 
given consumer group. Also for setting the group to a specific offset.
It even allows to reset a consumer group to a given timestamp by running bisect 
(binary search) on the messages.

Unfortunately we're using a proprietary message format, that's why we don't 
have any plans (or capacity) to open source it at the moment.

However builiding that tool was straight forward, it shouldn't take you more 
than a day or two to build something similar. Ping me if you need some help.

Regards

Jan


On 11 Jul 2016, at 13:00, Jörg Wagner  wrote:

Hello!

We recently switched to Kafka 0.9.0.1 and currently I don't seem to be able to 
figure out how to read the consumer offsets via cli. We are using the 0.9.0.1 
new consumer and are storing the offsets in kafka.

Status:
kafka-consumer-offset-checker.sh is old and deprecated, points to 
kafka-consumer-groups.sh
kafka-consumer-groups.sh in old consumer mode shows lots of data, but unknown 
for offsets and lag
kafka-consumer-groups.sh in new consumer mode shows only one single consumer 
group, not the one(s) expected. The one available though shows all the correct 
data I would like to see for a group not shown.

Short: since 0.9.0.1 I can't see the consumer offsets anymore.

Anyone else experiencing this?

Cheers
Jörg


--
Mit freundlichem Gruß

Jörg Wagner

Systemadministrator
Search & Account Security

1&1 Mail & Media Development  & Technology GmbH | Sapporobogen 6-8 | 80637 
München | Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagn...@1und1.de | Web: www.1und1.de

Amtsgericht Montabaur, HRB 5452

Geschäftsführer: Frank Einhellinger, Thomas Ludwig, Jan Oetjen


Member of United Internet

Re: Offset Tools

[Jan Omar]

Hi Joerg,

We recently built a tool for fetching current offsets (per partition) for a 
given consumer group. Also for setting the group to a specific offset.
It even allows to reset a consumer group to a given timestamp by running bisect 
(binary search) on the messages. 

Unfortunately we're using a proprietary message format, that's why we don't 
have any plans (or capacity) to open source it at the moment.

However builiding that tool was straight forward, it shouldn't take you more 
than a day or two to build something similar. Ping me if you need some help.

Regards

Jan 

> On 11 Jul 2016, at 13:00, Jörg Wagner  wrote:
> 
> Hello!
> 
> We recently switched to Kafka 0.9.0.1 and currently I don't seem to be able 
> to figure out how to read the consumer offsets via cli. We are using the 
> 0.9.0.1 new consumer and are storing the offsets in kafka.
> 
> Status:
> kafka-consumer-offset-checker.sh is old and deprecated, points to 
> kafka-consumer-groups.sh
> kafka-consumer-groups.sh in old consumer mode shows lots of data, but unknown 
> for offsets and lag
> kafka-consumer-groups.sh in new consumer mode shows only one single consumer 
> group, not the one(s) expected. The one available though shows all the 
> correct data I would like to see for a group not shown.
> 
> Short: since 0.9.0.1 I can't see the consumer offsets anymore.
> 
> Anyone else experiencing this?
> 
> Cheers
> Jörg