Hi Gabriele, Thanks a lot for the protocol reference to the default value. It looks like this case isn't handled in the Java AdminClient then. I'll follow up on the dev list.
Regards, Sean On Wed, Feb 5, 2020 at 1:35 AM Gabriele Paggi <[email protected]> wrote: > Hi Sean, > > Unless this has changed, this is the expected behavior when there's no > offset associated with a topic-partition under a given consumer group, > in which case the Kafka broker will return -1: > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetFetchRequest > "Note that if there is no offset associated with a topic-partition > under that consumer group the broker does not set an error code (since > it is not really an error), but returns empty metadata and sets the > offset field to -1." > > I addressed that issue in another Kafka Exporter by forcibly setting > the lag to -1 so that I can monitor simple consumers, which don't > support discovery, that are not consuming yet from newly added > partitions: https://github.com/danielqsj/kafka_exporter/pull/17 > > > On Tue, 4 Feb 2020 at 20:10, Sean Glover <[email protected]> > wrote: > > > > Hi, > > > > I maintain Kafka Lag Exporter (an offset lag/latency monitoring tool). I > > use the Apache Kafka Java AdminClient and KafkaConsumer to retrieve > offset > > information to calculate lag and other metrics, similar to how the > > kafka-consumer-groups.sh bin script works. A few users have reported > Kafka > > Lag Exporter crashing due to an internal AdminClient exception caused by > > the response from OffsetFetchRequest: > "java.lang.IllegalArgumentException: > > Invalid negative offset" (see stack trace below). > > > > We found out that the user is using Azure EventHub For Kafka and that > > partition offsets for consumer groups are initialized with offset -1. > I've > > never heard of this practice before so I thought I'd ask if this is an > > initialization default, or a default that users/services do commonly > when > > users don't want to use Kafka managed offsets. Since > > kafka-consumer-groups.sh relies on AdminClient, this issue can be > > reproduced using that script as well. > > > > Kafka Lag Exporter issue: > > https://github.com/lightbend/kafka-lag-exporter/issues/120 > > > > Stack trace: > > > > Caused by: java.lang.IllegalArgumentException: Invalid negative offset > > at > > > org.apache.kafka.clients.consumer.OffsetAndMetadata.<init>(OffsetAndMetadata.java:50) > > at > > > org.apache.kafka.clients.admin.KafkaAdminClient$25.handleResponse(KafkaAdminClient.java:3018) > > at > > > org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1076) > > at > > > org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1204) > > ... 1 common frames omitted > > > > Regards, > > Sean > > > > -- > Gabriele >
