Hi all,

Recently we upgraded our application from the more primitive Java client APIs 
(kafka.javaapi.consumer.SimpleConsumer, kafka.api.FetchRequest and friends) to 
the more friendly poll-based org.apache.kafka.clients.consumer.KafkaConsumer 
using Kafka Java client libraries version 1.1.0.

The upgrade went fine and meant we could remove a LOT of custom code we had 
previously needed to use. This was also released into a version of the 
application that went into QA / staging environments of a client of ours. The 
upgrade (environment was not wiped) in the staging environment went fine as 
well without any errors. Due to an unrelated serious issue that we discovered a 
couple of days later, we had to advice the client a roll-back from the new 
application version (still had not gone into production). We didn't think this 
would lead to any issues.

What DID occur when the application was downgraded in the staging client 
environment, however, was Kafka clients starting to get a lot of offset fetch 
errors:

WARN  [2019-04-24 01:19:43,269] custom.application.package.KafkaClient: Failed 
to fetch messages from Kafka. Error code: 1

Now, error code 1 means offset out of range as far as I'm aware. I would not 
have expected this since I was pretty sure offsets were stored in ZooKeeper 
with the old SimpleConsumer (I can see the offsets being stored in nodes like 
/consumers/<CONSUMER GROUP ID>/offsets/<TOPIC>/0) and in Kafka internal topic 
__consumer_offsets with the new KafkaConsumer (I can NOT see offsets in 
ZooKeeper for a couple of new-style consumers we already had).

The ZooKeeper-stored consumer offsets should also never have been "out of 
range" since there would strictly always be more events since they were last 
used. Could this error simply mean that the ZooKeeper offsets was TOO OLD 
(meaning Kafka had thrown away messages)?

I wouldn't have been surprised to see some kind of error if the offset data was 
shared.

This is NOT a serious issue for us at the moment, we were just surprised that 
the downgrade ran into errors.

Regards,
Andreas Nilsson

Reply via email to