Looks like it, you need to be sure the offset topic is using compaction, and the broker is set to enable compaction.
On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1...@gmail.com> wrote: > Hi, > > I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up > all old log segments(reach delete.retention time), I got unknown offset. > > bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer > --group testGroup --describe > GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER > testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2 > testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2 > > In this situation, I cannot consume anything using new-consumer java > driver if I disable auto-commit. > I think this happens because new-consumer driver stores offset in broker > as a topic(not in zookeeper), and after reaching delete.retention time, it > got deleted and becomes unknown. And since I disabled auto-commit, it can > never know where it is, then it cannot consume anything. > > Is this what happened here? What should I do in this situation? > > Thanks, > Jun
