Hello, I am using Kafka broker and Java client library v 0.11.0.0.
When I restart my Kafka consumer application which uses Java Kafka client library to retrieve messages, I notice that for each partition, the message associated with the last offset that was committed successfully gets re-consumed. I am "not" using auto-commit feature of the Java Kafka client library. So for example, for some topic partition 1 - 1. Consumer application commits offset 100 manually. 2. Consumer application gets restarted. 3. Consumer polls for messages from Kafka. This time, the first message polled from Kafka is the one with offset 100. Shouldn't the new batch of messages polled from Kafka start from offset 101? This causes duplication of messages - one for each partition on application restart. I understand that message duplication is possible in other scenarios too, but this particular behavior coming from the client library seemed strange. In my case, I am writing a framework for our use case, and we would like to avoid this specific message duplication scenario. Questions: * Wondering if there is any reason for this or am I missing something in my code. * If this was done by design, what would be easiest way for user application to circumvent this behavior? Thanks in advance.
