Juho Autio created KAFKA-4299:
---------------------------------
Summary: Consumer offsets reset for all topics after increasing
partitions for one topic
Key: KAFKA-4299
URL: https://issues.apache.org/jira/browse/KAFKA-4299
Project: Kafka
Issue Type: Bug
Affects Versions: 0.10.0.1
Reporter: Juho Autio
I increased partitions for one existing topic (2->10), but was surprised to see
that it entirely reset the committed offsets of my consumer group.
All topics & partitions were reset to the earliest offset available, and the
consumer read everything again.
Documentation doesn't mention anything like this. Is this how it's supposed to
work, or a bug?
I would've expected the consumer offsets to not decrease at all, especially for
the topics that I didn't even touch.
For the altered topic I would've expected that consuming the previously
existing partitions 0 and 1 would've continued from the position where they
were, and naturally starting to read the new added partitions from 0.
I added partitions according to the "Modifying topics" section of Kafka 0.10.0
Documentation:
{quote}
To add partitions you can do
{code}
> bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic
> altered_topic --partitions 10
{code}
{quote}
Previously this topic had 2 partitions.
For the consumer I'm using
{{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
And version is:
{code}
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.0.1</version>
{code}
Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
This is quite problematic because we can't afford waiting for consumers to read
the full buffer from the beginning (for all topics!) when increasing partitions
for a topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
