[jira] [Commented] (KAFKA-4299) Consumer offsets reset for all topics after increasing partitions for one topic
[
https://issues.apache.org/jira/browse/KAFKA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574930#comment-15574930
]
Juho Autio commented on KAFKA-4299:
---
I altered some topics in test environment and no offsets were reset this time.
I will try altering a topic next week in production to see if it happens again.
> Consumer offsets reset for all topics after increasing partitions for one
> topic
> ---
>
> Key: KAFKA-4299
> URL: https://issues.apache.org/jira/browse/KAFKA-4299
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Juho Autio
>
> I increased partitions for one existing topic (2->10), but was surprised to
> see that it entirely reset the committed offsets of my consumer group.
> All topics & partitions were reset to the earliest offset available, and the
> consumer read everything again.
> Documentation doesn't mention anything like this. Is this how it's supposed
> to work, or a bug?
> I would've expected the consumer offsets to not decrease at all, especially
> for the topics that I didn't even touch.
> For the altered topic I would've expected that consuming the previously
> existing partitions 0 and 1 would've continued from the position where they
> were, and naturally starting to read the new added partitions from 0.
> I added partitions according to the "Modifying topics" section of Kafka
> 0.10.0 Documentation:
> {quote}
> To add partitions you can do
> {code}
> > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic
> altered_topic --partitions 10
> {code}
> {quote}
> Previously this topic had 2 partitions.
> For the consumer I'm using
> {{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
> And version is:
> {code}
> org.apache.kafka
> kafka_2.11
> 0.10.0.1
> {code}
> Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
> This is quite problematic because we can't afford waiting for consumers to
> read the full buffer from the beginning (for all topics!) when increasing
> partitions for a topic.
> Some possibly relevant settings we have for the consumer:
> {code}
> kafka.partition.assignment.strategy = "range"
> kafka.auto.offset.reset = "smallest"
> kafka.auto.commit.enable = "false"
> kafka.offsets.storage = "kafka"
> kafka.dual.commit.enabled = false
> kafka.consumer.timeout.ms = "2000"
> kafka.auto.create.topics.enable = true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4299) Consumer offsets reset for all topics after increasing partitions for one topic
[
https://issues.apache.org/jira/browse/KAFKA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574324#comment-15574324
]
Juho Autio commented on KAFKA-4299:
---
Thanks for testing [~vahid]. What about offsets of the pre-existing partitions
in the topic that you altered? Were those reset?
I updated to issue description some consumer parameters that we have set.
> Consumer offsets reset for all topics after increasing partitions for one
> topic
> ---
>
> Key: KAFKA-4299
> URL: https://issues.apache.org/jira/browse/KAFKA-4299
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Juho Autio
>
> I increased partitions for one existing topic (2->10), but was surprised to
> see that it entirely reset the committed offsets of my consumer group.
> All topics & partitions were reset to the earliest offset available, and the
> consumer read everything again.
> Documentation doesn't mention anything like this. Is this how it's supposed
> to work, or a bug?
> I would've expected the consumer offsets to not decrease at all, especially
> for the topics that I didn't even touch.
> For the altered topic I would've expected that consuming the previously
> existing partitions 0 and 1 would've continued from the position where they
> were, and naturally starting to read the new added partitions from 0.
> I added partitions according to the "Modifying topics" section of Kafka
> 0.10.0 Documentation:
> {quote}
> To add partitions you can do
> {code}
> > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic
> altered_topic --partitions 10
> {code}
> {quote}
> Previously this topic had 2 partitions.
> For the consumer I'm using
> {{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
> And version is:
> {code}
> org.apache.kafka
> kafka_2.11
> 0.10.0.1
> {code}
> Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
> This is quite problematic because we can't afford waiting for consumers to
> read the full buffer from the beginning (for all topics!) when increasing
> partitions for a topic.
> Some possibly relevant settings we have for the consumer:
> {code}
> kafka.partition.assignment.strategy = "range"
> kafka.auto.offset.reset = "smallest"
> kafka.auto.commit.enable = "false"
> kafka.offsets.storage = "kafka"
> kafka.dual.commit.enabled = false
> kafka.consumer.timeout.ms = "2000"
> kafka.auto.create.topics.enable = true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4299) Consumer offsets reset for all topics after increasing partitions for one topic
[
https://issues.apache.org/jira/browse/KAFKA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573249#comment-15573249
]
Vahid Hashemian commented on KAFKA-4299:
I was not able to reproduce this using the console consumer. After increasing
the number of partitions of one topic, the offsets of the other topic in the
group remained unchanged.
> Consumer offsets reset for all topics after increasing partitions for one
> topic
> ---
>
> Key: KAFKA-4299
> URL: https://issues.apache.org/jira/browse/KAFKA-4299
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Juho Autio
>
> I increased partitions for one existing topic (2->10), but was surprised to
> see that it entirely reset the committed offsets of my consumer group.
> All topics & partitions were reset to the earliest offset available, and the
> consumer read everything again.
> Documentation doesn't mention anything like this. Is this how it's supposed
> to work, or a bug?
> I would've expected the consumer offsets to not decrease at all, especially
> for the topics that I didn't even touch.
> For the altered topic I would've expected that consuming the previously
> existing partitions 0 and 1 would've continued from the position where they
> were, and naturally starting to read the new added partitions from 0.
> I added partitions according to the "Modifying topics" section of Kafka
> 0.10.0 Documentation:
> {quote}
> To add partitions you can do
> {code}
> > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic
> altered_topic --partitions 10
> {code}
> {quote}
> Previously this topic had 2 partitions.
> For the consumer I'm using
> {{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
> And version is:
> {code}
> org.apache.kafka
> kafka_2.11
> 0.10.0.1
> {code}
> Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
> This is quite problematic because we can't afford waiting for consumers to
> read the full buffer from the beginning (for all topics!) when increasing
> partitions for a topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4299) Consumer offsets reset for all topics after increasing partitions for one topic
[
https://issues.apache.org/jira/browse/KAFKA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571675#comment-15571675
]
Ismael Juma commented on KAFKA-4299:
cc [~hachikuji]
> Consumer offsets reset for all topics after increasing partitions for one
> topic
> ---
>
> Key: KAFKA-4299
> URL: https://issues.apache.org/jira/browse/KAFKA-4299
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Juho Autio
>
> I increased partitions for one existing topic (2->10), but was surprised to
> see that it entirely reset the committed offsets of my consumer group.
> All topics & partitions were reset to the earliest offset available, and the
> consumer read everything again.
> Documentation doesn't mention anything like this. Is this how it's supposed
> to work, or a bug?
> I would've expected the consumer offsets to not decrease at all, especially
> for the topics that I didn't even touch.
> For the altered topic I would've expected that consuming the previously
> existing partitions 0 and 1 would've continued from the position where they
> were, and naturally starting to read the new added partitions from 0.
> I added partitions according to the "Modifying topics" section of Kafka
> 0.10.0 Documentation:
> {quote}
> To add partitions you can do
> {code}
> > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic
> altered_topic --partitions 10
> {code}
> {quote}
> Previously this topic had 2 partitions.
> For the consumer I'm using
> {{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
> And version is:
> {code}
> org.apache.kafka
> kafka_2.11
> 0.10.0.1
> {code}
> Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
> This is quite problematic because we can't afford waiting for consumers to
> read the full buffer from the beginning (for all topics!) when increasing
> partitions for a topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
