[ https://issues.apache.org/jira/browse/KAFKA-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haruki Okada updated KAFKA-12261: --------------------------------- Description: As of now, auto.offset.reset of ConsumerConfig is "latest" by default. This could be a pitfall that causes message delivery loss when we split topic's partitions like below: Say we have a topic-X which have only 1 partition. # split topic-X to 2 partitions by kafka-topics.sh --alter --topic topic-X --partitions 2 (topic-X-1 is added) # producer knows that new partitions are added by refreshing metadata. starts to produce to topic-X-1 # bit later, consumer knows that new partitions are added and triggering consumer rebalance, then starts consuming topic-X-1 * ** upon starting consumption, it resets its offset to log-end-offset If the producer sent several records before 3, they could be not-delivered to the consumer. This behavior isn't preferable in most cases, so it should be documented in AUTO_OFFSET_RESET_DOC at least. was: As of now, auto.offset.reset of ConsumerConfig is "latest" by default. This could be a pitfall that causes message delivery loss when we split topic's partitions like below: Say we have a topic-X which have only 1 partition. # split topic-X to 2 partitions by kafka-topics.sh --alter --topic topic-X --partitions 2 (topic-X-1 is added) # producer knows that new partitions are added by refreshing metadata. starts to produce to topic-X-1 # bit later, consumer knows that new partitions are added and triggering consumer rebalance, then starts consuming topic-X-1 * ** upon starting consumption, it resets its offset to log-end-offset If the producer sent several records before 3, they could be not-delivered to the consumer. This behavior isn't preferable in most cases, so auto.offset.reset should be set to "earliest" by default to avoid this pitfall. > Splitting partition causes message loss for consumers with > auto.offset.reset=latest > ----------------------------------------------------------------------------------- > > Key: KAFKA-12261 > URL: https://issues.apache.org/jira/browse/KAFKA-12261 > Project: Kafka > Issue Type: Improvement > Affects Versions: 2.7.0 > Reporter: Haruki Okada > Assignee: Luke Chen > Priority: Minor > > As of now, auto.offset.reset of ConsumerConfig is "latest" by default. > > This could be a pitfall that causes message delivery loss when we split > topic's partitions like below: > Say we have a topic-X which have only 1 partition. > # split topic-X to 2 partitions by kafka-topics.sh --alter --topic topic-X > --partitions 2 (topic-X-1 is added) > # producer knows that new partitions are added by refreshing metadata. > starts to produce to topic-X-1 > # bit later, consumer knows that new partitions are added and triggering > consumer rebalance, then starts consuming topic-X-1 > * > ** upon starting consumption, it resets its offset to log-end-offset > If the producer sent several records before 3, they could be not-delivered to > the consumer. > > > This behavior isn't preferable in most cases, so it should be documented in > AUTO_OFFSET_RESET_DOC at least. -- This message was sent by Atlassian Jira (v8.3.4#803005)