[
https://issues.apache.org/jira/browse/KAFKA-20035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18049813#comment-18049813
]
Chia-Ping Tsai commented on KAFKA-20035:
----------------------------------------
To address this, the Group Coordinator should be able to distinguish between
"pre-existing partitions" (which should follow the user's config) and "newly
created partitions" (which should start from the beginning)
*Snapshot Partition Count:* When a group first subscribes to a topic, the
Coordinator records the current partition count of that topic (e.g.,
{{{}initial_partition_count = 3{}}}) and persists it in the group metadata.
*Signal "Newly Added" Partitions via Heartbeat* Instead of enforcing the offset
reset logic on the server side, the Coordinator simply identifies partitions
created after the group's subscription (where {{{}partition_id >=
initial_partition_count{}}}). It will marks these partitions with a flag (e.g.,
{{{}is_newly_created: true{}}}) in the {{ConsumerGroupHeartbeatResponse}} sent
to the consumer.
*Client-Side Adaptive Strategy* The Consumer client receives this signal. If
the client is configured with {{{}auto.offset.reset = latest{}}}, it can use
this flag to intelligently switch to {{earliest}} for these specific newly
added partitions. This ensures the consumer captures all data generated since
the expansion, while respecting the user's configuration for existing
partitions.
> Prevent data loss during partition expansion by enforcing "earliest" offset
> reset for dynamically added partitions
> ------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20035
> URL: https://issues.apache.org/jira/browse/KAFKA-20035
> Project: Kafka
> Issue Type: Bug
> Reporter: Chia-Ping Tsai
> Assignee: Chia-Ping Tsai
> Priority: Critical
>
> Currently, when a consumer group is configured with {{{}auto.offset.reset =
> latest{}}}, dynamically adding new partitions to a subscribed topic can lead
> to data loss due to a race condition.
> The scenario is as follows:
> # A group subscribes to a topic with {{{}auto.offset.reset = latest{}}}.
> # The topic is expanded (e.g., from 3 to 4 partitions).
> # Producers immediately start writing data to the new partition (Partition
> 3).
> # The Group Coordinator detects the change and assigns Partition 3 to a
> member.
> # The member initializes the partition. Since there is no committed offset,
> it applies the {{latest}} policy.
> # *Result:* Any messages written to Partition 3 between step 3 and step 5
> are skipped and lost.
> From a user's perspective, {{latest}} should mean "start consuming from the
> point of subscription," not "skip data from newly created infrastructure."
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
