chia7712 commented on code in PR #22784: URL: https://github.com/apache/kafka/pull/22784#discussion_r3609144548
########## docs/operations/basic-kafka-operations.md: ########## @@ -62,6 +62,18 @@ $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic my_topic * **Key Distribution Changes**: If data is partitioned by `hash(key) % number_of_partitions`, the default partitioner's mapping logic changes when the partition count increases. This means that messages with the same key may be routed to different partitions after the expansion, potentially affecting message ordering guarantees for existing keys. Kafka will not attempt to automatically redistribute existing data. * **Potential Data Loss with `auto.offset.reset=latest`**: Existing consumers configured with `auto.offset.reset=latest` might miss messages produced to the new partitions during the window between partition creation and consumer discovery. This occurs because consumers may not immediately detect the new partitions, and any messages produced to those partitions before the consumer rebalances will be skipped. + **Recommendation:** Use `auto.offset.reset=by_duration:<duration>` instead of `latest` for consumers that read from topics whose partition count may increase. When a partition has no committed offset, `by_duration` performs a `ListOffsets` lookup for `now() - duration` to determine the starting position. If the target timestamp is earlier than the partition's creation time, the lookup returns offset `0`, ensuring that records produced during the partition-discovery gap are still consumed. Consumers with a valid committed offset are unaffected, so restarts do not replay historical data. + + Size `<duration>` to cover the worst-case partition-discovery latency for the group protocol in use: + + | Group protocol | Discovery mechanism | Governing config (default) | Recommended by_duration value | + |----------------------|------------------------------------------------|-----------------------------------------------------------------|-------------------------------------------------------| + | `consumer` (KIP-848) | Server pushes assignment on the next heartbeat | `group.consumer.heartbeat.interval.ms` (`5000` ms, server-side) | `PT5S` or slightly higher than the heartbeat interval | + | `classic` | Client-side periodic metadata refresh | `metadata.max.age.ms` (`300000` ms, client-side) | `PT5M` or slightly higher than `metadata.max.age.ms` | + + If either the heartbeat interval or `metadata.max.age.ms` is tuned away from the default, increase `<duration>` accordingly. `<duration>` uses the ISO-8601 duration format (`PnDTnHnMn.nS`). The trade-off is bounded: on the first assignment of a partition, the consumer replays at most `<duration>` worth of records, which is negligible compared with a full historical replay. + + **Clock synchronization requirement:** `by_duration` computes the target timestamp using the client's wall-clock time (`now() - duration`), while the `ListOffsets` lookup compares it against broker-side message timestamps. Accurate clock synchronization between clients and brokers is therefore required. If the client clock runs ahead of the brokers, the consumer may start from a position that is too recent and skip records. If the client clock lags behind, the consumer may start too early and replay historical data. Review Comment: Would you highlight that using `by_duration` will cause the consumer to read more of the backlogs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
