[
https://issues.apache.org/jira/browse/KAFKA-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806253#comment-15806253
]
Ryan P commented on KAFKA-4606:
-------------------------------
Artificial was the wrong word, but I still think an argument can be made to
allow for consumer offsets topic to expand. Either way I was hoping to spark
some conversation about how to handle this situation more gracefully if at all.
Users have and will continue to expand this topic should the deme it necessary.
This causes some unexpected behavior as some brokers will calculate the
consumer coordinator differently than the brokers who were restarted after the
expansion.
> make getOffsetsTopicPartitionCount mutable
> --------------------------------------------
>
> Key: KAFKA-4606
> URL: https://issues.apache.org/jira/browse/KAFKA-4606
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Ryan P
>
> Under certain circumstances users may wish to increase the size of their
> __consumer_offsets topic to spread the load as their consumer load increases.
> The current limitation, in which we try to prevent the topic's expansion
> makes this incredibly difficult. This is done intentionally to reduce the
> risk of unexpected behavior once the group id has been assigned a new
> coordinator.
> Instead Kafka should consider ways to handle this operation more gracefully
> with the following behaviors.
> 1. Calculate the number of __consumer_offsets partitions from the metadata
> cache with each metadata update
> 2. Force a rebalance after the new partitions are added to the topic. *I'm
> sure to take into consideration here as well. Alternatively the operation
> could be documented as risky emphasizing the behavior of the reset policy
> under these conditions.
> Currently the number of partitions for the consumer_offsets topic is fixed.
> This mean you will need to restart each broker to ensure that coordinator
> discovery happens as expected. That seems less than ideal to me for what
> seems to be a fairly artificial limitation of the consumer group
> implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
