Ryan P created KAFKA-4606:
-----------------------------
Summary: make getOffsetsTopicPartitionCount mutable
Key: KAFKA-4606
URL: https://issues.apache.org/jira/browse/KAFKA-4606
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Ryan P
Under certain circumstances users may wish to increase the size of their
__consumer_offsets topic to spread the load as their consumer load increases.
The current limitation, in which we try to prevent the topic's expansion makes
this incredibly difficult. This is done intentionally to reduce the risk of
unexpected behavior once the group id has been assigned a new coordinator.
Instead Kafka should consider ways to handle this operation more gracefully
with the following behaviors.
1. Calculate the number of __consumer_offsets partitions from the metadata
cache with each metadata update
2. Force a rebalance after the new partitions are added to the topic. *I'm sure
to take into consideration here as well. Alternatively the operation could be
documented as risky emphasizing the behavior of the reset policy under these
conditions.
Currently the number of partitions for the consumer_offsets topic is fixed.
This mean you will need to restart each broker to ensure that coordinator
discovery happens as expected. That seems less than ideal to me for what seems
to be a fairly artificial limitation of the consumer group implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
