Ryan Berdeen created KAFKA-1677:
-----------------------------------
Summary: Governor on concurrent replica reassignments
Key: KAFKA-1677
URL: https://issues.apache.org/jira/browse/KAFKA-1677
Project: Kafka
Issue Type: New Feature
Components: controller
Reporter: Ryan Berdeen
Assignee: Neha Narkhede
We have seen a cluster be killed via too many concurrent partition transfers.
An ideal solution is a configuration setting to limit the number of concurrent
transfers per host (dynamically tunable). (eg: transfer_limit defined in
http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/#Ring).
To work around this, we generate our assignments, then use a tool to feed the
reassignments in small batches.
The size of the batch is based on either
* *the number partitions*, e.g., reassign all replicas for the first 2
partitions that have any moves
* *the number of individual replica moves*, e.g. when reassigning \[1,2,3,4] to
\[5,6,7,8], first reassign to \[5,6,3,4] then reassign to \[5,6,7,8]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
