Repository: kafka Updated Branches: refs/heads/trunk e20eba958 -> 697811551
MINOR: Documentation for Rack Awareness Author: Ben Stopford <[email protected]> Reviewers: Ismael Juma <[email protected]> Closes #1369 from benstopford/rack-awareness-docs Project: http://git-wip-us.apache.org/repos/asf/kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/69781155 Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/69781155 Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/69781155 Branch: refs/heads/trunk Commit: 6978115514fc74021e4f2c761402d6b0c954d50c Parents: e20eba9 Author: Ben Stopford <[email protected]> Authored: Wed May 11 15:28:18 2016 +0100 Committer: Ismael Juma <[email protected]> Committed: Wed May 11 15:28:18 2016 +0100 ---------------------------------------------------------------------- docs/ops.html | 11 +++++++++++ 1 file changed, 11 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kafka/blob/69781155/docs/ops.html ---------------------------------------------------------------------- diff --git a/docs/ops.html b/docs/ops.html index f64a701..faf5453 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -98,6 +98,17 @@ Since running this command can be tedious you can also configure Kafka to do thi auto.leader.rebalance.enable=true </pre> +<h4><a id="basic_ops_racks" href="#basic_ops_racks">Balancing Replicas Across Racks</a></h4> +The rack awareness feature spreads replicas of the same partition across different racks. This extends the guarantees Kafka provides for broker-failure to cover rack-failure, limiting the risk of data loss should all the brokers on a rack fail at once. The feature can also be applied to other broker groupings such as availability zones in EC2. +<p></p> +You can specify that a broker belongs to a particular rack by adding a property to the broker config: +<pre> broker.rack=my-rack-id</pre> +When a topic is <a href="#basic_ops_add_topic">created</a>, <a href="#basic_ops_modify_topic">modified</a> or replicas are <a href="#basic_ops_cluster_expansion">redistributed</a>, the rack constraint will be honoured, ensuring replicas span as many racks as they can (a partition will span min(#racks, replication-factor) different racks). +<p></p> +The algorithm used to assign replicas to brokers ensures that the number of leaders per broker will be constant, regardless of how brokers are distributed across racks. This ensures balanced throughput. +<p></p> +However if racks are assigned different numbers of brokers, the assignment of replicas will not be even. Racks with fewer brokers will get more replicas, meaning they will use more storage and put more resources into replication. Hence it is sensible to configure an equal number of brokers per rack. + <h4><a id="basic_ops_mirror_maker" href="#basic_ops_mirror_maker">Mirroring data between clusters</a></h4> We refer to the process of replicating data <i>between</i> Kafka clusters "mirroring" to avoid confusion with the replication that happens amongst the nodes in a single cluster. Kafka comes with a tool for mirroring data between Kafka clusters. The tool reads from a source cluster and writes to a destination cluster, like this:
