[
https://issues.apache.org/jira/browse/KAFKA-7690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated KAFKA-7690:
-----------------------------
Description:
h3. *Background*
if target topic partitions lager than broker size when create a topic or add
partition, one broker will be assigned more than one partition. if current all
disk is not balance, such as one disk has one partition and the other one has
four partitions due to topic delete or others, the mutil partitions will be all
allocated in the first disk, and if the target topic has a large flow, it is
easily to fill up the disk io.
h3. *Improvement strategy*
when mutil ** partition is going to be allocated on a broker, the strategy is
as follow:
1、calculate the target topic partition count and total partition count on each
disk.
topic count
2、sorted by the target topic partition count wich ascending order, if the
target topic partition count is equal, sorted by the total partitions on each
disk.
h3. *Example*
||disk||target topic partition count||total partition count||
|disk1|0|11|
|disk2|0|9|
when tow partitions are assigned on this broker, if use origin strategy, the
result is as follows:
||disk||target topic partition count||total partition count||
|disk1|0|11|
|disk2|2|11|
use new strategy, the result is as follows:
||disk||target topic partition count||total partition count||
|disk1|1|12|
|disk2|1|10|
if the topic has a huge flow such as 50MB/s per partition, it is easily to fill
up disk2 io if use origin strategy. However if use new strategy, it's well in
disk io rebalance.
h3. *Summary*
This strategy is good to build a big cluster and create a topic which has huge
amount partition, the disk io will be more balanced.
was:
h3. *Background*
if target topic partitions lager than broker size when create a topic or add
partition, one broker will be assigned more than one partition. if current all
disk is not balance, such as one disk has one partition and the other one has
four partitions due to topic delete or others, the mutil partitions will be all
allocated in the first disk, and if the target topic has a large flow, it is
easily to fill up the disk io.
h3. *Improvement strategy*
when mutil ** partition is going to be allocated on a broker, the strategy is
as follow:
1、calculate the target topic partition count and total partition count on each
disk.
topic count
2、sorted by the target topic partition count wich ascending order, if the
target topic partition count is equal, sorted by the total partitions on each
disk.
h3. *Example*
> Change disk allocation policy for multiple partitions on a broker when topic
> is created
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-7690
> URL: https://issues.apache.org/jira/browse/KAFKA-7690
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.10.2.0, 1.0.0, 2.0.0
> Reporter: haiyangyu
> Priority: Major
>
> h3. *Background*
> if target topic partitions lager than broker size when create a topic or add
> partition, one broker will be assigned more than one partition. if current
> all disk is not balance, such as one disk has one partition and the other one
> has four partitions due to topic delete or others, the mutil partitions will
> be all allocated in the first disk, and if the target topic has a large flow,
> it is easily to fill up the disk io.
> h3. *Improvement strategy*
> when mutil ** partition is going to be allocated on a broker, the strategy is
> as follow:
> 1、calculate the target topic partition count and total partition count on
> each disk.
> topic count
> 2、sorted by the target topic partition count wich ascending order, if the
> target topic partition count is equal, sorted by the total partitions on each
> disk.
> h3. *Example*
>
> ||disk||target topic partition count||total partition count||
> |disk1|0|11|
> |disk2|0|9|
> when tow partitions are assigned on this broker, if use origin strategy, the
> result is as follows:
> ||disk||target topic partition count||total partition count||
> |disk1|0|11|
> |disk2|2|11|
> use new strategy, the result is as follows:
> ||disk||target topic partition count||total partition count||
> |disk1|1|12|
> |disk2|1|10|
> if the topic has a huge flow such as 50MB/s per partition, it is easily to
> fill up disk2 io if use origin strategy. However if use new strategy, it's
> well in disk io rebalance.
> h3. *Summary*
> This strategy is good to build a big cluster and create a topic which has
> huge amount partition, the disk io will be more balanced.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
