[
https://issues.apache.org/jira/browse/KAFKA-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064740#comment-15064740
]
Gwen Shapira commented on KAFKA-3015:
-------------------------------------
+1
Currently we work around the issue by using RAID-0 to present all disks as one
huge directory to Kafka. This puts the burden on the users, who need to know
how to set up RAID 0 and also tune few parameters because our defaults are not
geared toward one directory with many disks.
Will be nice if Kafka will do the right thing here.
> Improve JBOD data balancing
> ---------------------------
>
> Key: KAFKA-3015
> URL: https://issues.apache.org/jira/browse/KAFKA-3015
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jay Kreps
>
> When running with multiple data directories (i.e. JBOD) we currently place
> partitions entirely within one data directory. This tends to lead to poor
> balancing across disks as some topics have more throughput/retention and not
> all disks get data from all topics. You can't fix this problem with smarter
> partition placement strategies because ultimately you don't know when a
> partition is created when or how heavily it will be used (this is a subtle
> point, and the tendency is to try to think of some more sophisticated way to
> place partitions based on current data size but this is actually
> exceptionally dangerous and can lead to much worse imbalance when creating
> many partitions at once as they would all go to the disk with the least
> data). We don't support online rebalancing across directories/disks so this
> imbalance is a big problem and limits the usefulness of this configuration.
> Implementing online rebalancing of data across disks without downtime is
> actually quite hard and requires lots of I/O since you have to actually
> rewrite full partitions of data.
> An alternative would be to place each partition in *all* directories/drives
> and round-robin *segments* within the partition across the directories. So
> the layout would be something like:
> drive-a/mytopic-0/
> 0000000.data
> 0000000.index
> 0024680.data
> 0024680.index
> drive-a/mytopic-0/
> 0012345.data
> 0012345.index
> 0036912.data
> 0036912.index
> This is a little harder to implement than the current approach but not very
> hard, and it is a lot easier than implementing online data balancing across
> disks while retaining the current approach. I think this could easily be done
> in a backwards compatible way.
> I think the balancing you would get from this in most cases would be good
> enough to make JBOD the default configuration. Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
