Auto rebalancing is a very important feature to run Kafka in a production environment. Given the confluent already have this feature, are there any space that the open source version could have this feature as well? Or, is it the idea that opensource version shouldn't be used in a high load production environment?
________________________________ 发件人: sunil chaudhari <sunilmchaudhar...@gmail.com> 发送时间: 2022年10月27日 3:11 收件人: users@kafka.apache.org <users@kafka.apache.org> 主题: Re: Balancing traffic between multiple directories Hi Lehar, You are right. There is no better way in open source Kafka. However confluent has something called as Auto Rebalancing feature. Can you check if there is free version with this feature? It start balancing of brokers automatically when it see there is uneven distribution of partitions. Regards, Sunil. On Wed, 26 Oct 2022 at 12:03 PM, Lehar Jain <leha...@media.net.invalid> wrote: > Hey Andrew, > > Thanks for the reply. Currently, we are using the same method as you > described. Wanted to make sure if there is a better way. > > It seems there isn't currently. So we will keep using this only. > > On Tue, Oct 25, 2022 at 7:23 PM Andrew Grant <agr...@confluent.io.invalid> > wrote: > > > Hey Lehar, > > > > > > I don’t think there’s a way to control this during topic creation. I just > > took a look through > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala > > and it does appear partition assignment does not account for each > broker’s > > different log directories. I also took a look at the kafka-topics.sh > script > > and it has a --replica-assignment argument but that looks to only allow > > specifying brokers. During topic creation, once a replica has been > chosen I > > think we then choose the directory with the fewest number of partitions - > > see > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L1192 > > > > > > What I think you can do is move existing partitions around with the > > kafka-reassign-partitions.sh script. From running the command locally: > > > > > > --reassignment-json-file <String: The JSON file with the partition > > > > manual assignment json file path> reassignment configurationThe > > format > > > > to use is - > > > > {"partitions": > > > > [{"topic": "foo", > > > > "partition": 1, > > > > "replicas": [1,2,3], > > > > "log_dirs": > > ["dir1","dir2","dir3"] > > > > }], > > > > "version":1 > > > > } > > > > Note that "log_dirs" is optional. > > When > > > > it is specified, its length > must > > > > equal the length of the > replicas > > > > list. The value in this list > can > > be > > > > either "any" or the absolution > > path > > > > of the log directory on the > > broker. > > > > If absolute log directory path > is > > > > specified, the replica will be > > moved > > > > to the specified log directory > on > > > > the broker. > > > > > > There’s the log_dirs field you can use in the JSON file to move > partitions > > between directories. > > > > > > Hope that helps a bit. > > > > > > Andrew > > > > On Tue, Oct 25, 2022 at 6:56 AM Lehar Jain <leha...@media.net.invalid> > > wrote: > > > > > Hey, > > > > > > We run Kafka brokers with multiple log directories. I wanted to know > how > > > Kafka balances traffic between various directories. Can we have our own > > > strategy to distribute different partitions to different directories. > As > > > currently, we are facing an imbalance in sizes of the aforementioned > > > directories, some directories have a lot of empty space whereas others > > are > > > getting filled quickly. > > > > > > > > > Regards > > > > > >
