Hi Peter, AFAIK, everything depends on:
1) How you have configured your topic a) number of partitions (here I understand you have 15 partitions) b) partition replication configuration (each partition necessarily has a leader - primary responsible to hold the data - and for reads and writes) you can configure the topic to have a number of replicas 2) How you publish messages to the topic a) The publisher is responsible to choose the partition. This can be done consciously (by setting the partition id while sending the message to the topic) or unconsciously (by using the DefaultPartitioner or any other partitioner scheme). All messages sent to a specific partition will be written first to the leader (meaning that the disk configured for the partition leader will receive the load) and then replicated to the replica (followers). Kafka does not automatically distribute the data equally to the different brokers - you need to think about your architecture having that in mind. I hope it helps On Thu, Aug 6, 2020 at 10:23 PM Péter Nagykátai <st4r.f1...@gmail.com> wrote: > I initially started with one data disk (mounted solely to hold Kafka data) > and recently added a new one. > > On Thu, Aug 6, 2020 at 10:13 PM <manoj.agraw...@cognizant.com> wrote: > > > What do you mean older disk ? > > > > On 8/6/20, 12:05 PM, "Péter Nagykátai" <st4r.f1...@gmail.com> wrote: > > > > [External] > > > > > > Yeah, but it doesn't do that. My "older" disks have ~70 partitions, > the > > newer ones ~5 partitions. That's why I'm asking what went wrong. > > > > On Thu, Aug 6, 2020 at 8:35 PM <manoj.agraw...@cognizant.com> wrote: > > > > > Kafka evenly distributed number of partition on each disk so in > > your case > > > every disk should have 3/2 topic partitions . > > > It is producer job to evenly produce data by partition key to > topic > > > partition . > > > How it partition key , it is auto generated or producer sending key > > along > > > with message . > > > > > > > > > On 8/6/20, 7:29 AM, "Péter Nagykátai" <st4r.f1...@gmail.com> > wrote: > > > > > > [External] > > > > > > > > > Hello, > > > > > > I have a Kafka cluster with 3 brokers (v2.3.0) and each broker > > has 2 > > > disks > > > attached. I added a new topic (heavyweight) and was surprised > > that > > > even if > > > the topic has 15 partitions, those weren't distributed evenly > on > > the > > > disks. > > > Thus I got one disk that's almost empty and the other almost > > filled > > > up. Is > > > there any way to have Kafka evenly distribute data on its > disks? > > > > > > Thank you! > > > > > > > > > This e-mail and any files transmitted with it are for the sole use > > of the > > > intended recipient(s) and may contain confidential and privileged > > > information. If you are not the intended recipient(s), please reply > > to the > > > sender and destroy all copies of the original message. Any > > unauthorized > > > review, use, disclosure, dissemination, forwarding, printing or > > copying of > > > this email, and/or any action taken in reliance on the contents of > > this > > > e-mail is strictly prohibited and may be unlawful. Where permitted > by > > > applicable law, this e-mail and other e-mail communications sent to > > and > > > from Cognizant e-mail addresses may be monitored. > > > This e-mail and any files transmitted with it are for the sole use > > of the > > > intended recipient(s) and may contain confidential and privileged > > > information. If you are not the intended recipient(s), please reply > > to the > > > sender and destroy all copies of the original message. Any > > unauthorized > > > review, use, disclosure, dissemination, forwarding, printing or > > copying of > > > this email, and/or any action taken in reliance on the contents of > > this > > > e-mail is strictly prohibited and may be unlawful. Where permitted > by > > > applicable law, this e-mail and other e-mail communications sent to > > and > > > from Cognizant e-mail addresses may be monitored. > > > > > > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. If you are not the intended recipient(s), please reply to > the > > sender and destroy all copies of the original message. Any unauthorized > > review, use, disclosure, dissemination, forwarding, printing or copying > of > > this email, and/or any action taken in reliance on the contents of this > > e-mail is strictly prohibited and may be unlawful. Where permitted by > > applicable law, this e-mail and other e-mail communications sent to and > > from Cognizant e-mail addresses may be monitored. > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. If you are not the intended recipient(s), please reply to > the > > sender and destroy all copies of the original message. Any unauthorized > > review, use, disclosure, dissemination, forwarding, printing or copying > of > > this email, and/or any action taken in reliance on the contents of this > > e-mail is strictly prohibited and may be unlawful. Where permitted by > > applicable law, this e-mail and other e-mail communications sent to and > > from Cognizant e-mail addresses may be monitored. > > >
