Thanks for the report, Karolis.
I have a potential theory for how this happened and I'm wondering if it's
possibly valid:
I have 9 partitions on a machine with 3 disks and they get assigned exactly
as you'd expect:
d1: t{1,2,3}
d2: t{4,5,6}
d3: t{7,8,9}
Then, a disk fails or something somewhere else and kafka decides to assign
t10 to d1 on this machine.
d1: t{1,2,3,10}
d2: t{4,5,6}
d3: t{7,8,9}
Then something happens where kafka wants to move a partition off of d2:
d1: t{1,2,3,10}
d2: t{4,5}
d3: t{7,8,9}
Is this scenario something that can happen?
In our actual deployment we have 5 servers with 3 disks each, 1 topic, 15
partitions, and a replication factor of 3.
On Tue, Nov 29, 2016 at 4:04 PM, Karolis Pocius <karolis.poc...@adform.com>
wrote:
> It's difficult enough to balance kafka brokers with a single log
> directory, not to mention attempting to juggle multiple ones. While JBOD is
> great in terms of capacity, it's a pain in terms of management. After 6
> months of constant manual reassignments I ended up going with RAID1+0 which
> is what LinkedIn uses as well as Confluent recommends.
>
> Hats off to you if you manage to find a solution to this, just wanted to
> share my painful experience.
>
>
>
> On 2016.11.29 21:35, Tim Visher wrote:
>
>> Hello,
>>
>> My kafka deploy has 5 servers with 3 log disks each. Over the weekend I
>> noticed that on 2 of the 5 servers the partitions appear to be imbalanced
>> amongst the log.dirs.
>>
>> ```
>> kafka3
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 3
>> /var/lib/kafka/disk3
>> 3
>> kafka5
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 4
>> /var/lib/kafka/disk3
>> 2
>> kafka1
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 3
>> /var/lib/kafka/disk3
>> 3
>> kafka4
>> /var/lib/kafka/disk1
>> 4
>> /var/lib/kafka/disk2
>> 2
>> /var/lib/kafka/disk3
>> 3
>> kafka2
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 3
>> /var/lib/kafka/disk3
>> 3
>> ```
>>
>> You can see that 5 and 4 are both unbalanced.
>>
>> Is there a reason for that? The partitions themselves are pretty much
>> perfectly balanced, but the directory chosen for them is not.
>>
>> Is this an anti-pattern to be using multiple log.dirs per server?
>>
>> Thanks in advance!
>>
>> --
>>
>> In Christ,
>>
>> Timmy V.
>>
>> http://blog.twonegatives.com/
>> http://five.sentenc.es/ -- Spend less time on mail
>>
>>
>
> Best Regards
>
> Karolis Pocius
> IT System Engineer
>
> Email: karolis.poc...@adform.com
> Mobile: +370 620 22108
> Sporto g. 18, LT-09238 Vilnius, Lithuania
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
