Hi Peter,
I have checked the logs on all 4 brokers and could only see
/data/1/data/kafka being used, log.dirs config in the logs showed all
the disks but no errors.
I managed to get the other 5 disks working by adding the path
/data/[2-6]/data/kafka and setting the owner as kafka and restarting
kafka.
So now when i create topics i see disks 2-6 being used but not disk 1.
I have stopped all the brokers deleted all files /data/1/kafka/data on
all brokers and started them, but i still don't see disk 1 being used
when creating topics, it's as if there is some dirty configuration
somewhere, maybe in zookeeper?
Any help here would be much appreciated :)
Many Thanks,
Chris
On 2022-09-16 08:22, Chris Peart wrote:
Thanks Peter,
I'll check the logs next week and let you know my findings.
Many Thanks
Chris
On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com> wrote:
The next thing I'd check is the broker logs. The parameters read from
the config should appear in the logs when kafka starts up. Search the
logs for 'log.dirs' and ensure the correct configs are loaded.
--
Peter
On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk> wrote:
Hi Peter,
Thanks for your response, we have the following configuration:
Partition count=4
Replication factor=3
All four brokers have topics-partitions in /data/1/kafka/data and are
receiving data.
Each server has 6 x 2TB disks for kaka data.
Many Thanks,
Chris
On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com> wrote:
Hi Chris,
Can you share the partition count and replication factor of your
partitions? Also, do all four brokers contain topic-partition
directories in /data/1/kafka/data or just a single broker? Depending on
your topic config, it may be entirely normal that his has happened.
--
Peter Bukowinski
On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk> wrote:
Hi All,
I have a 4 node kafka cluster running version 2.8.1, we have started
pushing data to the cluster but can only see one disk being used.
We had 6 disk configured as non-raid and 1 partition per disk, we have
the following in fstab:
/dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1
2
/dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail
1 2
We configured server.properties to be:
log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data
i can see all our topics in /data/1/kafka/data but don't see anything
in /data/2-5
Any help would be appreciated as this is going to production next week?
Many Thanks,
Chris
