Great to hear, Chris! I would recommend not using zookeeper addresses in any of your kafka commands as that has been deprecated for some time. Switch to using `--bootstrap-server`.
— Peter > On Sep 22, 2022, at 1:08 AM, Chris Peart <ch...@peart.me.uk> wrote: > > Hi Peter, > > > > Thanks for getting back to me on this, i have done the following checks as > suggested: > > Kafka version: 2.8.1 > > I can connect to all 4 brokers on port 9092 from my client system > > I can see this on 3 of the 4 brokers have the following message: > > DEBUG [Controller id=X] Broker Y has been elected as the controller, so > stopping the election process. (kafka.controller.KafkaController) > > I was just going to send you the command for my topic creation and realised i > was not using the new zookeeper path in my ZK connection string, so i was > using /kafka instead of /kafkaprod MyBad :-( > > > > So all was working after all, once i removed the data/files from /data/kafka > and changing the zookeeper.connect string to use /kafkaprod. > > > > We now have all six disk working on all 4 brokers. > > > > Thanks again for your help on this and sorry for not spotting my mistake > sooner :-) > > > > Many Thanks > > Chris > > > > On 2022-09-21 22:24, Peter Bukowinski wrote: > >> Hmmm. Let’s start with some low level troubleshooting. >> >> What kafka version are you using? >> >> Can reach the ip:port of all broker listener addresses? I like to use `nc >> -vz ip port` to validate connectivity from my kafka client. >> >> Check the controller.log on all brokers. All but one of them should say >> something like `DEBUG [Controller id=X] Broker Y has been elected as the >> controller, so stopping the election process. >> (kafka.controller.KafkaController)`. This will tell you that all brokers are >> in agreement that only one of them is the controller. This will confirm >> zookeeper is configured and working correctly. >> >> Can you share the command you’re using to create your topics? >> >> When you run `kafka-broker-api-versions.sh --bootstrap-server host:port | >> grep ‘>’ | sort -n -k3`, do you get a list of all your brokers? >> >> — >> Peter >> >>> On Sep 21, 2022, at 10:26 AM, Chris Peart <ch...@peart.me.uk >>> <mailto:ch...@peart.me.uk>> wrote: >>> >>> Hi Peter, >>> Kafka brokers are staying up and show when I query zookeeper brokers/ids >>> Is the any other files that require deletion as it doesn’t make sense I >>> have 0 brokers, when trying to create a topic. >>> >>> I get no errors when listing topics, but obviously don’t see any topics as >>> I cannot create any. >>> >>> I have rebooted the Kafka brokers and restarted the all the zookeeper >>> services on our production zookeeper cluster. >>> >>> Not sure what other steps I can take to resolve this, any thoughts would be >>> appreciated. >>> >>> Many Thanks >>> Chris >>> >>> >>>> On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com >>>> <mailto:pmb...@gmail.com>> wrote: >>>> >>>> Hi Chris, >>>> >>>> Are you sure kafka is starting up (and remains up) successfully? From the >>>> error, it seems like none of the brokers are online. >>>> >>>> — >>>> Peter >>>> >>>> >>>>> On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk >>>>> <mailto:ch...@peart.me.uk>> wrote: >>>>> >>>>> >>>>> >>>>> Hi Peter, >>>>> >>>>> I have made the changes i suggested: >>>>> >>>>> Stopping Kafka >>>>> >>>>> Deleting all files and folders in /Kafka/data >>>>> >>>>> Changing the zookeeper setting to point to a different path in the >>>>> zookeeper cluster >>>>> >>>>> Start Kafka >>>>> >>>>> I see the usual files in /kafka/data/ meta.properties and the offset >>>>> files in all 6 disks. >>>>> >>>>> I see all 4 brokers in zookeeper using the new path i specified in >>>>> server.properties. >>>>> >>>>> When i try to create a topic now i receive: Replication factor: 3 larger >>>>> than available brokers: 0 >>>>> >>>>> I see no errors in server.log & controller.log. >>>>> >>>>> Any advice would be great please as i've exhausted all my options. >>>>> >>>>> Many Thanks, >>>>> >>>>> Chris >>>>> >>>>>> On 2022-09-20 21:43, Chris Peart wrote: >>>>>> >>>>>> Thanks Peter, >>>>>> I'll give this a go tomorrow and let you know how I get on. >>>>>> Many Thanks, >>>>>> Chris >>>>>>> >>>>>>> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com >>>>>>> <mailto:pmb...@gmail.com>> wrote: >>>>>> Hi Chris, >>>>>> If the configs are correct and the permissions on all the >>>>>> /data/X/kafka/data directories are correct, then kafka should use all of >>>>>> the log dirs when creating topics. Remember that kafka will not >>>>>> automatically move any existing topic data when the cluster configs >>>>>> change. I'd test by creating a topic with more partitions than storage >>>>>> locations. >>>>>> If you'd rather start fresh, you have the steps correct. An alternative >>>>>> to changing the zk path is to use zkCli to remove the paths. If you use >>>>>> a zookeeper chroot, just delete everything from that chroot down from >>>>>> zkCli, e.g. `rmr /[kafka-chroot]` >>>>>> -- >>>>>> Peter >>>>>>> >>>>>>> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk >>>>>>> <mailto:ch...@peart.me.uk>> wrote: >>>>>> Thinking about this, as this is not in production it might be easier >>>>>> just reset everything. >>>>>> Would it be something like: >>>>>> Stopping Kafka >>>>>> Deleting all files and folders in /Kafka/data >>>>>> Changing the zookeeper setting to point to a different path in the >>>>>> zookeeper cluster >>>>>> Start Kafka >>>>>> Some help on resetting Kafka would be great if ok please. >>>>>> Many Thanks >>>>>> Chris >>>>>>> >>>>>>> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk >>>>>>> <mailto:ch...@peart.me.uk>> wrote: >>>>>> Hi Peter, >>>>>> I have checked the logs on all 4 brokers and could only see >>>>>> /data/1/data/kafka being used, log.dirs config in the logs showed all >>>>>> the disks but no errors. >>>>>> I managed to get the other 5 disks working by adding the path >>>>>> /data/[2-6]/data/kafka and setting the owner as kafka and restarting >>>>>> kafka. >>>>>> So now when i create topics i see disks 2-6 being used but not disk 1. >>>>>> I have stopped all the brokers deleted all files /data/1/kafka/data on >>>>>> all brokers and started them, but i still don't see disk 1 being used >>>>>> when creating topics, it's as if there is some dirty configuration >>>>>> somewhere, maybe in zookeeper? >>>>>> Any help here would be much appreciated :) >>>>>> Many Thanks, >>>>>> Chris >>>>>>> >>>>>>> On 2022-09-16 08:22, Chris Peart wrote: >>>>>> Thanks Peter, >>>>>> I'll check the logs next week and let you know my findings. >>>>>> Many Thanks >>>>>> Chris >>>>>>> >>>>>>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com >>>>>>> <mailto:pmb...@gmail.com>> wrote: >>>>>> The next thing I'd check is the broker logs. The parameters read from >>>>>> the config should appear in the logs when kafka starts up. Search the >>>>>> logs for 'log.dirs' and ensure the correct configs are loaded. >>>>>> -- >>>>>> Peter >>>>>>> >>>>>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk >>>>>>> <mailto:ch...@peart.me.uk>> wrote: >>>>>> Hi Peter, >>>>>> Thanks for your response, we have the following configuration: >>>>>> Partition count=4 >>>>>> Replication factor=3 >>>>>> All four brokers have topics-partitions in /data/1/kafka/data and are >>>>>> receiving data. >>>>>> Each server has 6 x 2TB disks for kaka data. >>>>>> Many Thanks, >>>>>> Chris >>>>>>> >>>>>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com >>>>>>> <mailto:pmb...@gmail.com>> wrote: >>>>>> Hi Chris, >>>>>> Can you share the partition count and replication factor of your >>>>>> partitions? Also, do all four brokers contain topic-partition >>>>>> directories in /data/1/kafka/data or just a single broker? Depending on >>>>>> your topic config, it may be entirely normal that his has happened. >>>>>> -- >>>>>> Peter Bukowinski >>>>>>> >>>>>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk >>>>>>> <mailto:ch...@peart.me.uk>> wrote: >>>>>> Hi All, >>>>>> I have a 4 node kafka cluster running version 2.8.1, we have started >>>>>> pushing data to the cluster but can only see one disk being used. >>>>>> We had 6 disk configured as non-raid and 1 partition per disk, we have >>>>>> the following in fstab: >>>>>> /dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> /dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> /dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> /dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> /dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> /dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> We configured server.properties to be: >>>>>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data >>>>>> i can see all our topics in /data/1/kafka/data but don't see anything in >>>>>> /data/2-5 >>>>>> Any help would be appreciated as this is going to production next week? >>>>>> Many Thanks, >>>>>> Chris
