If it was replaced, and a new broker was brought online with the same id, wherever topic partitions had been previously assigned to it should have been recreated.
At this point, however, I would shut down the cluster, delete the znodes, delete the topic directories from the brokers, then bring the cluster back up. -- Peter > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann <vinc...@rischmann.fr> wrote: > > It doesn't exist anymore, we replaced it after a hardware failure. > > Thinking about it I don't think I reassigned the partitions for broker 5 to > the new broker before deleting these topics, I didn't realize that it was > necessary for all brokers to be online. > > Since broker 5 is never coming back again I'm guessing my only choice is to > manually modify the znodes ? > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: >> If any brokers are offline, kafka can’t successfully delete a topic. >> What’s the state of broker 5? >> >> -- Peter (from phone) >> >>>> On Dec 13, 2019, at 8:55 AM, Vincent Rischmann <vinc...@rischmann.fr> >>>> wrote: >>> >>> Hi, >>> >>> I've deleted a bunch of topics yesterday on our cluster but some are now >>> stuck in "marked for deletion". >>> >>> * i've looked in the data directory of every broker and there's no data >>> left for the topics, the directory doesn't exist anymore. >>> * in zookeeper the znode `brokers/topics/mytopic` still exists >>> * the znode `admin/delete_topics/mytopic` still exists >>> >>> I've tried the following to no avail: >>> >>> * restarting all brokers >>> * removing the `admin/delete_topics/mytopic` node and re-running >>> `kafka-topics.sh --delete --topic mytopic` >>> >>> In the kafka-controller.log of some brokers I see this which seems relevant: >>> >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not >>> sending request (type=StopReplicaRequest, controllerId=6, >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not >>> sending request (type=StopReplicaRequest, controllerId=6, >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) >>> >>> and >>> >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling >>> deletion for topics mytopic (kafka.controller.TopicDeletionManager) >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not >>> retrying deletion of topic mytopic at this time since it is marked >>> ineligible for deletion (kafka.controller.TopicDeletionManager) >>> >>> Since the data directory is already deleted I'm thinking of simply removing >>> the znode `brokers/topics/mytopic` from zookeeper manually. >>> >>> Does anyone has another suggestion ? Is it safe to remove the znode >>> manually ? >>> >>> Thanks. >>
