The broker id wasn't reused no, it's a new id. Unfortunately we can't afford bringing down the cluster, I'll have to do this with the cluster online.
On Mon, Dec 16, 2019, at 17:27, Peter Bukowinski wrote: > If it was replaced, and a new broker was brought online with the same > id, wherever topic partitions had been previously assigned to it should > have been recreated. > > At this point, however, I would shut down the cluster, delete the > znodes, delete the topic directories from the brokers, then bring the > cluster back up. > > -- Peter > > > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann <vinc...@rischmann.fr> wrote: > > > > It doesn't exist anymore, we replaced it after a hardware failure. > > > > Thinking about it I don't think I reassigned the partitions for broker 5 to > > the new broker before deleting these topics, I didn't realize that it was > > necessary for all brokers to be online. > > > > Since broker 5 is never coming back again I'm guessing my only choice is to > > manually modify the znodes ? > > > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: > >> If any brokers are offline, kafka can’t successfully delete a topic. > >> What’s the state of broker 5? > >> > >> -- Peter (from phone) > >> > >>>> On Dec 13, 2019, at 8:55 AM, Vincent Rischmann <vinc...@rischmann.fr> > >>>> wrote: > >>> > >>> Hi, > >>> > >>> I've deleted a bunch of topics yesterday on our cluster but some are now > >>> stuck in "marked for deletion". > >>> > >>> * i've looked in the data directory of every broker and there's no data > >>> left for the topics, the directory doesn't exist anymore. > >>> * in zookeeper the znode `brokers/topics/mytopic` still exists > >>> * the znode `admin/delete_topics/mytopic` still exists > >>> > >>> I've tried the following to no avail: > >>> > >>> * restarting all brokers > >>> * removing the `admin/delete_topics/mytopic` node and re-running > >>> `kafka-topics.sh --delete --topic mytopic` > >>> > >>> In the kafka-controller.log of some brokers I see this which seems > >>> relevant: > >>> > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > >>> sending request (type=StopReplicaRequest, controllerId=6, > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to > >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > >>> sending request (type=StopReplicaRequest, controllerId=6, > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to > >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >>> > >>> and > >>> > >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], > >>> Handling deletion for topics mytopic > >>> (kafka.controller.TopicDeletionManager) > >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > >>> retrying deletion of topic mytopic at this time since it is marked > >>> ineligible for deletion (kafka.controller.TopicDeletionManager) > >>> > >>> Since the data directory is already deleted I'm thinking of simply > >>> removing the znode `brokers/topics/mytopic` from zookeeper manually. > >>> > >>> Does anyone has another suggestion ? Is it safe to remove the znode > >>> manually ? > >>> > >>> Thanks. > >> >
