Just to report in case anyone finds this via a web search.
I did the following:
rmr brokers/topics/mytopic
rm admin/delete_topics/mytopic
and it worked fine. The broker reported an exception in `kafka-controller.log`
after the last rm ran but it still ran without any problems.
I restarted the broker which was the controller at the time just to be sure.
On Wed, Dec 18, 2019, at 00:33, Vincent Rischmann wrote:
> The broker id wasn't reused no, it's a new id.
>
> Unfortunately we can't afford bringing down the cluster, I'll have to
> do this with the cluster online.
>
> On Mon, Dec 16, 2019, at 17:27, Peter Bukowinski wrote:
> > If it was replaced, and a new broker was brought online with the same
> > id, wherever topic partitions had been previously assigned to it should
> > have been recreated.
> >
> > At this point, however, I would shut down the cluster, delete the
> > znodes, delete the topic directories from the brokers, then bring the
> > cluster back up.
> >
> > -- Peter
> >
> > > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann <[email protected]>
> > > wrote:
> > >
> > > It doesn't exist anymore, we replaced it after a hardware failure.
> > >
> > > Thinking about it I don't think I reassigned the partitions for broker 5
> > > to the new broker before deleting these topics, I didn't realize that it
> > > was necessary for all brokers to be online.
> > >
> > > Since broker 5 is never coming back again I'm guessing my only choice is
> > > to manually modify the znodes ?
> > >
> > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote:
> > >> If any brokers are offline, kafka can’t successfully delete a topic.
> > >> What’s the state of broker 5?
> > >>
> > >> -- Peter (from phone)
> > >>
> > >>>> On Dec 13, 2019, at 8:55 AM, Vincent Rischmann <[email protected]>
> > >>>> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> I've deleted a bunch of topics yesterday on our cluster but some are
> > >>> now stuck in "marked for deletion".
> > >>>
> > >>> * i've looked in the data directory of every broker and there's no data
> > >>> left for the topics, the directory doesn't exist anymore.
> > >>> * in zookeeper the znode `brokers/topics/mytopic` still exists
> > >>> * the znode `admin/delete_topics/mytopic` still exists
> > >>>
> > >>> I've tried the following to no avail:
> > >>>
> > >>> * restarting all brokers
> > >>> * removing the `admin/delete_topics/mytopic` node and re-running
> > >>> `kafka-topics.sh --delete --topic mytopic`
> > >>>
> > >>> In the kafka-controller.log of some brokers I see this which seems
> > >>> relevant:
> > >>>
> > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not
> > >>> sending request (type=StopReplicaRequest, controllerId=6,
> > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to
> > >>> broker 5, since it is offline.
> > >>> (kafka.controller.ControllerChannelManager)
> > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not
> > >>> sending request (type=StopReplicaRequest, controllerId=6,
> > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to
> > >>> broker 5, since it is offline.
> > >>> (kafka.controller.ControllerChannelManager)
> > >>>
> > >>> and
> > >>>
> > >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1],
> > >>> Handling deletion for topics mytopic
> > >>> (kafka.controller.TopicDeletionManager)
> > >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not
> > >>> retrying deletion of topic mytopic at this time since it is marked
> > >>> ineligible for deletion (kafka.controller.TopicDeletionManager)
> > >>>
> > >>> Since the data directory is already deleted I'm thinking of simply
> > >>> removing the znode `brokers/topics/mytopic` from zookeeper manually.
> > >>>
> > >>> Does anyone has another suggestion ? Is it safe to remove the znode
> > >>> manually ?
> > >>>
> > >>> Thanks.
> > >>
> >
>
