Re: Topics marked for deletion stuck as ineligible for deletion
Just to report in case anyone finds this via a web search. I did the following: rmr brokers/topics/mytopic rm admin/delete_topics/mytopic and it worked fine. The broker reported an exception in `kafka-controller.log` after the last rm ran but it still ran without any problems. I restarted the broker which was the controller at the time just to be sure. On Wed, Dec 18, 2019, at 00:33, Vincent Rischmann wrote: > The broker id wasn't reused no, it's a new id. > > Unfortunately we can't afford bringing down the cluster, I'll have to > do this with the cluster online. > > On Mon, Dec 16, 2019, at 17:27, Peter Bukowinski wrote: > > If it was replaced, and a new broker was brought online with the same > > id, wherever topic partitions had been previously assigned to it should > > have been recreated. > > > > At this point, however, I would shut down the cluster, delete the > > znodes, delete the topic directories from the brokers, then bring the > > cluster back up. > > > > -- Peter > > > > > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann > > > wrote: > > > > > > It doesn't exist anymore, we replaced it after a hardware failure. > > > > > > Thinking about it I don't think I reassigned the partitions for broker 5 > > > to the new broker before deleting these topics, I didn't realize that it > > > was necessary for all brokers to be online. > > > > > > Since broker 5 is never coming back again I'm guessing my only choice is > > > to manually modify the znodes ? > > > > > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: > > >> If any brokers are offline, kafka can’t successfully delete a topic. > > >> What’s the state of broker 5? > > >> > > >> -- Peter (from phone) > > >> > > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann > > wrote: > > >>> > > >>> Hi, > > >>> > > >>> I've deleted a bunch of topics yesterday on our cluster but some are > > >>> now stuck in "marked for deletion". > > >>> > > >>> * i've looked in the data directory of every broker and there's no data > > >>> left for the topics, the directory doesn't exist anymore. > > >>> * in zookeeper the znode `brokers/topics/mytopic` still exists > > >>> * the znode `admin/delete_topics/mytopic` still exists > > >>> > > >>> I've tried the following to no avail: > > >>> > > >>> * restarting all brokers > > >>> * removing the `admin/delete_topics/mytopic` node and re-running > > >>> `kafka-topics.sh --delete --topic mytopic` > > >>> > > >>> In the kafka-controller.log of some brokers I see this which seems > > >>> relevant: > > >>> > > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > >>> sending request (type=StopReplicaRequest, controllerId=6, > > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to > > >>> broker 5, since it is offline. > > >>> (kafka.controller.ControllerChannelManager) > > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > >>> sending request (type=StopReplicaRequest, controllerId=6, > > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to > > >>> broker 5, since it is offline. > > >>> (kafka.controller.ControllerChannelManager) > > >>> > > >>> and > > >>> > > >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], > > >>> Handling deletion for topics mytopic > > >>> (kafka.controller.TopicDeletionManager) > > >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > > >>> retrying deletion of topic mytopic at this time since it is marked > > >>> ineligible for deletion (kafka.controller.TopicDeletionManager) > > >>> > > >>> Since the data directory is already deleted I'm thinking of simply > > >>> removing the znode `brokers/topics/mytopic` from zookeeper manually. > > >>> > > >>> Does anyone has another suggestion ? Is it safe to remove the znode > > >>> manually ? > > >>> > > >>> Thanks. > > >> > > >
Re: Topics marked for deletion stuck as ineligible for deletion
The broker id wasn't reused no, it's a new id. Unfortunately we can't afford bringing down the cluster, I'll have to do this with the cluster online. On Mon, Dec 16, 2019, at 17:27, Peter Bukowinski wrote: > If it was replaced, and a new broker was brought online with the same > id, wherever topic partitions had been previously assigned to it should > have been recreated. > > At this point, however, I would shut down the cluster, delete the > znodes, delete the topic directories from the brokers, then bring the > cluster back up. > > -- Peter > > > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann wrote: > > > > It doesn't exist anymore, we replaced it after a hardware failure. > > > > Thinking about it I don't think I reassigned the partitions for broker 5 to > > the new broker before deleting these topics, I didn't realize that it was > > necessary for all brokers to be online. > > > > Since broker 5 is never coming back again I'm guessing my only choice is to > > manually modify the znodes ? > > > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: > >> If any brokers are offline, kafka can’t successfully delete a topic. > >> What’s the state of broker 5? > >> > >> -- Peter (from phone) > >> > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann > wrote: > >>> > >>> Hi, > >>> > >>> I've deleted a bunch of topics yesterday on our cluster but some are now > >>> stuck in "marked for deletion". > >>> > >>> * i've looked in the data directory of every broker and there's no data > >>> left for the topics, the directory doesn't exist anymore. > >>> * in zookeeper the znode `brokers/topics/mytopic` still exists > >>> * the znode `admin/delete_topics/mytopic` still exists > >>> > >>> I've tried the following to no avail: > >>> > >>> * restarting all brokers > >>> * removing the `admin/delete_topics/mytopic` node and re-running > >>> `kafka-topics.sh --delete --topic mytopic` > >>> > >>> In the kafka-controller.log of some brokers I see this which seems > >>> relevant: > >>> > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > >>> sending request (type=StopReplicaRequest, controllerId=6, > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to > >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > >>> sending request (type=StopReplicaRequest, controllerId=6, > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to > >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >>> > >>> and > >>> > >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], > >>> Handling deletion for topics mytopic > >>> (kafka.controller.TopicDeletionManager) > >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > >>> retrying deletion of topic mytopic at this time since it is marked > >>> ineligible for deletion (kafka.controller.TopicDeletionManager) > >>> > >>> Since the data directory is already deleted I'm thinking of simply > >>> removing the znode `brokers/topics/mytopic` from zookeeper manually. > >>> > >>> Does anyone has another suggestion ? Is it safe to remove the znode > >>> manually ? > >>> > >>> Thanks. > >> >
Re: Topics marked for deletion stuck as ineligible for deletion
If it was replaced, and a new broker was brought online with the same id, wherever topic partitions had been previously assigned to it should have been recreated. At this point, however, I would shut down the cluster, delete the znodes, delete the topic directories from the brokers, then bring the cluster back up. -- Peter > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann wrote: > > It doesn't exist anymore, we replaced it after a hardware failure. > > Thinking about it I don't think I reassigned the partitions for broker 5 to > the new broker before deleting these topics, I didn't realize that it was > necessary for all brokers to be online. > > Since broker 5 is never coming back again I'm guessing my only choice is to > manually modify the znodes ? > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: >> If any brokers are offline, kafka can’t successfully delete a topic. >> What’s the state of broker 5? >> >> -- Peter (from phone) >> On Dec 13, 2019, at 8:55 AM, Vincent Rischmann wrote: >>> >>> Hi, >>> >>> I've deleted a bunch of topics yesterday on our cluster but some are now >>> stuck in "marked for deletion". >>> >>> * i've looked in the data directory of every broker and there's no data >>> left for the topics, the directory doesn't exist anymore. >>> * in zookeeper the znode `brokers/topics/mytopic` still exists >>> * the znode `admin/delete_topics/mytopic` still exists >>> >>> I've tried the following to no avail: >>> >>> * restarting all brokers >>> * removing the `admin/delete_topics/mytopic` node and re-running >>> `kafka-topics.sh --delete --topic mytopic` >>> >>> In the kafka-controller.log of some brokers I see this which seems relevant: >>> >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not >>> sending request (type=StopReplicaRequest, controllerId=6, >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not >>> sending request (type=StopReplicaRequest, controllerId=6, >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) >>> >>> and >>> >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling >>> deletion for topics mytopic (kafka.controller.TopicDeletionManager) >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not >>> retrying deletion of topic mytopic at this time since it is marked >>> ineligible for deletion (kafka.controller.TopicDeletionManager) >>> >>> Since the data directory is already deleted I'm thinking of simply removing >>> the znode `brokers/topics/mytopic` from zookeeper manually. >>> >>> Does anyone has another suggestion ? Is it safe to remove the znode >>> manually ? >>> >>> Thanks. >>
Re: Topics marked for deletion stuck as ineligible for deletion
It doesn't exist anymore, we replaced it after a hardware failure. Thinking about it I don't think I reassigned the partitions for broker 5 to the new broker before deleting these topics, I didn't realize that it was necessary for all brokers to be online. Since broker 5 is never coming back again I'm guessing my only choice is to manually modify the znodes ? On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: > If any brokers are offline, kafka can’t successfully delete a topic. > What’s the state of broker 5? > > -- Peter (from phone) > > > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann wrote: > > > > Hi, > > > > I've deleted a bunch of topics yesterday on our cluster but some are now > > stuck in "marked for deletion". > > > > * i've looked in the data directory of every broker and there's no data > > left for the topics, the directory doesn't exist anymore. > > * in zookeeper the znode `brokers/topics/mytopic` still exists > > * the znode `admin/delete_topics/mytopic` still exists > > > > I've tried the following to no avail: > > > > * restarting all brokers > > * removing the `admin/delete_topics/mytopic` node and re-running > > `kafka-topics.sh --delete --topic mytopic` > > > > In the kafka-controller.log of some brokers I see this which seems relevant: > > > >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > sending request (type=StopReplicaRequest, controllerId=6, > > controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to > > broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > sending request (type=StopReplicaRequest, controllerId=6, > > controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to > > broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > > > > and > > > >12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], > > Handling deletion for topics mytopic (kafka.controller.TopicDeletionManager) > >12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > > retrying deletion of topic mytopic at this time since it is marked > > ineligible for deletion (kafka.controller.TopicDeletionManager) > > > > Since the data directory is already deleted I'm thinking of simply removing > > the znode `brokers/topics/mytopic` from zookeeper manually. > > > > Does anyone has another suggestion ? Is it safe to remove the znode > > manually ? > > > > Thanks. >
Re: Topics marked for deletion stuck as ineligible for deletion
If any brokers are offline, kafka can’t successfully delete a topic. What’s the state of broker 5? -- Peter (from phone) > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann wrote: > > Hi, > > I've deleted a bunch of topics yesterday on our cluster but some are now > stuck in "marked for deletion". > > * i've looked in the data directory of every broker and there's no data left > for the topics, the directory doesn't exist anymore. > * in zookeeper the znode `brokers/topics/mytopic` still exists > * the znode `admin/delete_topics/mytopic` still exists > > I've tried the following to no avail: > > * restarting all brokers > * removing the `admin/delete_topics/mytopic` node and re-running > `kafka-topics.sh --delete --topic mytopic` > > In the kafka-controller.log of some brokers I see this which seems relevant: > >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, > deletePartitions=false, partitions=mytopic-17) to broker 5, since it is > offline. (kafka.controller.ControllerChannelManager) >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, > deletePartitions=false, partitions=mytopic-24) to broker 5, since it is > offline. (kafka.controller.ControllerChannelManager) > > and > >12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling > deletion for topics mytopic (kafka.controller.TopicDeletionManager) >12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > retrying deletion of topic mytopic at this time since it is marked ineligible > for deletion (kafka.controller.TopicDeletionManager) > > Since the data directory is already deleted I'm thinking of simply removing > the znode `brokers/topics/mytopic` from zookeeper manually. > > Does anyone has another suggestion ? Is it safe to remove the znode manually ? > > Thanks.
