Re: Topics marked for deletion stuck as ineligible for deletion
Just to report in case anyone finds this via a web search. I did the following: rmr brokers/topics/mytopic rm admin/delete_topics/mytopic and it worked fine. The broker reported an exception in `kafka-controller.log` after the last rm ran but it still ran without any problems. I restarted the broker which was the controller at the time just to be sure. On Wed, Dec 18, 2019, at 00:33, Vincent Rischmann wrote: > The broker id wasn't reused no, it's a new id. > > Unfortunately we can't afford bringing down the cluster, I'll have to > do this with the cluster online. > > On Mon, Dec 16, 2019, at 17:27, Peter Bukowinski wrote: > > If it was replaced, and a new broker was brought online with the same > > id, wherever topic partitions had been previously assigned to it should > > have been recreated. > > > > At this point, however, I would shut down the cluster, delete the > > znodes, delete the topic directories from the brokers, then bring the > > cluster back up. > > > > -- Peter > > > > > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann > > > wrote: > > > > > > It doesn't exist anymore, we replaced it after a hardware failure. > > > > > > Thinking about it I don't think I reassigned the partitions for broker 5 > > > to the new broker before deleting these topics, I didn't realize that it > > > was necessary for all brokers to be online. > > > > > > Since broker 5 is never coming back again I'm guessing my only choice is > > > to manually modify the znodes ? > > > > > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: > > >> If any brokers are offline, kafka can’t successfully delete a topic. > > >> What’s the state of broker 5? > > >> > > >> -- Peter (from phone) > > >> > > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann > > wrote: > > >>> > > >>> Hi, > > >>> > > >>> I've deleted a bunch of topics yesterday on our cluster but some are > > >>> now stuck in "marked for deletion". > > >>> > > >>> * i've looked in the data directory of every broker and there's no data > > >>> left for the topics, the directory doesn't exist anymore. > > >>> * in zookeeper the znode `brokers/topics/mytopic` still exists > > >>> * the znode `admin/delete_topics/mytopic` still exists > > >>> > > >>> I've tried the following to no avail: > > >>> > > >>> * restarting all brokers > > >>> * removing the `admin/delete_topics/mytopic` node and re-running > > >>> `kafka-topics.sh --delete --topic mytopic` > > >>> > > >>> In the kafka-controller.log of some brokers I see this which seems > > >>> relevant: > > >>> > > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > >>> sending request (type=StopReplicaRequest, controllerId=6, > > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to > > >>> broker 5, since it is offline. > > >>> (kafka.controller.ControllerChannelManager) > > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > >>> sending request (type=StopReplicaRequest, controllerId=6, > > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to > > >>> broker 5, since it is offline. > > >>> (kafka.controller.ControllerChannelManager) > > >>> > > >>> and > > >>> > > >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], > > >>> Handling deletion for topics mytopic > > >>> (kafka.controller.TopicDeletionManager) > > >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > > >>> retrying deletion of topic mytopic at this time since it is marked > > >>> ineligible for deletion (kafka.controller.TopicDeletionManager) > > >>> > > >>> Since the data directory is already deleted I'm thinking of simply > > >>> removing the znode `brokers/topics/mytopic` from zookeeper manually. > > >>> > > >>> Does anyone has another suggestion ? Is it safe to remove the znode > > >>> manually ? > > >>> > > >>> Thanks. > > >> > > >
Re: Topics marked for deletion stuck as ineligible for deletion
The broker id wasn't reused no, it's a new id. Unfortunately we can't afford bringing down the cluster, I'll have to do this with the cluster online. On Mon, Dec 16, 2019, at 17:27, Peter Bukowinski wrote: > If it was replaced, and a new broker was brought online with the same > id, wherever topic partitions had been previously assigned to it should > have been recreated. > > At this point, however, I would shut down the cluster, delete the > znodes, delete the topic directories from the brokers, then bring the > cluster back up. > > -- Peter > > > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann wrote: > > > > It doesn't exist anymore, we replaced it after a hardware failure. > > > > Thinking about it I don't think I reassigned the partitions for broker 5 to > > the new broker before deleting these topics, I didn't realize that it was > > necessary for all brokers to be online. > > > > Since broker 5 is never coming back again I'm guessing my only choice is to > > manually modify the znodes ? > > > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: > >> If any brokers are offline, kafka can’t successfully delete a topic. > >> What’s the state of broker 5? > >> > >> -- Peter (from phone) > >> > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann > wrote: > >>> > >>> Hi, > >>> > >>> I've deleted a bunch of topics yesterday on our cluster but some are now > >>> stuck in "marked for deletion". > >>> > >>> * i've looked in the data directory of every broker and there's no data > >>> left for the topics, the directory doesn't exist anymore. > >>> * in zookeeper the znode `brokers/topics/mytopic` still exists > >>> * the znode `admin/delete_topics/mytopic` still exists > >>> > >>> I've tried the following to no avail: > >>> > >>> * restarting all brokers > >>> * removing the `admin/delete_topics/mytopic` node and re-running > >>> `kafka-topics.sh --delete --topic mytopic` > >>> > >>> In the kafka-controller.log of some brokers I see this which seems > >>> relevant: > >>> > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > >>> sending request (type=StopReplicaRequest, controllerId=6, > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to > >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > >>> sending request (type=StopReplicaRequest, controllerId=6, > >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to > >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >>> > >>> and > >>> > >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], > >>> Handling deletion for topics mytopic > >>> (kafka.controller.TopicDeletionManager) > >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > >>> retrying deletion of topic mytopic at this time since it is marked > >>> ineligible for deletion (kafka.controller.TopicDeletionManager) > >>> > >>> Since the data directory is already deleted I'm thinking of simply > >>> removing the znode `brokers/topics/mytopic` from zookeeper manually. > >>> > >>> Does anyone has another suggestion ? Is it safe to remove the znode > >>> manually ? > >>> > >>> Thanks. > >> >
Re: Topics marked for deletion stuck as ineligible for deletion
If it was replaced, and a new broker was brought online with the same id, wherever topic partitions had been previously assigned to it should have been recreated. At this point, however, I would shut down the cluster, delete the znodes, delete the topic directories from the brokers, then bring the cluster back up. -- Peter > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann wrote: > > It doesn't exist anymore, we replaced it after a hardware failure. > > Thinking about it I don't think I reassigned the partitions for broker 5 to > the new broker before deleting these topics, I didn't realize that it was > necessary for all brokers to be online. > > Since broker 5 is never coming back again I'm guessing my only choice is to > manually modify the znodes ? > >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: >> If any brokers are offline, kafka can’t successfully delete a topic. >> What’s the state of broker 5? >> >> -- Peter (from phone) >> On Dec 13, 2019, at 8:55 AM, Vincent Rischmann wrote: >>> >>> Hi, >>> >>> I've deleted a bunch of topics yesterday on our cluster but some are now >>> stuck in "marked for deletion". >>> >>> * i've looked in the data directory of every broker and there's no data >>> left for the topics, the directory doesn't exist anymore. >>> * in zookeeper the znode `brokers/topics/mytopic` still exists >>> * the znode `admin/delete_topics/mytopic` still exists >>> >>> I've tried the following to no avail: >>> >>> * restarting all brokers >>> * removing the `admin/delete_topics/mytopic` node and re-running >>> `kafka-topics.sh --delete --topic mytopic` >>> >>> In the kafka-controller.log of some brokers I see this which seems relevant: >>> >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not >>> sending request (type=StopReplicaRequest, controllerId=6, >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) >>> [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not >>> sending request (type=StopReplicaRequest, controllerId=6, >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager) >>> >>> and >>> >>> 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling >>> deletion for topics mytopic (kafka.controller.TopicDeletionManager) >>> 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not >>> retrying deletion of topic mytopic at this time since it is marked >>> ineligible for deletion (kafka.controller.TopicDeletionManager) >>> >>> Since the data directory is already deleted I'm thinking of simply removing >>> the znode `brokers/topics/mytopic` from zookeeper manually. >>> >>> Does anyone has another suggestion ? Is it safe to remove the znode >>> manually ? >>> >>> Thanks. >>
Re: Topics marked for deletion stuck as ineligible for deletion
It doesn't exist anymore, we replaced it after a hardware failure. Thinking about it I don't think I reassigned the partitions for broker 5 to the new broker before deleting these topics, I didn't realize that it was necessary for all brokers to be online. Since broker 5 is never coming back again I'm guessing my only choice is to manually modify the znodes ? On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote: > If any brokers are offline, kafka can’t successfully delete a topic. > What’s the state of broker 5? > > -- Peter (from phone) > > > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann wrote: > > > > Hi, > > > > I've deleted a bunch of topics yesterday on our cluster but some are now > > stuck in "marked for deletion". > > > > * i've looked in the data directory of every broker and there's no data > > left for the topics, the directory doesn't exist anymore. > > * in zookeeper the znode `brokers/topics/mytopic` still exists > > * the znode `admin/delete_topics/mytopic` still exists > > > > I've tried the following to no avail: > > > > * restarting all brokers > > * removing the `admin/delete_topics/mytopic` node and re-running > > `kafka-topics.sh --delete --topic mytopic` > > > > In the kafka-controller.log of some brokers I see this which seems relevant: > > > >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > sending request (type=StopReplicaRequest, controllerId=6, > > controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to > > broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > > sending request (type=StopReplicaRequest, controllerId=6, > > controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to > > broker 5, since it is offline. (kafka.controller.ControllerChannelManager) > > > > and > > > >12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], > > Handling deletion for topics mytopic (kafka.controller.TopicDeletionManager) > >12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > > retrying deletion of topic mytopic at this time since it is marked > > ineligible for deletion (kafka.controller.TopicDeletionManager) > > > > Since the data directory is already deleted I'm thinking of simply removing > > the znode `brokers/topics/mytopic` from zookeeper manually. > > > > Does anyone has another suggestion ? Is it safe to remove the znode > > manually ? > > > > Thanks. >
Re: Topics marked for deletion stuck as ineligible for deletion
If any brokers are offline, kafka can’t successfully delete a topic. What’s the state of broker 5? -- Peter (from phone) > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann wrote: > > Hi, > > I've deleted a bunch of topics yesterday on our cluster but some are now > stuck in "marked for deletion". > > * i've looked in the data directory of every broker and there's no data left > for the topics, the directory doesn't exist anymore. > * in zookeeper the znode `brokers/topics/mytopic` still exists > * the znode `admin/delete_topics/mytopic` still exists > > I've tried the following to no avail: > > * restarting all brokers > * removing the `admin/delete_topics/mytopic` node and re-running > `kafka-topics.sh --delete --topic mytopic` > > In the kafka-controller.log of some brokers I see this which seems relevant: > >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, > deletePartitions=false, partitions=mytopic-17) to broker 5, since it is > offline. (kafka.controller.ControllerChannelManager) >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not > sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, > deletePartitions=false, partitions=mytopic-24) to broker 5, since it is > offline. (kafka.controller.ControllerChannelManager) > > and > >12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling > deletion for topics mytopic (kafka.controller.TopicDeletionManager) >12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not > retrying deletion of topic mytopic at this time since it is marked ineligible > for deletion (kafka.controller.TopicDeletionManager) > > Since the data directory is already deleted I'm thinking of simply removing > the znode `brokers/topics/mytopic` from zookeeper manually. > > Does anyone has another suggestion ? Is it safe to remove the znode manually ? > > Thanks.
Topics marked for deletion stuck as ineligible for deletion
Hi, I've deleted a bunch of topics yesterday on our cluster but some are now stuck in "marked for deletion". * i've looked in the data directory of every broker and there's no data left for the topics, the directory doesn't exist anymore. * in zookeeper the znode `brokers/topics/mytopic` still exists * the znode `admin/delete_topics/mytopic` still exists I've tried the following to no avail: * restarting all brokers * removing the `admin/delete_topics/mytopic` node and re-running `kafka-topics.sh --delete --topic mytopic` In the kafka-controller.log of some brokers I see this which seems relevant: [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to broker 5, since it is offline. (kafka.controller.ControllerChannelManager) [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to broker 5, since it is offline. (kafka.controller.ControllerChannelManager) and 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling deletion for topics mytopic (kafka.controller.TopicDeletionManager) 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not retrying deletion of topic mytopic at this time since it is marked ineligible for deletion (kafka.controller.TopicDeletionManager) Since the data directory is already deleted I'm thinking of simply removing the znode `brokers/topics/mytopic` from zookeeper manually. Does anyone has another suggestion ? Is it safe to remove the znode manually ? Thanks.
