Hi, I've deleted a bunch of topics yesterday on our cluster but some are now stuck in "marked for deletion".
* i've looked in the data directory of every broker and there's no data left for the topics, the directory doesn't exist anymore. * in zookeeper the znode `brokers/topics/mytopic` still exists * the znode `admin/delete_topics/mytopic` still exists I've tried the following to no avail: * restarting all brokers * removing the `admin/delete_topics/mytopic` node and re-running `kafka-topics.sh --delete --topic mytopic` In the kafka-controller.log of some brokers I see this which seems relevant: [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to broker 5, since it is offline. (kafka.controller.ControllerChannelManager) [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not sending request (type=StopReplicaRequest, controllerId=6, controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to broker 5, since it is offline. (kafka.controller.ControllerChannelManager) and 12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling deletion for topics mytopic (kafka.controller.TopicDeletionManager) 12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not retrying deletion of topic mytopic at this time since it is marked ineligible for deletion (kafka.controller.TopicDeletionManager) Since the data directory is already deleted I'm thinking of simply removing the znode `brokers/topics/mytopic` from zookeeper manually. Does anyone has another suggestion ? Is it safe to remove the znode manually ? Thanks.