[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505688#comment-14505688 ] Sriharsha Chintalapani commented on KAFKA-2046: --- [~onurkaraman] I am curious to see if this issue was reproducible with the 0.8.2.1 or trunk with controller.message.queue.size defaults or does it require further look into delete topic functionality. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502065#comment-14502065 ] Onur Karaman commented on KAFKA-2046: - I forgot to give an update on Clark's case. I looked at the zookeeper transaction logs and confirmed that during his attempt, the /admin/delete_topics/mytopic znode was never created and that the znodes under /brokers/topics/mytopic were deleted. He was using a separate tools repository while I was using trunk from open-source. This tools repository was using a jar from about a year ago, which means kafka-topics.sh was running an older delete topic protocol against our brokers which were expecting the latest protocol. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494497#comment-14494497 ] Onur Karaman commented on KAFKA-2046: - I made a jira for this: https://issues.apache.org/jira/browse/KAFKA-2122 > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386942#comment-14386942 ] Jay Kreps commented on KAFKA-2046: -- Hmm, so we have a configuration which, if you set it to anything other than the default, causes a deadlock? Is there a reason we left this configurable once we realized infinite was the only safe value? Could we remove it now? :-) > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386228#comment-14386228 ] Onur Karaman commented on KAFKA-2046: - Yeah the issue still happens on [~clarkhaskins]' cluster with the default controller.message.queue.size but I haven't been able to reproduce it on my test cluster. We'll try again on his and get the logs and heapdumps to get a better idea of what happened. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386021#comment-14386021 ] Neha Narkhede commented on KAFKA-2046: -- [~onurkaraman] Could [~clarkhaskins] reproduce the delete topic issue once you changed the controller.message.queue.size to its default size (infinite)? > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386015#comment-14386015 ] Sriharsha Chintalapani commented on KAFKA-2046: --- [~nehanarkhede] [~onurkaraman] In 0.8.2 we did make controller.message.queue.size to be infinite (Int.maxValue) by default because of these issues. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386014#comment-14386014 ] Onur Karaman commented on KAFKA-2046: - Correct. Basically Clark and I had two separate occurrences of delete topic not working: 1. We're still trying to reproduce Clark's occurrence. 2. I ran into a deadlock that came from controller.message.queue.size being overridden to a small value. It was addressed on 3/26 by just using the default value of Integer.MAX_VALUE. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385996#comment-14385996 ] Neha Narkhede commented on KAFKA-2046: -- [~onurkaraman] Shouldn't controller.message.queue.size be infinite? It seems that the moment there is some backlog of state changes on the controller, it will deadlock causing bad things to happen. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382275#comment-14382275 ] Onur Karaman commented on KAFKA-2046: - Bumping controller.message.queue.size didn't seem to affect Clark's scenario. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[
https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380776#comment-14380776
]
Onur Karaman commented on KAFKA-2046:
-
It looks like I hit a deadlock yesterday between the DeleteTopicsThread and
RequestSendThread on the controller when controller.message.queue.size was
small.
When the blocking queue shared between DeleteTopicsThread and RequestSendThread
is full, the DeleteTopicsThread puts will be blocked while holding the
controller lock until the RequestSendThread takes items from the queue, but the
RequestSendThread runs a callback after sending a request which needs the
controller lock in order to finish processing a request, causing the hang.
Delete topic performs a state transition to ReplicaDeletionStarted, and this
state transition involves a callback (deleteTopicStopReplicaCallback) that
waits on the controller lock.
This explains why I had only seen one replica from grep "handling stop replica
(delete=true)" kafka-state-change.log, as it hanged on the callback of that
replica's transition to ReplicaDeletionStarted.
Bumping up the controller.message.queue.size doesn't get rid of the deadlock
but should make it less common. I think this should only be considered a
temporary fix, as tweaking a config value shouldn't decide whether or not we
hit a deadlock.
Here are snippets of the thread dump:
{code}
"Controller-xyz-to-broker-abc-send-thread" ...
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
...
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
...
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at kafka.utils.Utils$.inLock(Utils.scala:564)
at
kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$deleteTopicStopReplicaCallback(TopicDeletionManager.scala:371)
at
kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2$$anonfun$apply$3.apply(TopicDeletionManager.scala:338)
at
kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2$$anonfun$apply$3.apply(TopicDeletionManager.scala:338)
at
kafka.controller.ControllerBrokerRequestBatch$$anonfun$addStopReplicaRequestForBrokers$2$$anonfun$apply$mcVI$sp$2.apply(ControllerChannelManager.scala:231)
at
kafka.controller.ControllerBrokerRequestBatch$$anonfun$addStopReplicaRequestForBrokers$2$$anonfun$apply$mcVI$sp$2.apply(ControllerChannelManager.scala:231)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:161)
"delete-topics-thread-xyz" ...
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
...
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
at
kafka.controller.ControllerChannelManager.sendRequest(ControllerChannelManager.scala:57)
...
at
kafka.controller.ControllerBrokerRequestBatch.sendRequestsToBrokers(ControllerChannelManager.scala:310)
at
kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:115)
...
at
kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:327)
at
kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onPartitionDeletion(TopicDeletionManager.scala:360)
...
at
kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onTopicDeletion(TopicDeletionManager.scala:305)
...
at
kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:390)
at kafka.utils.Utils$.inLock(Utils.scala:566)
at
kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:390)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
{code}
> Delete topic still doesn't work
> ---
>
> Key: KAFKA-2046
> URL: https://issues.apache.org/jira/browse/KAFKA-2046
> Project: Kafka
> Issue Type: Bug
>Reporter: Clark Haskins
>Assignee: Onur Karaman
>
> I just attempted to delete at 128 partition topic with all inbound producers
> stopped.
> The result was as follows:
> The /admin/delete_topics znode was empty
> the topic under /brokers/topics was removed
> The Kafka topics command showed that the topic was removed
> However, the data on disk on each broker was not deleted and the topic has
> not yet been re-created by starting up the inbound mirror maker.
> Let me know what additiona
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379381#comment-14379381 ] Onur Karaman commented on KAFKA-2046: - I haven't been able to reproduce the issue after using the default value for controller.message.queue.size . > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Onur Karaman > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[
https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378944#comment-14378944
]
Onur Karaman commented on KAFKA-2046:
-
I just tried deleting a topic with 128 partitions and noticed that the delete
topic node and the topic node were still in zk, and all but one replica on the
brokers had not been deleted.
grep "handling stop replica (delete=false)" kafka-state-change.log produced
output for all of the partitions. So the controller was able to send a
StopReplicaRequest to the brokers to transition from OnlineReplica to
OfflineReplica.
However, grep "handling stop replica (delete=true)" kafka-state-change.log only
revealed only one replica. This was the replica that I noticed had actually
been deleted from the filesystem. The other replicas never received the
delete=true StopReplicaRequest. So the transition from OfflineReplica to
ReplicaDeletionStarted for all the other replicas hang. A thread dump on the
controller indicates that it's getting stuck because of a LinkedBlockingQueue
in ControllerChannelManager:
{code}
"delete-topics-thread-xyz"...
java.lang.Thread.State: WAITING (parking)
...
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
at
kafka.controller.ControllerChannelManager.sendRequest(ControllerChannelManager.scala:57)
...
at
kafka.controller.KafkaController.sendRequest(KafkaController.scala:670)
at
kafka.controller.ControllerBrokerRequestBatch$$anonfun$sendRequestsToBrokers$3$$anonfun$apply$10.apply(ControllerChannelManager.scala:320)
at
kafka.controller.ControllerBrokerRequestBatch$$anonfun$sendRequestsToBrokers$3$$anonfun$apply$10.apply(ControllerChannelManager.scala:317)
at scala.collection.immutable.List.foreach(List.scala:318)
at
kafka.controller.ControllerBrokerRequestBatch$$anonfun$sendRequestsToBrokers$3.apply(ControllerChannelManager.scala:317)
at
kafka.controller.ControllerBrokerRequestBatch$$anonfun$sendRequestsToBrokers$3.apply(ControllerChannelManager.scala:310)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
at
kafka.controller.ControllerBrokerRequestBatch.sendRequestsToBrokers(ControllerChannelManager.scala:310)
at
kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:115)
at
kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:337)
at
kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:327)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at
kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:327)
{code}
controller.message.queue.size property is currently set to be very small. I'll
try bumping this up and see if this addresses the issue.
> Delete topic still doesn't work
> ---
>
> Key: KAFKA-2046
> URL: https://issues.apache.org/jira/browse/KAFKA-2046
> Project: Kafka
> Issue Type: Bug
>Reporter: Clark Haskins
>Assignee: Onur Karaman
>
> I just attempted to delete at 128 partition topic with all inbound producers
> stopped.
> The result was as follows:
> The /admin/delete_topics znode was empty
> the topic under /brokers/topics was removed
> The Kafka topics command showed that the topic was removed
> However, the data on disk on each broker was not deleted and the topic has
> not yet been re-created by starting up the inbound mirror maker.
> Let me know what additional information is needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378842#comment-14378842 ] Joel Koshy commented on KAFKA-2046: --- Hey Clark - Onur and I are looking into this. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Sriharsha Chintalapani > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378783#comment-14378783 ] Neha Narkhede commented on KAFKA-2046: -- [~clarkhaskins] As discussed previously, the minimum amount of information that is needed to troubleshoot any issue, not just delete topic is- 1. Controller logs (possibly at TRACE) 2. Server logs 3 State change log (DEBUG works) > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins >Assignee: Sriharsha Chintalapani > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378741#comment-14378741 ] Sriharsha Chintalapani commented on KAFKA-2046: --- [~clarkhaskins] can you add the details on how the big the cluster was and also do you have state-change.log files on the brokers where the Log data was not deleted. > Delete topic still doesn't work > --- > > Key: KAFKA-2046 > URL: https://issues.apache.org/jira/browse/KAFKA-2046 > Project: Kafka > Issue Type: Bug >Reporter: Clark Haskins > > I just attempted to delete at 128 partition topic with all inbound producers > stopped. > The result was as follows: > The /admin/delete_topics znode was empty > the topic under /brokers/topics was removed > The Kafka topics command showed that the topic was removed > However, the data on disk on each broker was not deleted and the topic has > not yet been re-created by starting up the inbound mirror maker. > Let me know what additional information is needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
