Thank you Mayuresh, that's an excellent write-up. As you mention about reassignment blocking deletion; I think we are seeing an issue with our brokers and reassignment so that could well be the root cause of the deletion issue we experienced. The zKclient suggestion that I followed before resolved the issue on ZK but the brokers never resolved so we rebuilt the system (luckily this is in a staging environment).
Thanks Adrian -----Original Message----- From: Mayuresh Gharat [mailto:gharatmayures...@gmail.com] Sent: 07 April 2017 18:56 To: users@kafka.apache.org Subject: Re: Topic deletion Hi Adrian, When you delete a topic, it is marked a under /admin/delete_topics path in zookeeper. Once that is done, controller gets a notification that a topic is marked for delete. The controller then notifies the replcias of the topic-partitions to delete the data for the topic and the topic itself. After KAFKA-1911, delete topic is made asynchronous and faster. The replicas on receiving the notification from controller will rename the topic-partition directory to something like topic-partition.delete and return to controller saying delete topic is done. The controller when it receives the ack from all the replicas that topic has been deleted, it will delete the znode under /admin/delete_topics path. Your delete topic is successful at this stage. The renamed directory gets removed asynchronously on those replicas brokers at some later point in time. Also remember, if you topic-partitions are been reassigned, delete topic will not be triggered till the reassignment is finished. We at Linkedin, have been suing Delete topic for a while now and it has been working fine after KAFKA-1911. Hope this helps. Thanks, Mayuresh On Fri, Apr 7, 2017 at 2:15 AM, Adrian McCague <adrian.mcca...@zopa.com> wrote: > Indeed, I wish it was explained somewhere what marked for deletion > does, and what process handles it. I will have to go and investigate the > source. > > I can confirm the zkCli did the trick, thanks for the hint! > > Adrian > > -----Original Message----- > From: Akhilesh Pathodia [mailto:pathodia.akhil...@gmail.com] > Sent: 07 April 2017 09:57 > To: users@kafka.apache.org > Subject: Re: Topic deletion > > I am not sure but kafka delete command does not delete the topic > actually, it only marks it for deletion. Probably it is fixed in later > version of kafka. > > On Fri, Apr 7, 2017 at 2:14 PM, Adrian McCague > <adrian.mcca...@zopa.com> > wrote: > > > Hi Akhilesh, > > > > Why would this approach need to be taken over the kafka-topics tool, > > out of interest? > > > > Thanks > > Adrian > > > > -----Original Message----- > > From: Akhilesh Pathodia [mailto:pathodia.akhil...@gmail.com] > > Sent: 07 April 2017 09:37 > > To: users@kafka.apache.org > > Subject: Re: Topic deletion > > > > Hi Adrian, > > > > You will have to delete the broker directory from zookeeper. This > > can be done from zookeeper cli. Connect to zookeeper cli using > > below > command: > > > > zookeeper-client -server <ZOOKEEPER_URI> > > > > Then run below command : > > > > rmr /brokers/topics/<TOPIC_NAME> > > > > Thanks, > > AKhilesh > > > > On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague > > <adrian.mcca...@zopa.com> > > wrote: > > > > > Hi all, > > > > > > I am trying to understand topic deletion in kafka, there appears > > > to be very little documentation or answers on how this works. > > > Typically they just say to turn on the feature on the broker (in my case > > > it is). > > > > > > I executed: > > > Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic > > > > > > Running this again yields: > > > Topic mytopic is already marked for deletion. > > > > > > --describe yields: > > > Topic: mytopic PartitionCount:6 ReplicationFactor:3 > Configs: > > > retention.ms=0 > > > Topic: mytopic Partition: 0 Leader: -1 Replicas: > > > 1006,1001,1005 Isr: > > > Topic mytopic Partition: 1 Leader: -1 Replicas: > > > 1001,1005,1003 Isr: > > > Topic: mytopic Partition: 2 Leader: -1 Replicas: > > > 1005,1003,1004 Isr: > > > Topic: mytopic Partition: 3 Leader: -1 Replicas: > > > 1003,1004,1007 Isr: > > > Topic: mytopic Partition: 4 Leader: -1 Replicas: > > > 1004,1007,1006 Isr: > > > Topic: mytopic Partition: 5 Leader: -1 Replicas: > > > 1007,1006,1001 Isr: > > > > > > You can see that the deletion mark has meant that the Leader is -1. > > > Also I read somewhere that retention needs to be set to something > > > low to trigger the deletion, hence the config of retention.ms=0 > > > > > > Consumers (or streams in my case) no longer see the topic: > > > org.apache.kafka.streams.errors.TopologyBuilderException: Invalid > > > topology building: stream-thread [StreamThread-1] Topic not found: > > > mytopic > > > > > > And I can't create a new topic in its place: > > > [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common. > > errors.TopicExistsException: > > > Topic 'mytopic' already exists. (kafka.admin.TopicCommand$) > > > > > > I am a little lost as to where to go next, could someone explain > > > how topic deletion is actually applied when a topic is 'marked' > > > for deletion as that may help trigger it. > > > > > > Thanks! > > > Adrian > > > > > > > > > -- -Regards, Mayuresh R. Gharat (862) 250-7125
