So, this is rather disappointing, especially since topic deletion is really the primary feature in 0.8.2 I'm interested in. The topic I was trying to delete above had no data for many months. The consumer which is triggering recreation of that topic has been restarted several times since that topic stopped receiving data.
In our case, we have a large number of topics, that no longer receive messages (many have not received messages in over 12 months). However, we have consumers that use a regex which matches multiple topics, some of which are these empty zombie topics. Thus, we have a chicken-and-egg problem: 1. Consumer uses a regex to discover matching topics. 2. Consumer starts consuming 'topic.foo.which.is.empty'. This results in regular meta-data requests for data for that topic. 3. 'topic.foo.which.is.empty' is deleted. 4. Consumer encounters an error trying to fetch this topic, so issues a meta-data request to find the leader for this topic. 5. Broker recreates the topic in response to this. We can stop consumers for maintenance and do a batch delete, but it is problematic, because there are realtime dependencies on those consumers being up and running. I expect this will necessarily result in my voting no for 0.8.2 RC2 (if I have a vote :)). Just read up on KAFKA-1507, and left my 2 cents: "I think relegating topic creation to an admin client would be very limitiing. It's extremely useful to have a self-service system where new applications can just create a new topic on demand (with reasonable defaults), without the need for an admin to come in and prepare topics ahead of a code release (leave that to dba's managing transactional databases!). I do like the idea of an automatic create topic request from a producer, in response to a topic not found exception, rather than auto-creating topics from meta-data requests (which happens asynchronously and causes the initial meta data request to fail usually!). Consumers should never create a topic, I should think." On Mon, Jan 26, 2015 at 11:14 AM, Harsha <ka...@harsha.io> wrote: > Jun, > I made an attempt at fixing that issue as part of this JIRA > https://issues.apache.org/jira/browse/KAFKA-1507 . > As Jay pointed out there should be admin api if there is more info on > this api I am interested in adding/fixing this issue. > Thanks, > Harsha > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote: > > Yes, that's the issue. Currently, topics can be auto-created on > > TopicMetadataRequest, which can be issued from both the producer and the > > consumer. To prevent that, you would need to stop the producer and the > > consumer before deleting a topic. We plan to address this issue once we > > have a separate request for creating topics. > > > > Thanks, > > > > Jun > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha <ka...@harsha.io> wrote: > > > > > There could be another case where if you have auto.create.topics.enable > > > to set to true ( its true by default) . Any TopicMetadataRequest can > > > recreate topics. So if you issued a delete topic command and you have > > > producers running or consumers? too which is issuing a > > > TopicMetadataRequest than the topic will be recreated. > > > -Harsha > > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote: > > > > cversion did change (incremented by 2) when I issue the delete > command. > > > > > > > > From the logs on the conroller broker (also the leader for the > topic), it > > > > looks like the delete proceeds, and then the topic gets recreated > > > > immediately (highlighted in yellow). It appears maybe it’s due to a > > > > consumer client app trying to consume the topic. Also, the consumer > is > > > > not > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the > > > > problem? > > > > > > > > > > > > 2015-01-26 07:02:14,281 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine$DeleteTopicsListener - > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics > > > > mytopic > > > > 2015-01-26 07:02:14,282 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Handling deletion for topics mytopic > > > > 2015-01-26 07:02:14,286 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started > > > > 2015-01-26 07:02:14,286 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic > > > > deletion callback for mytopic > > > > 2015-01-26 07:02:14,289 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], > > > > Partition deletion callback for [mytopic,0] > > > > 2015-01-26 07:02:14,295 INFO [delete-topics-thread-6] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to OfflineReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,303 INFO [delete-topics-thread-6] > > > > controller.KafkaController - [Controller 6]: New leader and ISR for > > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]} > > > > 2015-01-26 07:02:14,312 INFO [delete-topics-thread-6] > > > > controller.KafkaController - [Controller 6]: New leader and ISR for > > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]} > > > > 2015-01-26 07:02:14,313 INFO [delete-topics-thread-6] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-5] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-7] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-7] > > > > log.OffsetIndex - Deleting index > > > > /mypath/mytopic-0/00000000000000000000.index > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-7] > log.LogManager > > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0. > > > > 2015-01-26 07:02:14,314 INFO [Controller-6-to-broker-6-send-thread] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas > > > > [Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,314 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Handling deletion for topics mytopic > > > > 2015-01-26 07:02:14,316 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Deletion for replicas 7 for partition > > > > [mytopic,0] of topic mytopic in progress > > > > 2015-01-26 07:02:14,316 INFO [Controller-6-to-broker-7-send-thread] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas > > > > [Topic=mytopic,Partition=0,Replica=7] > > > > 2015-01-26 07:02:14,316 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Handling deletion for topics mytopic > > > > 2015-01-26 07:02:14,318 INFO [delete-topics-thread-6] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to NonExistentReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=6],[Topic=mytopic,Partition=0,Replica=7] > > > > 2015-01-26 07:02:14,318 INFO [delete-topics-thread-6] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to OfflinePartition for > > > > partitions [mytopic,0] > > > > 2015-01-26 07:02:14,318 INFO [delete-topics-thread-6] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to NonExistentPartition for > > > > partitions [mytopic,0] > > > > 2015-01-26 07:02:14,325 WARN [kafka-request-handler-2] > > > > server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request > > > > with correlation id 2619442 from client ReplicaFetcherThread-0-6 on > > > > partition [mytopic,0] failed due to Partition [mytopic,0] doesn't > > > > exist on 6 > > > > 2015-01-26 07:02:14,371 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Deletion of topic mytopic successfully > > > > completed > > > > 2015-01-26 07:02:14,376 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine$TopicChangeListener - > > > > [TopicChangeListener on Controller 6]: New topics: [Set()], deleted > > > > topics: [Set()], new partition replica assignment [Map()] > > > > 2015-01-26 07:02:14,403 WARN [request-expiration-task] > > > > server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request > > > > with correlation id 2054626 from client > > > > myconsumer-1415333962444-cc78e67e-0-6 on partition [mytopic,0] failed > > > > due to Partition [mytopic,0] doesn't exist on 6 > > > > 2015-01-26 07:02:14,446 INFO [kafka-request-handler-5] > > > > admin.AdminUtils$ - Topic creation > > > > {"version":1,"partitions":{"0":[7,6]}} > > > > 2015-01-26 07:02:14,457 INFO [kafka-request-handler-5] > > > > server.KafkaApis - [KafkaApi-6] Auto creation of topic mytopic with 1 > > > > partitions and replication factor 2 is successful! > > > > 2015-01-26 07:02:14,458 INFO [kafka-network-thread-27330-2] > > > > network.Processor - Closing socket connection to /10.1.10.4. > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine$TopicChangeListener - > > > > [TopicChangeListener on Controller 6]: New topics: [Set(mytopic)], > > > > deleted topics: [Set()], new partition replica assignment > > > > [Map([mytopic,0] -> List(7, 6))] > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.KafkaController - [Controller 6]: New topic creation > > > > callback for [mytopic,0] > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.KafkaController - [Controller 6]: New partition creation > > > > callback for [mytopic,0] > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to NewPartition for partitions > > > > [mytopic,0] > > > > 2015-01-26 07:02:14,467 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to NewReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,468 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to OnlinePartition for > partitions > > > > [mytopic,0] > > > > 2015-01-26 07:02:14,530 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to OnlineReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,532 INFO [kafka-request-handler-5] log.Log - > > > > Completed load of log mytopic-0 with log end offset 0 > > > > 2015-01-26 07:02:14,533 INFO [kafka-request-handler-5] > log.LogManager > > > > - Created log for partition [mytopic,0] in /mypath with properties > > > > {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, > > > > segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, > > > > delete.retention.ms -> 86400000, index.interval.bytes -> 4096, > > > > retention.bytes -> 50000000000, min.insync.replicas -> 1, > > > > cleanup.policy -> delete, unclean.leader.election.enable -> true, > > > > segment.ms -> 604800000, max.message.bytes -> 1000012, > flush.messages > > > > -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, > retention.ms > > > > -> 86400000, segment.jitter.ms -> 0}. > > > > 2015-01-26 07:02:14,540 INFO [kafka-request-handler-5] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,540 INFO [kafka-request-handler-5] log.Log - > > > > Truncating log mytopic-0 to offset 0. > > > > 2015-01-26 07:02:14,547 INFO [kafka-request-handler-5] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Added fetcher for partitions List([[mytopic,0], initOffset 0 to > broker > > > > id:7,host:mybroker:54321] ) > > > > > > > > Here’s the log in the other broker with a replica of the topic. > > > > > > > > 2015-01-26 07:02:14,312 INFO [kafka-request-handler-7] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-2] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-2] > > > > log.OffsetIndex - Deleting index > > > > /mypath/mytopic-0/00000000000000000000.index > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-2] > log.LogManager > > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0. > > > > 2015-01-26 07:02:14,530 INFO [kafka-request-handler-0] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,532 INFO [kafka-request-handler-0] log.Log - > > > > Completed load of log mytopic-0 with log end offset 0 > > > > 2015-01-26 07:02:14,533 INFO [kafka-request-handler-0] > log.LogManager > > > > - Created log for partition [mytopic,0] in /mypath with properties > > > > {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, > > > > segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, > > > > delete.retention.ms -> 86400000, index.interval.bytes -> 4096, > > > > retention.bytes -> 50000000000, min.insync.replicas -> 1, > > > > cleanup.policy -> delete, unclean.leader.election.enable -> true, > > > > segment.ms -> 604800000, max.message.bytes -> 1000012, > flush.messages > > > > -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, > retention.ms > > > > -> 86400000, segment.jitter.ms -> 0}. > > > > > > > > On Mon, Jan 26, 2015 at 1:26 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > Could you do a get on /admin/deleted_topics and see if its cversion > > > > changes > > > > > after issuing the delete topic command? This will tell us whether > the > > > > > path /admin/deleted_topics//mytopic > > > > > is actually created or not. Anything in the controller log that > > > indicates a > > > > > delete topic event is fired? > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Sun, Jan 25, 2015 at 9:55 PM, Jason Rosenberg <j...@squareup.com > > > > > wrote: > > > > > > > > > > > yes > > > > > > > > > > > > On Mon, Jan 26, 2015 at 12:18 AM, Jun Rao <j...@confluent.io> > wrote: > > > > > > > > > > > > > Do you have delete.topic.enable turned on in all brokers? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg < > j...@squareup.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > So far, I have been unable to get delete topic to work, with > > > release > > > > > > > > candidate 2 for 0.8.2. > > > > > > > > > > > > > > > > It worked ok when I ran it in the debugger locally, on a > single > > > node > > > > > > > > instance. But when I run it in our staging environment, it > is not > > > > > > > > successfully even marking the topic for delete, for some > reason. > > > > > > > > > > > > > > > > I am setting delete.topic.enable to true (and see the > > > confirmation of > > > > > > > this > > > > > > > > in the startup logs, e.g.): > > > > > > > > > > > > > > > > INFO [main] utils.VerifiableProperties - Property > > > delete.topic.enable > > > > > > > > is overridden to true > > > > > > > > > > > > > > > > I run this command: > > > > > > > > > > > > > > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper > > > > > > > > myzkconnect:12345/mynamespace --delete --topic mytopic > > > > > > > > > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkConnection). > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkEventThread). > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > Topic mytopic is marked for deletion. > > > > > > > > Note: This will have no impact if delete.topic.enable is not > set > > > to > > > > > > true. > > > > > > > > > > > > > > > > I then do a —list which should at least show the topic > marked for > > > > > > > deletion: > > > > > > > > > > > > > > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper > > > > > > > > myzkconnect:12345/mynamespace --list --topic mytopic > > > > > > > > > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkConnection). > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkEventThread). > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > mytopic > > > > > > > > > > > > > > > > Note, it doesn’t list it as ‘marked for deletion’. > Furthermore, > > > > > waiting > > > > > > > > multiple hours still doesn’t result in the topic being > deleted. > > > > > > > > > > > > > > > > The topic has 1 partition, 2 replicas, and no data stored. > > > > > > > > > > > > > > > > In zookeeper, the /admin/deleted_topics/ path is empty. > > > > > > > > > > > > > > > > The zookeeper code looks pretty straightforward, but for some > > > reason > > > > > is > > > > > > > not > > > > > > > > writing the deleted_topics path. We are running zookeeper > 3.4.6. > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
