It then seems to me that delete-topic will not actually work "smoothly" until create topic request is added since it is too much to require people to turn off their clients while deleting topics. In this case shall we make it clear in the release docs of in 0.8.2, or even still mark it as not-supported?
Guozhang On Mon, Jan 26, 2015 at 8:14 AM, Harsha <ka...@harsha.io> wrote: > Jun, > I made an attempt at fixing that issue as part of this JIRA > https://issues.apache.org/jira/browse/KAFKA-1507 . > As Jay pointed out there should be admin api if there is more info on > this api I am interested in adding/fixing this issue. > Thanks, > Harsha > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote: > > Yes, that's the issue. Currently, topics can be auto-created on > > TopicMetadataRequest, which can be issued from both the producer and the > > consumer. To prevent that, you would need to stop the producer and the > > consumer before deleting a topic. We plan to address this issue once we > > have a separate request for creating topics. > > > > Thanks, > > > > Jun > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha <ka...@harsha.io> wrote: > > > > > There could be another case where if you have auto.create.topics.enable > > > to set to true ( its true by default) . Any TopicMetadataRequest can > > > recreate topics. So if you issued a delete topic command and you have > > > producers running or consumers? too which is issuing a > > > TopicMetadataRequest than the topic will be recreated. > > > -Harsha > > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote: > > > > cversion did change (incremented by 2) when I issue the delete > command. > > > > > > > > From the logs on the conroller broker (also the leader for the > topic), it > > > > looks like the delete proceeds, and then the topic gets recreated > > > > immediately (highlighted in yellow). It appears maybe it’s due to a > > > > consumer client app trying to consume the topic. Also, the consumer > is > > > > not > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the > > > > problem? > > > > > > > > > > > > 2015-01-26 07:02:14,281 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine$DeleteTopicsListener - > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics > > > > mytopic > > > > 2015-01-26 07:02:14,282 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Handling deletion for topics mytopic > > > > 2015-01-26 07:02:14,286 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started > > > > 2015-01-26 07:02:14,286 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic > > > > deletion callback for mytopic > > > > 2015-01-26 07:02:14,289 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], > > > > Partition deletion callback for [mytopic,0] > > > > 2015-01-26 07:02:14,295 INFO [delete-topics-thread-6] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to OfflineReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,303 INFO [delete-topics-thread-6] > > > > controller.KafkaController - [Controller 6]: New leader and ISR for > > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]} > > > > 2015-01-26 07:02:14,312 INFO [delete-topics-thread-6] > > > > controller.KafkaController - [Controller 6]: New leader and ISR for > > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]} > > > > 2015-01-26 07:02:14,313 INFO [delete-topics-thread-6] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-5] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-7] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-7] > > > > log.OffsetIndex - Deleting index > > > > /mypath/mytopic-0/00000000000000000000.index > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-7] > log.LogManager > > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0. > > > > 2015-01-26 07:02:14,314 INFO [Controller-6-to-broker-6-send-thread] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas > > > > [Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,314 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Handling deletion for topics mytopic > > > > 2015-01-26 07:02:14,316 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Deletion for replicas 7 for partition > > > > [mytopic,0] of topic mytopic in progress > > > > 2015-01-26 07:02:14,316 INFO [Controller-6-to-broker-7-send-thread] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas > > > > [Topic=mytopic,Partition=0,Replica=7] > > > > 2015-01-26 07:02:14,316 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Handling deletion for topics mytopic > > > > 2015-01-26 07:02:14,318 INFO [delete-topics-thread-6] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to NonExistentReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=6],[Topic=mytopic,Partition=0,Replica=7] > > > > 2015-01-26 07:02:14,318 INFO [delete-topics-thread-6] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to OfflinePartition for > > > > partitions [mytopic,0] > > > > 2015-01-26 07:02:14,318 INFO [delete-topics-thread-6] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to NonExistentPartition for > > > > partitions [mytopic,0] > > > > 2015-01-26 07:02:14,325 WARN [kafka-request-handler-2] > > > > server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request > > > > with correlation id 2619442 from client ReplicaFetcherThread-0-6 on > > > > partition [mytopic,0] failed due to Partition [mytopic,0] doesn't > > > > exist on 6 > > > > 2015-01-26 07:02:14,371 INFO [delete-topics-thread-6] > > > > controller.TopicDeletionManager$DeleteTopicsThread - > > > > [delete-topics-thread-6], Deletion of topic mytopic successfully > > > > completed > > > > 2015-01-26 07:02:14,376 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine$TopicChangeListener - > > > > [TopicChangeListener on Controller 6]: New topics: [Set()], deleted > > > > topics: [Set()], new partition replica assignment [Map()] > > > > 2015-01-26 07:02:14,403 WARN [request-expiration-task] > > > > server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request > > > > with correlation id 2054626 from client > > > > myconsumer-1415333962444-cc78e67e-0-6 on partition [mytopic,0] failed > > > > due to Partition [mytopic,0] doesn't exist on 6 > > > > 2015-01-26 07:02:14,446 INFO [kafka-request-handler-5] > > > > admin.AdminUtils$ - Topic creation > > > > {"version":1,"partitions":{"0":[7,6]}} > > > > 2015-01-26 07:02:14,457 INFO [kafka-request-handler-5] > > > > server.KafkaApis - [KafkaApi-6] Auto creation of topic mytopic with 1 > > > > partitions and replication factor 2 is successful! > > > > 2015-01-26 07:02:14,458 INFO [kafka-network-thread-27330-2] > > > > network.Processor - Closing socket connection to /10.1.10.4. > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine$TopicChangeListener - > > > > [TopicChangeListener on Controller 6]: New topics: [Set(mytopic)], > > > > deleted topics: [Set()], new partition replica assignment > > > > [Map([mytopic,0] -> List(7, 6))] > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.KafkaController - [Controller 6]: New topic creation > > > > callback for [mytopic,0] > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.KafkaController - [Controller 6]: New partition creation > > > > callback for [mytopic,0] > > > > 2015-01-26 07:02:14,466 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to NewPartition for partitions > > > > [mytopic,0] > > > > 2015-01-26 07:02:14,467 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to NewReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,468 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.PartitionStateMachine - [Partition state machine on > > > > Controller 6]: Invoking state change to OnlinePartition for > partitions > > > > [mytopic,0] > > > > 2015-01-26 07:02:14,530 INFO > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace] > > > > controller.ReplicaStateMachine - [Replica state machine on controller > > > > 6]: Invoking state change to OnlineReplica for replicas > > > > > > > > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6] > > > > 2015-01-26 07:02:14,532 INFO [kafka-request-handler-5] log.Log - > > > > Completed load of log mytopic-0 with log end offset 0 > > > > 2015-01-26 07:02:14,533 INFO [kafka-request-handler-5] > log.LogManager > > > > - Created log for partition [mytopic,0] in /mypath with properties > > > > {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, > > > > segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, > > > > delete.retention.ms -> 86400000, index.interval.bytes -> 4096, > > > > retention.bytes -> 50000000000, min.insync.replicas -> 1, > > > > cleanup.policy -> delete, unclean.leader.election.enable -> true, > > > > segment.ms -> 604800000, max.message.bytes -> 1000012, > flush.messages > > > > -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, > retention.ms > > > > -> 86400000, segment.jitter.ms -> 0}. > > > > 2015-01-26 07:02:14,540 INFO [kafka-request-handler-5] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,540 INFO [kafka-request-handler-5] log.Log - > > > > Truncating log mytopic-0 to offset 0. > > > > 2015-01-26 07:02:14,547 INFO [kafka-request-handler-5] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6] > > > > Added fetcher for partitions List([[mytopic,0], initOffset 0 to > broker > > > > id:7,host:mybroker:54321] ) > > > > > > > > Here’s the log in the other broker with a replica of the topic. > > > > > > > > 2015-01-26 07:02:14,312 INFO [kafka-request-handler-7] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-2] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-2] > > > > log.OffsetIndex - Deleting index > > > > /mypath/mytopic-0/00000000000000000000.index > > > > 2015-01-26 07:02:14,313 INFO [kafka-request-handler-2] > log.LogManager > > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0. > > > > 2015-01-26 07:02:14,530 INFO [kafka-request-handler-0] > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7] > > > > Removed fetcher for partitions [mytopic,0] > > > > 2015-01-26 07:02:14,532 INFO [kafka-request-handler-0] log.Log - > > > > Completed load of log mytopic-0 with log end offset 0 > > > > 2015-01-26 07:02:14,533 INFO [kafka-request-handler-0] > log.LogManager > > > > - Created log for partition [mytopic,0] in /mypath with properties > > > > {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, > > > > segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, > > > > delete.retention.ms -> 86400000, index.interval.bytes -> 4096, > > > > retention.bytes -> 50000000000, min.insync.replicas -> 1, > > > > cleanup.policy -> delete, unclean.leader.election.enable -> true, > > > > segment.ms -> 604800000, max.message.bytes -> 1000012, > flush.messages > > > > -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, > retention.ms > > > > -> 86400000, segment.jitter.ms -> 0}. > > > > > > > > On Mon, Jan 26, 2015 at 1:26 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > Could you do a get on /admin/deleted_topics and see if its cversion > > > > changes > > > > > after issuing the delete topic command? This will tell us whether > the > > > > > path /admin/deleted_topics//mytopic > > > > > is actually created or not. Anything in the controller log that > > > indicates a > > > > > delete topic event is fired? > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Sun, Jan 25, 2015 at 9:55 PM, Jason Rosenberg <j...@squareup.com > > > > > wrote: > > > > > > > > > > > yes > > > > > > > > > > > > On Mon, Jan 26, 2015 at 12:18 AM, Jun Rao <j...@confluent.io> > wrote: > > > > > > > > > > > > > Do you have delete.topic.enable turned on in all brokers? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg < > j...@squareup.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > So far, I have been unable to get delete topic to work, with > > > release > > > > > > > > candidate 2 for 0.8.2. > > > > > > > > > > > > > > > > It worked ok when I ran it in the debugger locally, on a > single > > > node > > > > > > > > instance. But when I run it in our staging environment, it > is not > > > > > > > > successfully even marking the topic for delete, for some > reason. > > > > > > > > > > > > > > > > I am setting delete.topic.enable to true (and see the > > > confirmation of > > > > > > > this > > > > > > > > in the startup logs, e.g.): > > > > > > > > > > > > > > > > INFO [main] utils.VerifiableProperties - Property > > > delete.topic.enable > > > > > > > > is overridden to true > > > > > > > > > > > > > > > > I run this command: > > > > > > > > > > > > > > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper > > > > > > > > myzkconnect:12345/mynamespace --delete --topic mytopic > > > > > > > > > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkConnection). > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkEventThread). > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > Topic mytopic is marked for deletion. > > > > > > > > Note: This will have no impact if delete.topic.enable is not > set > > > to > > > > > > true. > > > > > > > > > > > > > > > > I then do a —list which should at least show the topic > marked for > > > > > > > deletion: > > > > > > > > > > > > > > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper > > > > > > > > myzkconnect:12345/mynamespace --list --topic mytopic > > > > > > > > > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkConnection). > > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > > (org.I0Itec.zkclient.ZkEventThread). > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > log4j:WARN See > > > http://logging.apache.org/log4j/1.2/faq.html#noconfig > > > > > > > > for more info. > > > > > > > > mytopic > > > > > > > > > > > > > > > > Note, it doesn’t list it as ‘marked for deletion’. > Furthermore, > > > > > waiting > > > > > > > > multiple hours still doesn’t result in the topic being > deleted. > > > > > > > > > > > > > > > > The topic has 1 partition, 2 replicas, and no data stored. > > > > > > > > > > > > > > > > In zookeeper, the /admin/deleted_topics/ path is empty. > > > > > > > > > > > > > > > > The zookeeper code looks pretty straightforward, but for some > > > reason > > > > > is > > > > > > > not > > > > > > > > writing the deleted_topics path. We are running zookeeper > 3.4.6. > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- -- Guozhang
