cversion did change (incremented by 2) when I issue the delete command.
>From the logs on the conroller broker (also the leader for the topic), it
looks like the delete proceeds, and then the topic gets recreated
immediately (highlighted in yellow). It appears maybe it’s due to a
consumer client app trying to consume the topic. Also, the consumer is not
yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
problem?
2015-01-26 07:02:14,281 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.PartitionStateMachine$DeleteTopicsListener -
[DeleteTopicsListener on 6]: Starting topic deletion for topics
mytopic
2015-01-26 07:02:14,282 INFO [delete-topics-thread-6]
controller.TopicDeletionManager$DeleteTopicsThread -
[delete-topics-thread-6], Handling deletion for topics mytopic
2015-01-26 07:02:14,286 INFO [delete-topics-thread-6]
controller.TopicDeletionManager$DeleteTopicsThread -
[delete-topics-thread-6], Deletion of topic mytopic (re)started
2015-01-26 07:02:14,286 INFO [delete-topics-thread-6]
controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
deletion callback for mytopic
2015-01-26 07:02:14,289 INFO [delete-topics-thread-6]
controller.TopicDeletionManager - [Topic Deletion Manager 6],
Partition deletion callback for [mytopic,0]
2015-01-26 07:02:14,295 INFO [delete-topics-thread-6]
controller.ReplicaStateMachine - [Replica state machine on controller
6]: Invoking state change to OfflineReplica for replicas
[Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
2015-01-26 07:02:14,303 INFO [delete-topics-thread-6]
controller.KafkaController - [Controller 6]: New leader and ISR for
partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
2015-01-26 07:02:14,312 INFO [delete-topics-thread-6]
controller.KafkaController - [Controller 6]: New leader and ISR for
partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
2015-01-26 07:02:14,313 INFO [delete-topics-thread-6]
controller.ReplicaStateMachine - [Replica state machine on controller
6]: Invoking state change to ReplicaDeletionStarted for replicas
[Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
2015-01-26 07:02:14,313 INFO [kafka-request-handler-5]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
Removed fetcher for partitions [mytopic,0]
2015-01-26 07:02:14,313 INFO [kafka-request-handler-7]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
Removed fetcher for partitions [mytopic,0]
2015-01-26 07:02:14,313 INFO [kafka-request-handler-7]
log.OffsetIndex - Deleting index
/mypath/mytopic-0/00000000000000000000.index
2015-01-26 07:02:14,313 INFO [kafka-request-handler-7] log.LogManager
- Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
2015-01-26 07:02:14,314 INFO [Controller-6-to-broker-6-send-thread]
controller.ReplicaStateMachine - [Replica state machine on controller
6]: Invoking state change to ReplicaDeletionSuccessful for replicas
[Topic=mytopic,Partition=0,Replica=6]
2015-01-26 07:02:14,314 INFO [delete-topics-thread-6]
controller.TopicDeletionManager$DeleteTopicsThread -
[delete-topics-thread-6], Handling deletion for topics mytopic
2015-01-26 07:02:14,316 INFO [delete-topics-thread-6]
controller.TopicDeletionManager$DeleteTopicsThread -
[delete-topics-thread-6], Deletion for replicas 7 for partition
[mytopic,0] of topic mytopic in progress
2015-01-26 07:02:14,316 INFO [Controller-6-to-broker-7-send-thread]
controller.ReplicaStateMachine - [Replica state machine on controller
6]: Invoking state change to ReplicaDeletionSuccessful for replicas
[Topic=mytopic,Partition=0,Replica=7]
2015-01-26 07:02:14,316 INFO [delete-topics-thread-6]
controller.TopicDeletionManager$DeleteTopicsThread -
[delete-topics-thread-6], Handling deletion for topics mytopic
2015-01-26 07:02:14,318 INFO [delete-topics-thread-6]
controller.ReplicaStateMachine - [Replica state machine on controller
6]: Invoking state change to NonExistentReplica for replicas
[Topic=mytopic,Partition=0,Replica=6],[Topic=mytopic,Partition=0,Replica=7]
2015-01-26 07:02:14,318 INFO [delete-topics-thread-6]
controller.PartitionStateMachine - [Partition state machine on
Controller 6]: Invoking state change to OfflinePartition for
partitions [mytopic,0]
2015-01-26 07:02:14,318 INFO [delete-topics-thread-6]
controller.PartitionStateMachine - [Partition state machine on
Controller 6]: Invoking state change to NonExistentPartition for
partitions [mytopic,0]
2015-01-26 07:02:14,325 WARN [kafka-request-handler-2]
server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request
with correlation id 2619442 from client ReplicaFetcherThread-0-6 on
partition [mytopic,0] failed due to Partition [mytopic,0] doesn't
exist on 6
2015-01-26 07:02:14,371 INFO [delete-topics-thread-6]
controller.TopicDeletionManager$DeleteTopicsThread -
[delete-topics-thread-6], Deletion of topic mytopic successfully
completed
2015-01-26 07:02:14,376 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.PartitionStateMachine$TopicChangeListener -
[TopicChangeListener on Controller 6]: New topics: [Set()], deleted
topics: [Set()], new partition replica assignment [Map()]
2015-01-26 07:02:14,403 WARN [request-expiration-task]
server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request
with correlation id 2054626 from client
myconsumer-1415333962444-cc78e67e-0-6 on partition [mytopic,0] failed
due to Partition [mytopic,0] doesn't exist on 6
2015-01-26 07:02:14,446 INFO [kafka-request-handler-5]
admin.AdminUtils$ - Topic creation
{"version":1,"partitions":{"0":[7,6]}}
2015-01-26 07:02:14,457 INFO [kafka-request-handler-5]
server.KafkaApis - [KafkaApi-6] Auto creation of topic mytopic with 1
partitions and replication factor 2 is successful!
2015-01-26 07:02:14,458 INFO [kafka-network-thread-27330-2]
network.Processor - Closing socket connection to /10.1.10.4.
2015-01-26 07:02:14,466 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.PartitionStateMachine$TopicChangeListener -
[TopicChangeListener on Controller 6]: New topics: [Set(mytopic)],
deleted topics: [Set()], new partition replica assignment
[Map([mytopic,0] -> List(7, 6))]
2015-01-26 07:02:14,466 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.KafkaController - [Controller 6]: New topic creation
callback for [mytopic,0]
2015-01-26 07:02:14,466 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.KafkaController - [Controller 6]: New partition creation
callback for [mytopic,0]
2015-01-26 07:02:14,466 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.PartitionStateMachine - [Partition state machine on
Controller 6]: Invoking state change to NewPartition for partitions
[mytopic,0]
2015-01-26 07:02:14,467 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.ReplicaStateMachine - [Replica state machine on controller
6]: Invoking state change to NewReplica for replicas
[Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
2015-01-26 07:02:14,468 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.PartitionStateMachine - [Partition state machine on
Controller 6]: Invoking state change to OnlinePartition for partitions
[mytopic,0]
2015-01-26 07:02:14,530 INFO
[ZkClient-EventThread-21-myzkserver:12345/mynamespace]
controller.ReplicaStateMachine - [Replica state machine on controller
6]: Invoking state change to OnlineReplica for replicas
[Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
2015-01-26 07:02:14,532 INFO [kafka-request-handler-5] log.Log -
Completed load of log mytopic-0 with log end offset 0
2015-01-26 07:02:14,533 INFO [kafka-request-handler-5] log.LogManager
- Created log for partition [mytopic,0] in /mypath with properties
{segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000,
segment.bytes -> 1073741824, flush.ms -> 9223372036854775807,
delete.retention.ms -> 86400000, index.interval.bytes -> 4096,
retention.bytes -> 50000000000, min.insync.replicas -> 1,
cleanup.policy -> delete, unclean.leader.election.enable -> true,
segment.ms -> 604800000, max.message.bytes -> 1000012, flush.messages
-> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, retention.ms
-> 86400000, segment.jitter.ms -> 0}.
2015-01-26 07:02:14,540 INFO [kafka-request-handler-5]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
Removed fetcher for partitions [mytopic,0]
2015-01-26 07:02:14,540 INFO [kafka-request-handler-5] log.Log -
Truncating log mytopic-0 to offset 0.
2015-01-26 07:02:14,547 INFO [kafka-request-handler-5]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
Added fetcher for partitions List([[mytopic,0], initOffset 0 to broker
id:7,host:mybroker:54321] )
Here’s the log in the other broker with a replica of the topic.
2015-01-26 07:02:14,312 INFO [kafka-request-handler-7]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7]
Removed fetcher for partitions [mytopic,0]
2015-01-26 07:02:14,313 INFO [kafka-request-handler-2]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7]
Removed fetcher for partitions [mytopic,0]
2015-01-26 07:02:14,313 INFO [kafka-request-handler-2]
log.OffsetIndex - Deleting index
/mypath/mytopic-0/00000000000000000000.index
2015-01-26 07:02:14,313 INFO [kafka-request-handler-2] log.LogManager
- Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
2015-01-26 07:02:14,530 INFO [kafka-request-handler-0]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7]
Removed fetcher for partitions [mytopic,0]
2015-01-26 07:02:14,532 INFO [kafka-request-handler-0] log.Log -
Completed load of log mytopic-0 with log end offset 0
2015-01-26 07:02:14,533 INFO [kafka-request-handler-0] log.LogManager
- Created log for partition [mytopic,0] in /mypath with properties
{segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000,
segment.bytes -> 1073741824, flush.ms -> 9223372036854775807,
delete.retention.ms -> 86400000, index.interval.bytes -> 4096,
retention.bytes -> 50000000000, min.insync.replicas -> 1,
cleanup.policy -> delete, unclean.leader.election.enable -> true,
segment.ms -> 604800000, max.message.bytes -> 1000012, flush.messages
-> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, retention.ms
-> 86400000, segment.jitter.ms -> 0}.
On Mon, Jan 26, 2015 at 1:26 AM, Jun Rao <j...@confluent.io> wrote:
Could you do a get on /admin/deleted_topics and see if its cversion changes
> after issuing the delete topic command? This will tell us whether the
> path /admin/deleted_topics//mytopic
> is actually created or not. Anything in the controller log that indicates a
> delete topic event is fired?
>
> Thanks,
>
> Jun
>
> On Sun, Jan 25, 2015 at 9:55 PM, Jason Rosenberg <j...@squareup.com> wrote:
>
> > yes
> >
> > On Mon, Jan 26, 2015 at 12:18 AM, Jun Rao <j...@confluent.io> wrote:
> >
> > > Do you have delete.topic.enable turned on in all brokers?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg <j...@squareup.com>
> > wrote:
> > >
> > > > So far, I have been unable to get delete topic to work, with release
> > > > candidate 2 for 0.8.2.
> > > >
> > > > It worked ok when I ran it in the debugger locally, on a single node
> > > > instance. But when I run it in our staging environment, it is not
> > > > successfully even marking the topic for delete, for some reason.
> > > >
> > > > I am setting delete.topic.enable to true (and see the confirmation of
> > > this
> > > > in the startup logs, e.g.):
> > > >
> > > > INFO [main] utils.VerifiableProperties - Property delete.topic.enable
> > > > is overridden to true
> > > >
> > > > I run this command:
> > > >
> > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > > myzkconnect:12345/mynamespace --delete --topic mytopic
> > > >
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkConnection).
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkEventThread).
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > Topic mytopic is marked for deletion.
> > > > Note: This will have no impact if delete.topic.enable is not set to
> > true.
> > > >
> > > > I then do a —list which should at least show the topic marked for
> > > deletion:
> > > >
> > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > > myzkconnect:12345/mynamespace --list --topic mytopic
> > > >
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkConnection).
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkEventThread).
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > mytopic
> > > >
> > > > Note, it doesn’t list it as ‘marked for deletion’. Furthermore,
> waiting
> > > > multiple hours still doesn’t result in the topic being deleted.
> > > >
> > > > The topic has 1 partition, 2 replicas, and no data stored.
> > > >
> > > > In zookeeper, the /admin/deleted_topics/ path is empty.
> > > >
> > > > The zookeeper code looks pretty straightforward, but for some reason
> is
> > > not
> > > > writing the deleted_topics path. We are running zookeeper 3.4.6.
> > > >
> > > > Thoughts?
> > > >
> > > > Jason
> > > >
> > > >
> > >
> >
>
