Problem deleting topics in 0.8.2?
So I've got 3 kafka brokers that were started with delete.topic.enable set to true. When they start, I can see in the logs that the property was successfully set. The dataset in each broker is only approximately 2G (per du). When running kafaka-delete.sh with the correct arguments to delete all of the topics, it says that the topic is marked for deletion. When running again, it says that the topic is already marked for deletion. >From reading the documentation, my understanding is that one of the 10 (default) background threads would eventually process the deletes, and clean up both the topics in zookeeper, and the actual data on disk. In reality, it didnt seem to delete the data on disk or remove anything in zookeeper. What is the correct way to remove a topic in kafka 0.8.2 and what is the expected timeframe for that to complete expected to be? My "solution" was stopping the brokers and rm -rf /var/lib/kafka/*, but that is clearly a very poor one once we are done testing our kafka + storm setup. -- Jeff Schroeder Don't drink and derive, alcohol and analysis don't mix. http://www.digitalprognosis.com
Re: Problem deleting topics in 0.8.2?
Hi Jeff, Are you seeing any errors in state-change.log or controller.log after issuing kafka-topics.sh --delete command. There is another known issue is if you have auto.topic.enable.create = true (this is true by default) your consumer or producer can re-create the topic. So try stopping any of your consumers or producers run the delete topic command again. -Harsha On Wed, Mar 4, 2015, at 10:28 AM, Jeff Schroeder wrote: > So I've got 3 kafka brokers that were started with delete.topic.enable > set > to true. When they start, I can see in the logs that the property was > successfully set. The dataset in each broker is only approximately 2G > (per > du). When running kafaka-delete.sh with the correct arguments to delete > all > of the topics, it says that the topic is marked for deletion. When > running > again, it says that the topic is already marked for deletion. > > From reading the documentation, my understanding is that one of the 10 > (default) background threads would eventually process the deletes, and > clean up both the topics in zookeeper, and the actual data on disk. In > reality, it didnt seem to delete the data on disk or remove anything in > zookeeper. > > What is the correct way to remove a topic in kafka 0.8.2 and what is the > expected timeframe for that to complete expected to be? My "solution" was > stopping the brokers and rm -rf /var/lib/kafka/*, but that is clearly a > very poor one once we are done testing our kafka + storm setup. > > -- > Jeff Schroeder > > Don't drink and derive, alcohol and analysis don't mix. > http://www.digitalprognosis.com
Re: Problem deleting topics in 0.8.2?
Hi Jeff, The controller should have a Topic deletion thread running coordinating the delete in the cluster, and the progress should be logged to the controller log. Can you look at the controller log to see what's going on? Tim On Wed, Mar 4, 2015 at 10:28 AM, Jeff Schroeder wrote: > So I've got 3 kafka brokers that were started with delete.topic.enable set > to true. When they start, I can see in the logs that the property was > successfully set. The dataset in each broker is only approximately 2G (per > du). When running kafaka-delete.sh with the correct arguments to delete all > of the topics, it says that the topic is marked for deletion. When running > again, it says that the topic is already marked for deletion. > > From reading the documentation, my understanding is that one of the 10 > (default) background threads would eventually process the deletes, and > clean up both the topics in zookeeper, and the actual data on disk. In > reality, it didnt seem to delete the data on disk or remove anything in > zookeeper. > > What is the correct way to remove a topic in kafka 0.8.2 and what is the > expected timeframe for that to complete expected to be? My "solution" was > stopping the brokers and rm -rf /var/lib/kafka/*, but that is clearly a > very poor one once we are done testing our kafka + storm setup. > > -- > Jeff Schroeder > > Don't drink and derive, alcohol and analysis don't mix. > http://www.digitalprognosis.com
Re: Problem deleting topics in 0.8.2?
Timothy and Harsha, Conveniently, a coworker figured this out almost immediately after I sent this email. I was passing the zookeeper bits as: --zookeeper 'host1:2181,host2:2181,host3:2181/path/to/zk/chroot' When the actual correct thing to do was: --zookeeper 'host1:2181/path/to/zk/chroot' It appears that the topics are removed in ~30 seconds or less, which is more inline with what I'd expect. Timothy, would you mind sharing a "production log4j.properties that you use"? This is a bit more complicated because I'm running kafka under some hand rolled docker containers running on mesos via marathon, but it works quite well. Thanks for the quick response folks! On Wed, Mar 4, 2015 at 12:35 PM, Timothy Chen wrote: > Hi Jeff, > > The controller should have a Topic deletion thread running > coordinating the delete in the cluster, and the progress should be > logged to the controller log. > > Can you look at the controller log to see what's going on? > > Tim > > On Wed, Mar 4, 2015 at 10:28 AM, Jeff Schroeder > wrote: > > So I've got 3 kafka brokers that were started with delete.topic.enable > set > > to true. When they start, I can see in the logs that the property was > > successfully set. The dataset in each broker is only approximately 2G > (per > > du). When running kafaka-delete.sh with the correct arguments to delete > all > > of the topics, it says that the topic is marked for deletion. When > running > > again, it says that the topic is already marked for deletion. > > > > From reading the documentation, my understanding is that one of the 10 > > (default) background threads would eventually process the deletes, and > > clean up both the topics in zookeeper, and the actual data on disk. In > > reality, it didnt seem to delete the data on disk or remove anything in > > zookeeper. > > > > What is the correct way to remove a topic in kafka 0.8.2 and what is the > > expected timeframe for that to complete expected to be? My "solution" was > > stopping the brokers and rm -rf /var/lib/kafka/*, but that is clearly a > > very poor one once we are done testing our kafka + storm setup. > > > > -- > > Jeff Schroeder > > > > Don't drink and derive, alcohol and analysis don't mix. > > http://www.digitalprognosis.com > -- Jeff Schroeder Don't drink and derive, alcohol and analysis don't mix. http://www.digitalprognosis.com
