If it issues topic metadata requests to find out what brokers the topic partitions live on (so it can make specific requests, like offsets, or getting JMX metrics from those brokers for the partitions), then it could easily have caused this. We’ve seen this behavior from multiple types of monitoring applications.
-Todd On Mon, Dec 12, 2016 at 8:47 AM, Tim Visher <tim.vis...@gmail.com> wrote: > I wonder if datadog monitoring triggers that behavior. That's the only > other piece of our infrastructure that may have been talking to that topic. > > On Mon, Dec 12, 2016 at 12:40 AM, Surendra , Manchikanti < > surendra.manchika...@gmail.com> wrote: > > > If "auto.create.topics.enable" is set to true in your configurations , > any > > producer/consumer or fetch request will create the topic again. Set it to > > false and delete the topic. > > > > -- Surendra Manchikanti > > > > On Sat, Dec 10, 2016 at 10:59 AM, Todd Palino <tpal...@gmail.com> wrote: > > > > > Are you running something else besides the consumers that would > maintain > > a > > > memory of the topics and potentially recreate them by issuing a > metadata > > > request? For example, Burrow (the consumer monitoring app I wrote) does > > > this because it maintains a list of all topics in memory, and will end > up > > > recreating a topic that has been deleted as it issues a metadata > request > > to > > > try and find out what happened after an offset request for the topic > > fails. > > > > > > -Todd > > > > > > > > > On Fri, Dec 9, 2016 at 8:37 AM, Tim Visher <tim.vis...@gmail.com> > wrote: > > > > > > > On Fri, Dec 9, 2016 at 11:34 AM, Todd Palino <tpal...@gmail.com> > > wrote: > > > > > > > > > Given that you said you removed the log directories, and provided > > that > > > > when > > > > > you did the rmr on Zookeeper it was to the “/brokers/topics/(topic > > > name)” > > > > > path, you did the right things for a manual deletion. It sounds > like > > > you > > > > > may have a consumer (or other client) that is recreating the topic. > > Do > > > > you > > > > > have auto topic creation enabled? > > > > > > > > > > > > > That was the last epiphany we had. We had shut down the producer but > > not > > > > all the consumers and we do allow auto-topic creation. > > > > > > > > That said, we then proceeded to shut all of them down (the consumers) > > and > > > > the topic came back. I'm glad that we were doing the right steps > > though. > > > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > > On Fri, Dec 9, 2016 at 8:25 AM, Tim Visher <tim.vis...@gmail.com> > > > wrote: > > > > > > > > > > > I did all of that because setting delete.topic.enable=true wasn't > > > > > > effective. We set that across every broker, restarted them, and > > then > > > > > > deleted the topic, and it was still stuck in existence. > > > > > > > > > > > > On Fri, Dec 9, 2016 at 11:11 AM, Ali Akhtar < > ali.rac...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > You need to also delete / restart zookeeper, its probably > storing > > > the > > > > > > > topics there. (Or yeah, just enable it and then delete the > topic) > > > > > > > > > > > > > > On Fri, Dec 9, 2016 at 9:09 PM, Rodrigo Sandoval < > > > > > > > rodrigo.madfe...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Why did you do all those things instead of just setting > > > > > > > > delete.topic.enable=true? > > > > > > > > > > > > > > > > On Dec 9, 2016 13:40, "Tim Visher" <tim.vis...@gmail.com> > > wrote: > > > > > > > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > > > > > > > I'm really confused at the moment. We created a topic with > > > > brokers > > > > > > set > > > > > > > to > > > > > > > > > delete.topic.enable=false. > > > > > > > > > > > > > > > > > > We now need to delete that topic. To do that we shut down > all > > > the > > > > > > > > brokers, > > > > > > > > > deleted everything under log.dirs and logs.dir on all the > > kafka > > > > > > > brokers, > > > > > > > > > `rmr`ed the entire chroot that kafka was storing things > under > > > in > > > > > > > > zookeeper, > > > > > > > > > and then brought kafka back up. > > > > > > > > > > > > > > > > > > After doing all that, the topic comes back, every time. > > > > > > > > > > > > > > > > > > What can we do to delete that topic? > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > In Christ, > > > > > > > > > > > > > > > > > > Timmy V. > > > > > > > > > > > > > > > > > > http://blog.twonegatives.com/ > > > > > > > > > http://five.sentenc.es/ -- Spend less time on mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Todd Palino* > > > > > Staff Site Reliability Engineer > > > > > Data Infrastructure Streaming > > > > > > > > > > > > > > > > > > > > linkedin.com/in/toddpalino > > > > > > > > > > > > > > > > > > > > > -- > > > *Todd Palino* > > > Staff Site Reliability Engineer > > > Data Infrastructure Streaming > > > > > > > > > > > > linkedin.com/in/toddpalino > > > > > > -- *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
