Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
Gotcha. Thanks again. Will post back once I've tried this with an update! Chris On Fri, Jul 21, 2017 at 1:12 PM, Carl Haferdwrote: > I would recommend allowing each broker enough time to catch-up before > starting the next, but this may be less of a concern if the entire cluster > is being brought down and then started from scratch. To automate, we poll > until the Kafka process binds to its configured port (9092), and then once > all brokers are up we also poll until there are no longer any > under-replicated partitions before performing further maintenance. > > Carl > > On Fri, Jul 21, 2017 at 10:50 AM, Chris Neal wrote: > > > Thanks Carl. > > > > Always fun to do this stuff in production... ;) > > > > Appreciate the input. I'll try a full cycle and see how that works. > > > > In your opinion, if I stop all brokers and all Zookeeper nodes, then > > restart all Zookeepers...at that point can I start both brokers at the > same > > time, or should I let one broker fully start and read all the unflushed > > segments from disk before starting the second broker? > > > > Again, many thanks. > > Chris > > > > > > On Fri, Jul 21, 2017 at 12:13 PM, Carl Haferd > > > > > wrote: > > > > > I have encountered similar difficulties in a test environment and it > may > > be > > > necessary to stop the Kafka process on each broker and take Zookeeper > > > offline before removing the files and zookeeper paths. Otherwise there > > may > > > be a race condition between brokers which could cause the cluster to > > retain > > > information for the topic. > > > > > > Carl > > > > > > On Fri, Jul 21, 2017 at 9:06 AM, Chris Neal wrote: > > > > > > > Welp. Surprisingly, that did not fix the problem. :( > > > > > > > > I cleaned out all the entries for these topics from /config/topics, > and > > > > removed the logs from the file system for those topics, and the > > messages > > > > are still flying by in the server.log file. > > > > > > > > Also, more concerning, when I was looking through the log files for > the > > > > other broker in the cluster, I noticed the same type of message for a > > > topic > > > > that should actually be there: > > > > > > > > [2017-07-21 16:03:29,140] ERROR Conditional update of path > > > > /brokers/topics/perf_dstorage_raw/partitions/4/state with data > > > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > > > epoch":0,"isr":[0]} > > > > and expected version 0 failed due to > > > > org.apache.zookeeper.KeeperException$BadVersionException: > > > KeeperErrorCode > > > > = > > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state > > > > (kafka.utils.ZkUtils$) > > > > [2017-07-21 16:03:29,142] ERROR Conditional update of path > > > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > > > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > > > epoch":0,"isr":[0]} > > > > and expected version 0 failed due to > > > > org.apache.zookeeper.KeeperException$BadVersionException: > > > KeeperErrorCode > > > > = > > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > > > > (kafka.utils.ZkUtils$) > > > > [2017-07-21 16:03:29,142] ERROR Conditional update of path > > > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > > > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > > > epoch":0,"isr":[0]} > > > > and expected version 0 failed due to > > > > org.apache.zookeeper.KeeperException$BadVersionException: > > > KeeperErrorCode > > > > = > > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > > > > (kafka.utils.ZkUtils$) > > > > > > > > So, the issue is not isolated to just these > "should-have-been-removed" > > > > topics, unfortunately. > > > > > > > > Really appreciate the input so far everyone. Still looking though > for > > a > > > > solution. Many thanks. :) > > > > > > > > Chris > > > > > > > > On Fri, Jul 21, 2017 at 10:58 AM, M. Manna > wrote: > > > > > > > > > Just to add (in case the platoform is Windows) > > > > > > > > > > For Windows based cluster implementation, log/topic cleanup doesn't > > > work > > > > > out of the box. Users are more or less aware of it, and doing their > > own > > > > > maintenance as workaround. > > > > > If you have issues on Topic deletion not working properly on > Windows > > > > (i.e. > > > > > with topic deletion enable and all other settings). then you have > to > > > > > manually delete the files. > > > > > > > > > > > > > > > On 21 July 2017 at 16:53, Chris Neal wrote: > > > > > > > > > > > @Carl, > > > > > > > > > > > > There is nothing under /admin/delete_topics other than > > > > > > > > > > > > [] > > > > > > > > > > > > And nothing under /admin other than delete_topics :) > > > > > > > > > > > > The topics DO exist, however, under /config/topics! We may be on > > to > > > > > > something. I will
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
I would recommend allowing each broker enough time to catch-up before starting the next, but this may be less of a concern if the entire cluster is being brought down and then started from scratch. To automate, we poll until the Kafka process binds to its configured port (9092), and then once all brokers are up we also poll until there are no longer any under-replicated partitions before performing further maintenance. Carl On Fri, Jul 21, 2017 at 10:50 AM, Chris Nealwrote: > Thanks Carl. > > Always fun to do this stuff in production... ;) > > Appreciate the input. I'll try a full cycle and see how that works. > > In your opinion, if I stop all brokers and all Zookeeper nodes, then > restart all Zookeepers...at that point can I start both brokers at the same > time, or should I let one broker fully start and read all the unflushed > segments from disk before starting the second broker? > > Again, many thanks. > Chris > > > On Fri, Jul 21, 2017 at 12:13 PM, Carl Haferd > > wrote: > > > I have encountered similar difficulties in a test environment and it may > be > > necessary to stop the Kafka process on each broker and take Zookeeper > > offline before removing the files and zookeeper paths. Otherwise there > may > > be a race condition between brokers which could cause the cluster to > retain > > information for the topic. > > > > Carl > > > > On Fri, Jul 21, 2017 at 9:06 AM, Chris Neal wrote: > > > > > Welp. Surprisingly, that did not fix the problem. :( > > > > > > I cleaned out all the entries for these topics from /config/topics, and > > > removed the logs from the file system for those topics, and the > messages > > > are still flying by in the server.log file. > > > > > > Also, more concerning, when I was looking through the log files for the > > > other broker in the cluster, I noticed the same type of message for a > > topic > > > that should actually be there: > > > > > > [2017-07-21 16:03:29,140] ERROR Conditional update of path > > > /brokers/topics/perf_dstorage_raw/partitions/4/state with data > > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > > epoch":0,"isr":[0]} > > > and expected version 0 failed due to > > > org.apache.zookeeper.KeeperException$BadVersionException: > > KeeperErrorCode > > > = > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state > > > (kafka.utils.ZkUtils$) > > > [2017-07-21 16:03:29,142] ERROR Conditional update of path > > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > > epoch":0,"isr":[0]} > > > and expected version 0 failed due to > > > org.apache.zookeeper.KeeperException$BadVersionException: > > KeeperErrorCode > > > = > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > > > (kafka.utils.ZkUtils$) > > > [2017-07-21 16:03:29,142] ERROR Conditional update of path > > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > > epoch":0,"isr":[0]} > > > and expected version 0 failed due to > > > org.apache.zookeeper.KeeperException$BadVersionException: > > KeeperErrorCode > > > = > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > > > (kafka.utils.ZkUtils$) > > > > > > So, the issue is not isolated to just these "should-have-been-removed" > > > topics, unfortunately. > > > > > > Really appreciate the input so far everyone. Still looking though for > a > > > solution. Many thanks. :) > > > > > > Chris > > > > > > On Fri, Jul 21, 2017 at 10:58 AM, M. Manna wrote: > > > > > > > Just to add (in case the platoform is Windows) > > > > > > > > For Windows based cluster implementation, log/topic cleanup doesn't > > work > > > > out of the box. Users are more or less aware of it, and doing their > own > > > > maintenance as workaround. > > > > If you have issues on Topic deletion not working properly on Windows > > > (i.e. > > > > with topic deletion enable and all other settings). then you have to > > > > manually delete the files. > > > > > > > > > > > > On 21 July 2017 at 16:53, Chris Neal wrote: > > > > > > > > > @Carl, > > > > > > > > > > There is nothing under /admin/delete_topics other than > > > > > > > > > > [] > > > > > > > > > > And nothing under /admin other than delete_topics :) > > > > > > > > > > The topics DO exist, however, under /config/topics! We may be on > to > > > > > something. I will remove them here and see if that clears it up. > > > > > > > > > > Thanks so much for all the help! > > > > > Chris > > > > > > > > > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal > > wrote: > > > > > > > > > > > Thanks again for the replies. VERY much appreciated. I'll check > > > both > > > > > > /admin/delete_topics and /config/topics. > > > > > > > > > > > > Chris > > > > > > > > > > > > On Thu, Jul 20,
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
Thanks Carl. Always fun to do this stuff in production... ;) Appreciate the input. I'll try a full cycle and see how that works. In your opinion, if I stop all brokers and all Zookeeper nodes, then restart all Zookeepers...at that point can I start both brokers at the same time, or should I let one broker fully start and read all the unflushed segments from disk before starting the second broker? Again, many thanks. Chris On Fri, Jul 21, 2017 at 12:13 PM, Carl Haferdwrote: > I have encountered similar difficulties in a test environment and it may be > necessary to stop the Kafka process on each broker and take Zookeeper > offline before removing the files and zookeeper paths. Otherwise there may > be a race condition between brokers which could cause the cluster to retain > information for the topic. > > Carl > > On Fri, Jul 21, 2017 at 9:06 AM, Chris Neal wrote: > > > Welp. Surprisingly, that did not fix the problem. :( > > > > I cleaned out all the entries for these topics from /config/topics, and > > removed the logs from the file system for those topics, and the messages > > are still flying by in the server.log file. > > > > Also, more concerning, when I was looking through the log files for the > > other broker in the cluster, I noticed the same type of message for a > topic > > that should actually be there: > > > > [2017-07-21 16:03:29,140] ERROR Conditional update of path > > /brokers/topics/perf_dstorage_raw/partitions/4/state with data > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > epoch":0,"isr":[0]} > > and expected version 0 failed due to > > org.apache.zookeeper.KeeperException$BadVersionException: > KeeperErrorCode > > = > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state > > (kafka.utils.ZkUtils$) > > [2017-07-21 16:03:29,142] ERROR Conditional update of path > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > epoch":0,"isr":[0]} > > and expected version 0 failed due to > > org.apache.zookeeper.KeeperException$BadVersionException: > KeeperErrorCode > > = > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > > (kafka.utils.ZkUtils$) > > [2017-07-21 16:03:29,142] ERROR Conditional update of path > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > > {"controller_epoch":34,"leader":0,"version":1,"leader_ > epoch":0,"isr":[0]} > > and expected version 0 failed due to > > org.apache.zookeeper.KeeperException$BadVersionException: > KeeperErrorCode > > = > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > > (kafka.utils.ZkUtils$) > > > > So, the issue is not isolated to just these "should-have-been-removed" > > topics, unfortunately. > > > > Really appreciate the input so far everyone. Still looking though for a > > solution. Many thanks. :) > > > > Chris > > > > On Fri, Jul 21, 2017 at 10:58 AM, M. Manna wrote: > > > > > Just to add (in case the platoform is Windows) > > > > > > For Windows based cluster implementation, log/topic cleanup doesn't > work > > > out of the box. Users are more or less aware of it, and doing their own > > > maintenance as workaround. > > > If you have issues on Topic deletion not working properly on Windows > > (i.e. > > > with topic deletion enable and all other settings). then you have to > > > manually delete the files. > > > > > > > > > On 21 July 2017 at 16:53, Chris Neal wrote: > > > > > > > @Carl, > > > > > > > > There is nothing under /admin/delete_topics other than > > > > > > > > [] > > > > > > > > And nothing under /admin other than delete_topics :) > > > > > > > > The topics DO exist, however, under /config/topics! We may be on to > > > > something. I will remove them here and see if that clears it up. > > > > > > > > Thanks so much for all the help! > > > > Chris > > > > > > > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal > wrote: > > > > > > > > > Thanks again for the replies. VERY much appreciated. I'll check > > both > > > > > /admin/delete_topics and /config/topics. > > > > > > > > > > Chris > > > > > > > > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd > > > > > > > > > > > > wrote: > > > > > > > > > >> If delete normally works, there would hopefully be some log > entries > > > when > > > > >> it > > > > >> fails. Are there any unusual zookeeper entries in the > > > > >> /admin/delete_topics > > > > >> path or in the other /admin folders? > > > > >> > > > > >> Does the topic name still exist in zookeeper under /config/topics? > > If > > > > so, > > > > >> that should probably deleted as well. > > > > >> > > > > >> Carl > > > > >> > > > > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal > > wrote: > > > > >> > > > > >> > Delete is definitely there. The delete worked fine, based on > the > > > fact > > > > >>
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
I have encountered similar difficulties in a test environment and it may be necessary to stop the Kafka process on each broker and take Zookeeper offline before removing the files and zookeeper paths. Otherwise there may be a race condition between brokers which could cause the cluster to retain information for the topic. Carl On Fri, Jul 21, 2017 at 9:06 AM, Chris Nealwrote: > Welp. Surprisingly, that did not fix the problem. :( > > I cleaned out all the entries for these topics from /config/topics, and > removed the logs from the file system for those topics, and the messages > are still flying by in the server.log file. > > Also, more concerning, when I was looking through the log files for the > other broker in the cluster, I noticed the same type of message for a topic > that should actually be there: > > [2017-07-21 16:03:29,140] ERROR Conditional update of path > /brokers/topics/perf_dstorage_raw/partitions/4/state with data > {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]} > and expected version 0 failed due to > org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode > = > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state > (kafka.utils.ZkUtils$) > [2017-07-21 16:03:29,142] ERROR Conditional update of path > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]} > and expected version 0 failed due to > org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode > = > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > (kafka.utils.ZkUtils$) > [2017-07-21 16:03:29,142] ERROR Conditional update of path > /brokers/topics/perf_dstorage_raw/partitions/0/state with data > {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]} > and expected version 0 failed due to > org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode > = > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state > (kafka.utils.ZkUtils$) > > So, the issue is not isolated to just these "should-have-been-removed" > topics, unfortunately. > > Really appreciate the input so far everyone. Still looking though for a > solution. Many thanks. :) > > Chris > > On Fri, Jul 21, 2017 at 10:58 AM, M. Manna wrote: > > > Just to add (in case the platoform is Windows) > > > > For Windows based cluster implementation, log/topic cleanup doesn't work > > out of the box. Users are more or less aware of it, and doing their own > > maintenance as workaround. > > If you have issues on Topic deletion not working properly on Windows > (i.e. > > with topic deletion enable and all other settings). then you have to > > manually delete the files. > > > > > > On 21 July 2017 at 16:53, Chris Neal wrote: > > > > > @Carl, > > > > > > There is nothing under /admin/delete_topics other than > > > > > > [] > > > > > > And nothing under /admin other than delete_topics :) > > > > > > The topics DO exist, however, under /config/topics! We may be on to > > > something. I will remove them here and see if that clears it up. > > > > > > Thanks so much for all the help! > > > Chris > > > > > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal wrote: > > > > > > > Thanks again for the replies. VERY much appreciated. I'll check > both > > > > /admin/delete_topics and /config/topics. > > > > > > > > Chris > > > > > > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd > > > > > > > > > wrote: > > > > > > > >> If delete normally works, there would hopefully be some log entries > > when > > > >> it > > > >> fails. Are there any unusual zookeeper entries in the > > > >> /admin/delete_topics > > > >> path or in the other /admin folders? > > > >> > > > >> Does the topic name still exist in zookeeper under /config/topics? > If > > > so, > > > >> that should probably deleted as well. > > > >> > > > >> Carl > > > >> > > > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal > wrote: > > > >> > > > >> > Delete is definitely there. The delete worked fine, based on the > > fact > > > >> that > > > >> > there is nothing in Zookeeper, and that the controller reported > that > > > the > > > >> > delete was successful, it's just something seems to have gotten > out > > of > > > >> > sync. > > > >> > > > > >> > delete.topic.enabled is true. I've successfully deleted topics in > > the > > > >> > past, so I know it *should* work. :) > > > >> > > > > >> > I also had already checked in Zookeeper, and there is no directory > > for > > > >> the > > > >> > topics under /brokers/topics Very strange indeed. > > > >> > > > > >> > If I just remove the log directories from the filesystem, is that > > > >> enough to > > > >> > get the broker to stop asking about the topics? I would guess > there > > > >> would > > > >> > need to be more than just that,
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
Welp. Surprisingly, that did not fix the problem. :( I cleaned out all the entries for these topics from /config/topics, and removed the logs from the file system for those topics, and the messages are still flying by in the server.log file. Also, more concerning, when I was looking through the log files for the other broker in the cluster, I noticed the same type of message for a topic that should actually be there: [2017-07-21 16:03:29,140] ERROR Conditional update of path /brokers/topics/perf_dstorage_raw/partitions/4/state with data {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]} and expected version 0 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state (kafka.utils.ZkUtils$) [2017-07-21 16:03:29,142] ERROR Conditional update of path /brokers/topics/perf_dstorage_raw/partitions/0/state with data {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]} and expected version 0 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state (kafka.utils.ZkUtils$) [2017-07-21 16:03:29,142] ERROR Conditional update of path /brokers/topics/perf_dstorage_raw/partitions/0/state with data {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]} and expected version 0 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state (kafka.utils.ZkUtils$) So, the issue is not isolated to just these "should-have-been-removed" topics, unfortunately. Really appreciate the input so far everyone. Still looking though for a solution. Many thanks. :) Chris On Fri, Jul 21, 2017 at 10:58 AM, M. Mannawrote: > Just to add (in case the platoform is Windows) > > For Windows based cluster implementation, log/topic cleanup doesn't work > out of the box. Users are more or less aware of it, and doing their own > maintenance as workaround. > If you have issues on Topic deletion not working properly on Windows (i.e. > with topic deletion enable and all other settings). then you have to > manually delete the files. > > > On 21 July 2017 at 16:53, Chris Neal wrote: > > > @Carl, > > > > There is nothing under /admin/delete_topics other than > > > > [] > > > > And nothing under /admin other than delete_topics :) > > > > The topics DO exist, however, under /config/topics! We may be on to > > something. I will remove them here and see if that clears it up. > > > > Thanks so much for all the help! > > Chris > > > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal wrote: > > > > > Thanks again for the replies. VERY much appreciated. I'll check both > > > /admin/delete_topics and /config/topics. > > > > > > Chris > > > > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd > > > > > > wrote: > > > > > >> If delete normally works, there would hopefully be some log entries > when > > >> it > > >> fails. Are there any unusual zookeeper entries in the > > >> /admin/delete_topics > > >> path or in the other /admin folders? > > >> > > >> Does the topic name still exist in zookeeper under /config/topics? If > > so, > > >> that should probably deleted as well. > > >> > > >> Carl > > >> > > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal wrote: > > >> > > >> > Delete is definitely there. The delete worked fine, based on the > fact > > >> that > > >> > there is nothing in Zookeeper, and that the controller reported that > > the > > >> > delete was successful, it's just something seems to have gotten out > of > > >> > sync. > > >> > > > >> > delete.topic.enabled is true. I've successfully deleted topics in > the > > >> > past, so I know it *should* work. :) > > >> > > > >> > I also had already checked in Zookeeper, and there is no directory > for > > >> the > > >> > topics under /brokers/topics Very strange indeed. > > >> > > > >> > If I just remove the log directories from the filesystem, is that > > >> enough to > > >> > get the broker to stop asking about the topics? I would guess there > > >> would > > >> > need to be more than just that, but I could be wrong. > > >> > > > >> > Thanks guys for the suggestions though! > > >> > > > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis < > spo...@salesforce.com > > > > > >> > wrote: > > >> > > > >> > > I could be totally wrong, but I seem to recall that delete wasn't > > >> fully > > >> > > implemented in 0.8.x? > > >> > > > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd > > >> > > >> > > > > > >> > > wrote: > > >> > > > > >> > > > Chris, > > >> > > > > > >> > > > You could first check to make sure that delete.topic.enable is > > true > > >> and > > >> > > try > > >> > > > deleting again if
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
Just to add (in case the platoform is Windows) For Windows based cluster implementation, log/topic cleanup doesn't work out of the box. Users are more or less aware of it, and doing their own maintenance as workaround. If you have issues on Topic deletion not working properly on Windows (i.e. with topic deletion enable and all other settings). then you have to manually delete the files. On 21 July 2017 at 16:53, Chris Nealwrote: > @Carl, > > There is nothing under /admin/delete_topics other than > > [] > > And nothing under /admin other than delete_topics :) > > The topics DO exist, however, under /config/topics! We may be on to > something. I will remove them here and see if that clears it up. > > Thanks so much for all the help! > Chris > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal wrote: > > > Thanks again for the replies. VERY much appreciated. I'll check both > > /admin/delete_topics and /config/topics. > > > > Chris > > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd > > > wrote: > > > >> If delete normally works, there would hopefully be some log entries when > >> it > >> fails. Are there any unusual zookeeper entries in the > >> /admin/delete_topics > >> path or in the other /admin folders? > >> > >> Does the topic name still exist in zookeeper under /config/topics? If > so, > >> that should probably deleted as well. > >> > >> Carl > >> > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal wrote: > >> > >> > Delete is definitely there. The delete worked fine, based on the fact > >> that > >> > there is nothing in Zookeeper, and that the controller reported that > the > >> > delete was successful, it's just something seems to have gotten out of > >> > sync. > >> > > >> > delete.topic.enabled is true. I've successfully deleted topics in the > >> > past, so I know it *should* work. :) > >> > > >> > I also had already checked in Zookeeper, and there is no directory for > >> the > >> > topics under /brokers/topics Very strange indeed. > >> > > >> > If I just remove the log directories from the filesystem, is that > >> enough to > >> > get the broker to stop asking about the topics? I would guess there > >> would > >> > need to be more than just that, but I could be wrong. > >> > > >> > Thanks guys for the suggestions though! > >> > > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis > > >> > wrote: > >> > > >> > > I could be totally wrong, but I seem to recall that delete wasn't > >> fully > >> > > implemented in 0.8.x? > >> > > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd > >> > >> > > > > >> > > wrote: > >> > > > >> > > > Chris, > >> > > > > >> > > > You could first check to make sure that delete.topic.enable is > true > >> and > >> > > try > >> > > > deleting again if not. If that doesn't work with 0.8.1.1 you > might > >> > need > >> > > to > >> > > > manually remove the topic's log files from the configured log.dirs > >> > folder > >> > > > on each broker in addition to removing the topic's zookeeper path. > >> > > > > >> > > > Carl > >> > > > > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal > >> wrote: > >> > > > > >> > > > > Hi all, > >> > > > > > >> > > > > I have a weird situation here. I have deleted a few topics on > my > >> > > 0.8.1.1 > >> > > > > cluster (old, I know...). The deletes succeeded according to > the > >> > > > > controller.log: > >> > > > > > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on > Controller > >> 1]: > >> > > New > >> > > > > topics: [Set()], deleted topics: > >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition > >> replica > >> > > > > assignment [Map()] > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on > Controller > >> 1]: > >> > > New > >> > > > > topics: [Set()], deleted topics: > >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition > >> > replica > >> > > > > assignment [Map()] > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on > Controller > >> 1]: > >> > > New > >> > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], > >> new > >> > > > > partition replica assignment [Map()] > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on > Controller > >> 1]: > >> > > New > >> > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)] > , > >> new > >> > > > > partition replica assignment [Map()] > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > >> > > > > > >> > > > > I query Zookeeper and the path is not there under > /brokers/topics > >> as > >> > > > well. > >> > > > > > >> > > > >
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
@Carl, There is nothing under /admin/delete_topics other than [] And nothing under /admin other than delete_topics :) The topics DO exist, however, under /config/topics! We may be on to something. I will remove them here and see if that clears it up. Thanks so much for all the help! Chris On Thu, Jul 20, 2017 at 10:37 PM, Chris Nealwrote: > Thanks again for the replies. VERY much appreciated. I'll check both > /admin/delete_topics and /config/topics. > > Chris > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd > wrote: > >> If delete normally works, there would hopefully be some log entries when >> it >> fails. Are there any unusual zookeeper entries in the >> /admin/delete_topics >> path or in the other /admin folders? >> >> Does the topic name still exist in zookeeper under /config/topics? If so, >> that should probably deleted as well. >> >> Carl >> >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal wrote: >> >> > Delete is definitely there. The delete worked fine, based on the fact >> that >> > there is nothing in Zookeeper, and that the controller reported that the >> > delete was successful, it's just something seems to have gotten out of >> > sync. >> > >> > delete.topic.enabled is true. I've successfully deleted topics in the >> > past, so I know it *should* work. :) >> > >> > I also had already checked in Zookeeper, and there is no directory for >> the >> > topics under /brokers/topics Very strange indeed. >> > >> > If I just remove the log directories from the filesystem, is that >> enough to >> > get the broker to stop asking about the topics? I would guess there >> would >> > need to be more than just that, but I could be wrong. >> > >> > Thanks guys for the suggestions though! >> > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis >> > wrote: >> > >> > > I could be totally wrong, but I seem to recall that delete wasn't >> fully >> > > implemented in 0.8.x? >> > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd >> > > > > > >> > > wrote: >> > > >> > > > Chris, >> > > > >> > > > You could first check to make sure that delete.topic.enable is true >> and >> > > try >> > > > deleting again if not. If that doesn't work with 0.8.1.1 you might >> > need >> > > to >> > > > manually remove the topic's log files from the configured log.dirs >> > folder >> > > > on each broker in addition to removing the topic's zookeeper path. >> > > > >> > > > Carl >> > > > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal >> wrote: >> > > > >> > > > > Hi all, >> > > > > >> > > > > I have a weird situation here. I have deleted a few topics on my >> > > 0.8.1.1 >> > > > > cluster (old, I know...). The deletes succeeded according to the >> > > > > controller.log: >> > > > > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition >> replica >> > > > > assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition >> > replica >> > > > > assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], >> new >> > > > > partition replica assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], >> new >> > > > > partition replica assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > >> > > > > I query Zookeeper and the path is not there under /brokers/topics >> as >> > > > well. >> > > > > >> > > > > But, one of the nodes in my cluster continues to try and use them: >> > > > > >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ >> > > > partitions/3/state >> > > > > with data >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ >> > > > > epoch":2,"isr":[1,0]} >> > > > > and expected version 69 failed due to >> > > > > org.apache.zookeeper.KeeperException$NoNodeException: >> > KeeperErrorCode >> > > = >> > > > > NoNode for >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ >> > > > partitions/3/state >> > > > > (kafka.utils.ZkUtils$) >> > > > > [2017-07-20 17:04:36,723] INFO Partition >> > > > >
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
Thanks again for the replies. VERY much appreciated. I'll check both /admin/delete_topics and /config/topics. Chris On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferdwrote: > If delete normally works, there would hopefully be some log entries when it > fails. Are there any unusual zookeeper entries in the /admin/delete_topics > path or in the other /admin folders? > > Does the topic name still exist in zookeeper under /config/topics? If so, > that should probably deleted as well. > > Carl > > On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal wrote: > > > Delete is definitely there. The delete worked fine, based on the fact > that > > there is nothing in Zookeeper, and that the controller reported that the > > delete was successful, it's just something seems to have gotten out of > > sync. > > > > delete.topic.enabled is true. I've successfully deleted topics in the > > past, so I know it *should* work. :) > > > > I also had already checked in Zookeeper, and there is no directory for > the > > topics under /brokers/topics Very strange indeed. > > > > If I just remove the log directories from the filesystem, is that enough > to > > get the broker to stop asking about the topics? I would guess there > would > > need to be more than just that, but I could be wrong. > > > > Thanks guys for the suggestions though! > > > > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis > > wrote: > > > > > I could be totally wrong, but I seem to recall that delete wasn't fully > > > implemented in 0.8.x? > > > > > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd > > > > > > > > wrote: > > > > > > > Chris, > > > > > > > > You could first check to make sure that delete.topic.enable is true > and > > > try > > > > deleting again if not. If that doesn't work with 0.8.1.1 you might > > need > > > to > > > > manually remove the topic's log files from the configured log.dirs > > folder > > > > on each broker in addition to removing the topic's zookeeper path. > > > > > > > > Carl > > > > > > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I have a weird situation here. I have deleted a few topics on my > > > 0.8.1.1 > > > > > cluster (old, I know...). The deletes succeeded according to the > > > > > controller.log: > > > > > > > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller > 1]: > > > New > > > > > topics: [Set()], deleted topics: > > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition > replica > > > > > assignment [Map()] > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller > 1]: > > > New > > > > > topics: [Set()], deleted topics: > > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition > > replica > > > > > assignment [Map()] > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller > 1]: > > > New > > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new > > > > > partition replica assignment [Map()] > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller > 1]: > > > New > > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], > new > > > > > partition replica assignment [Map()] > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > > > > > > > I query Zookeeper and the path is not there under /brokers/topics > as > > > > well. > > > > > > > > > > But, one of the nodes in my cluster continues to try and use them: > > > > > > > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path > > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > > > > partitions/3/state > > > > > with data > > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > > > > > epoch":2,"isr":[1,0]} > > > > > and expected version 69 failed due to > > > > > org.apache.zookeeper.KeeperException$NoNodeException: > > KeeperErrorCode > > > = > > > > > NoNode for > > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > > > > partitions/3/state > > > > > (kafka.utils.ZkUtils$) > > > > > [2017-07-20 17:04:36,723] INFO Partition > > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > > > > zkVersion > > > > > [69] not equal to that in zookeeper, skip updating ISR > > > > > (kafka.cluster.Partition) > > > > > [2017-07-20 17:04:36,723] INFO Partition > > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > > > > zkVersion > > > > > [69] not equal to that in zookeeper, skip updating ISR > > > > > (kafka.cluster.Partition) > > > > > [2017-07-20 17:04:36,764] INFO Partition > [perf_goplatform-uat_raw,2] > > on > > > > > broker 1:
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
If delete normally works, there would hopefully be some log entries when it fails. Are there any unusual zookeeper entries in the /admin/delete_topics path or in the other /admin folders? Does the topic name still exist in zookeeper under /config/topics? If so, that should probably deleted as well. Carl On Thu, Jul 20, 2017 at 6:42 PM, Chris Nealwrote: > Delete is definitely there. The delete worked fine, based on the fact that > there is nothing in Zookeeper, and that the controller reported that the > delete was successful, it's just something seems to have gotten out of > sync. > > delete.topic.enabled is true. I've successfully deleted topics in the > past, so I know it *should* work. :) > > I also had already checked in Zookeeper, and there is no directory for the > topics under /brokers/topics Very strange indeed. > > If I just remove the log directories from the filesystem, is that enough to > get the broker to stop asking about the topics? I would guess there would > need to be more than just that, but I could be wrong. > > Thanks guys for the suggestions though! > > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis > wrote: > > > I could be totally wrong, but I seem to recall that delete wasn't fully > > implemented in 0.8.x? > > > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd > > > > > wrote: > > > > > Chris, > > > > > > You could first check to make sure that delete.topic.enable is true and > > try > > > deleting again if not. If that doesn't work with 0.8.1.1 you might > need > > to > > > manually remove the topic's log files from the configured log.dirs > folder > > > on each broker in addition to removing the topic's zookeeper path. > > > > > > Carl > > > > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal wrote: > > > > > > > Hi all, > > > > > > > > I have a weird situation here. I have deleted a few topics on my > > 0.8.1.1 > > > > cluster (old, I know...). The deletes succeeded according to the > > > > controller.log: > > > > > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: > > New > > > > topics: [Set()], deleted topics: > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica > > > > assignment [Map()] > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: > > New > > > > topics: [Set()], deleted topics: > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition > replica > > > > assignment [Map()] > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: > > New > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new > > > > partition replica assignment [Map()] > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: > > New > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new > > > > partition replica assignment [Map()] > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > > > > > I query Zookeeper and the path is not there under /brokers/topics as > > > well. > > > > > > > > But, one of the nodes in my cluster continues to try and use them: > > > > > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > > > partitions/3/state > > > > with data > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > > > > epoch":2,"isr":[1,0]} > > > > and expected version 69 failed due to > > > > org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode > > = > > > > NoNode for > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > > > partitions/3/state > > > > (kafka.utils.ZkUtils$) > > > > [2017-07-20 17:04:36,723] INFO Partition > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > > > zkVersion > > > > [69] not equal to that in zookeeper, skip updating ISR > > > > (kafka.cluster.Partition) > > > > [2017-07-20 17:04:36,723] INFO Partition > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > > > zkVersion > > > > [69] not equal to that in zookeeper, skip updating ISR > > > > (kafka.cluster.Partition) > > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] > on > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] > from > > > 1,0 > > > > to 1 (kafka.cluster.Partition) > > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] > on > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] > from > > > 1,0 > > > > to 1 (kafka.cluster.Partition) > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
Delete is definitely there. The delete worked fine, based on the fact that there is nothing in Zookeeper, and that the controller reported that the delete was successful, it's just something seems to have gotten out of sync. delete.topic.enabled is true. I've successfully deleted topics in the past, so I know it *should* work. :) I also had already checked in Zookeeper, and there is no directory for the topics under /brokers/topics Very strange indeed. If I just remove the log directories from the filesystem, is that enough to get the broker to stop asking about the topics? I would guess there would need to be more than just that, but I could be wrong. Thanks guys for the suggestions though! On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powiswrote: > I could be totally wrong, but I seem to recall that delete wasn't fully > implemented in 0.8.x? > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd > > wrote: > > > Chris, > > > > You could first check to make sure that delete.topic.enable is true and > try > > deleting again if not. If that doesn't work with 0.8.1.1 you might need > to > > manually remove the topic's log files from the configured log.dirs folder > > on each broker in addition to removing the topic's zookeeper path. > > > > Carl > > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal wrote: > > > > > Hi all, > > > > > > I have a weird situation here. I have deleted a few topics on my > 0.8.1.1 > > > cluster (old, I know...). The deletes succeeded according to the > > > controller.log: > > > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: > New > > > topics: [Set()], deleted topics: > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica > > > assignment [Map()] > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: > New > > > topics: [Set()], deleted topics: > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica > > > assignment [Map()] > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: > New > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new > > > partition replica assignment [Map()] > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: > New > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new > > > partition replica assignment [Map()] > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > > > I query Zookeeper and the path is not there under /brokers/topics as > > well. > > > > > > But, one of the nodes in my cluster continues to try and use them: > > > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > > partitions/3/state > > > with data > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > > > epoch":2,"isr":[1,0]} > > > and expected version 69 failed due to > > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode > = > > > NoNode for > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > > partitions/3/state > > > (kafka.utils.ZkUtils$) > > > [2017-07-20 17:04:36,723] INFO Partition > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > > zkVersion > > > [69] not equal to that in zookeeper, skip updating ISR > > > (kafka.cluster.Partition) > > > [2017-07-20 17:04:36,723] INFO Partition > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > > zkVersion > > > [69] not equal to that in zookeeper, skip updating ISR > > > (kafka.cluster.Partition) > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from > > 1,0 > > > to 1 (kafka.cluster.Partition) > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from > > 1,0 > > > to 1 (kafka.cluster.Partition) > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > > epoch":2,"isr":[1]} > > > and expected version 70 failed due to > > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode > = > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state > > > (kafka.utils.ZkUtils$) > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > > epoch":2,"isr":[1]} > > > and expected version 70 failed due to > > >
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
I could be totally wrong, but I seem to recall that delete wasn't fully implemented in 0.8.x? On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferdwrote: > Chris, > > You could first check to make sure that delete.topic.enable is true and try > deleting again if not. If that doesn't work with 0.8.1.1 you might need to > manually remove the topic's log files from the configured log.dirs folder > on each broker in addition to removing the topic's zookeeper path. > > Carl > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal wrote: > > > Hi all, > > > > I have a weird situation here. I have deleted a few topics on my 0.8.1.1 > > cluster (old, I know...). The deletes succeeded according to the > > controller.log: > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New > > topics: [Set()], deleted topics: > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica > > assignment [Map()] > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: New > > topics: [Set()], deleted topics: > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica > > assignment [Map()] > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: New > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new > > partition replica assignment [Map()] > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: New > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new > > partition replica assignment [Map()] > > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > > > I query Zookeeper and the path is not there under /brokers/topics as > well. > > > > But, one of the nodes in my cluster continues to try and use them: > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > partitions/3/state > > with data > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > > epoch":2,"isr":[1,0]} > > and expected version 69 failed due to > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > > NoNode for > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ > partitions/3/state > > (kafka.utils.ZkUtils$) > > [2017-07-20 17:04:36,723] INFO Partition > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > zkVersion > > [69] not equal to that in zookeeper, skip updating ISR > > (kafka.cluster.Partition) > > [2017-07-20 17:04:36,723] INFO Partition > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached > zkVersion > > [69] not equal to that in zookeeper, skip updating ISR > > (kafka.cluster.Partition) > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from > 1,0 > > to 1 (kafka.cluster.Partition) > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from > 1,0 > > to 1 (kafka.cluster.Partition) > > [2017-07-20 17:04:36,765] ERROR Conditional update of path > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > epoch":2,"isr":[1]} > > and expected version 70 failed due to > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state > > (kafka.utils.ZkUtils$) > > [2017-07-20 17:04:36,765] ERROR Conditional update of path > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data > > {"controller_epoch":34,"leader":1,"version":1,"leader_ > epoch":2,"isr":[1]} > > and expected version 70 failed due to > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state > > (kafka.utils.ZkUtils$) > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip > > updating ISR (kafka.cluster.Partition) > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip > > updating ISR (kafka.cluster.Partition) > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0 > > to 1 (kafka.cluster.Partition) > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0 > > to 1 (kafka.cluster.Partition) > > [2017-07-20 17:04:36,988] ERROR Conditional update of path > >
Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper
Chris, You could first check to make sure that delete.topic.enable is true and try deleting again if not. If that doesn't work with 0.8.1.1 you might need to manually remove the topic's log files from the configured log.dirs folder on each broker in addition to removing the topic's zookeeper path. Carl On Thu, Jul 20, 2017 at 10:06 AM, Chris Nealwrote: > Hi all, > > I have a weird situation here. I have deleted a few topics on my 0.8.1.1 > cluster (old, I know...). The deletes succeeded according to the > controller.log: > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New > topics: [Set()], deleted topics: > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica > assignment [Map()] > (kafka.controller.PartitionStateMachine$TopicChangeListener) > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: New > topics: [Set()], deleted topics: > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica > assignment [Map()] > (kafka.controller.PartitionStateMachine$TopicChangeListener) > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: New > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new > partition replica assignment [Map()] > (kafka.controller.PartitionStateMachine$TopicChangeListener) > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: New > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new > partition replica assignment [Map()] > (kafka.controller.PartitionStateMachine$TopicChangeListener) > > I query Zookeeper and the path is not there under /brokers/topics as well. > > But, one of the nodes in my cluster continues to try and use them: > > [2017-07-20 17:04:36,723] ERROR Conditional update of path > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state > with data > {"controller_epoch":34,"leader":1,"version":1,"leader_ > epoch":2,"isr":[1,0]} > and expected version 69 failed due to > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state > (kafka.utils.ZkUtils$) > [2017-07-20 17:04:36,723] INFO Partition > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion > [69] not equal to that in zookeeper, skip updating ISR > (kafka.cluster.Partition) > [2017-07-20 17:04:36,723] INFO Partition > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion > [69] not equal to that in zookeeper, skip updating ISR > (kafka.cluster.Partition) > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0 > to 1 (kafka.cluster.Partition) > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0 > to 1 (kafka.cluster.Partition) > [2017-07-20 17:04:36,765] ERROR Conditional update of path > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data > {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]} > and expected version 70 failed due to > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state > (kafka.utils.ZkUtils$) > [2017-07-20 17:04:36,765] ERROR Conditional update of path > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data > {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]} > and expected version 70 failed due to > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state > (kafka.utils.ZkUtils$) > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip > updating ISR (kafka.cluster.Partition) > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip > updating ISR (kafka.cluster.Partition) > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0 > to 1 (kafka.cluster.Partition) > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0 > to 1 (kafka.cluster.Partition) > [2017-07-20 17:04:36,988] ERROR Conditional update of path > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data > {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]} > and expected version 90 failed due to > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state > (kafka.utils.ZkUtils$) > [2017-07-20 17:04:36,988] ERROR