If delete normally works, there would hopefully be some log entries when it
fails. Are there any unusual zookeeper entries in the /admin/delete_topics
path or in the other /admin folders?
Does the topic name still exist in zookeeper under /config/topics? If so,
that should probably deleted as well.
Carl
On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cwn...@gmail.com> wrote:
> Delete is definitely there. The delete worked fine, based on the fact that
> there is nothing in Zookeeper, and that the controller reported that the
> delete was successful, it's just something seems to have gotten out of
> sync.
>
> delete.topic.enabled is true. I've successfully deleted topics in the
> past, so I know it *should* work. :)
>
> I also had already checked in Zookeeper, and there is no directory for the
> topics under /brokers/topics.... Very strange indeed.
>
> If I just remove the log directories from the filesystem, is that enough to
> get the broker to stop asking about the topics? I would guess there would
> need to be more than just that, but I could be wrong.
>
> Thanks guys for the suggestions though!
>
> On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <spo...@salesforce.com>
> wrote:
>
> > I could be totally wrong, but I seem to recall that delete wasn't fully
> > implemented in 0.8.x?
> >
> > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> <chaf...@groupon.com.invalid
> > >
> > wrote:
> >
> > > Chris,
> > >
> > > You could first check to make sure that delete.topic.enable is true and
> > try
> > > deleting again if not. If that doesn't work with 0.8.1.1 you might
> need
> > to
> > > manually remove the topic's log files from the configured log.dirs
> folder
> > > on each broker in addition to removing the topic's zookeeper path.
> > >
> > > Carl
> > >
> > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cwn...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have a weird situation here. I have deleted a few topics on my
> > 0.8.1.1
> > > > cluster (old, I know...). The deletes succeeded according to the
> > > > controller.log:
> > > >
> > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics:
> > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
> > > > assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics:
> > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition
> replica
> > > > assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
> > > > partition replica assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
> > > > partition replica assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > >
> > > > I query Zookeeper and the path is not there under /brokers/topics as
> > > well.
> > > >
> > > > But, one of the nodes in my cluster continues to try and use them:
> > > >
> > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > partitions/3/state
> > > > with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > epoch":2,"isr":[1,0]}
> > > > and expected version 69 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for
> > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > partitions/3/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > > zkVersion
> > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > > zkVersion
> > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
> from
> > > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
> from
> > > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":2,"isr":[1]}
> > > > and expected version 70 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> 2/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":2,"isr":[1]}
> > > > and expected version 70 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> 2/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from
> > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from
> > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":4,"isr":[1]}
> > > > and expected version 90 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":4,"isr":[1]}
> > > > and expected version 90 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > >
> > > > I've tried a rolling restart of the cluster to see if that fixed it,
> > but
> > > it
> > > > did not.
> > > >
> > > > Can someone please help me out here? I'm not sure how I can get
> things
> > > > back in sync.
> > > >
> > > > Thank you so much for your time.
> > > > Chris
> > > >
> > >
> >
>
