Chris,
You could first check to make sure that delete.topic.enable is true and try
deleting again if not. If that doesn't work with 0.8.1.1 you might need to
manually remove the topic's log files from the configured log.dirs folder
on each broker in addition to removing the topic's zookeeper path.
Carl
On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cwn...@gmail.com> wrote:
> Hi all,
>
> I have a weird situation here. I have deleted a few topics on my 0.8.1.1
> cluster (old, I know...). The deletes succeeded according to the
> controller.log:
>
> [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics:
> [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
> assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics:
> [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica
> assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
> partition replica assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
> partition replica assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
>
> I query Zookeeper and the path is not there under /brokers/topics as well.
>
> But, one of the nodes in my cluster continues to try and use them:
>
> [2017-07-20 17:04:36,723] ERROR Conditional update of path
> /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
> with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_
> epoch":2,"isr":[1,0]}
> and expected version 69 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
> /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,723] INFO Partition
> [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
> [69] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition)
> [2017-07-20 17:04:36,723] INFO Partition
> [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
> [69] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition)
> [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,765] ERROR Conditional update of path
> /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
> and expected version 70 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,765] ERROR Conditional update of path
> /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
> and expected version 70 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
> [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
> [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,988] ERROR Conditional update of path
> /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
> and expected version 90 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,988] ERROR Conditional update of path
> /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
> and expected version 90 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
> [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
>
> I've tried a rolling restart of the cluster to see if that fixed it, but it
> did not.
>
> Can someone please help me out here? I'm not sure how I can get things
> back in sync.
>
> Thank you so much for your time.
> Chris
>
