Hi all,
I have a weird situation here. I have deleted a few topics on my 0.8.1.1
cluster (old, I know...). The deletes succeeded according to the
controller.log:
[2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics:
[Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)
[2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics:
[Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica
assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)
[2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
partition replica assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)
[2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
partition replica assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)
I query Zookeeper and the path is not there under /brokers/topics as well.
But, one of the nodes in my cluster continues to try and use them:
[2017-07-20 17:04:36,723] ERROR Conditional update of path
/brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1,0]}
and expected version 69 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for
/brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,723] INFO Partition
[perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
[69] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2017-07-20 17:04:36,723] INFO Partition
[perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
[69] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,765] ERROR Conditional update of path
/brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
and expected version 70 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,765] ERROR Conditional update of path
/brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
and expected version 70 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
[2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
[2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,988] ERROR Conditional update of path
/brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
and expected version 90 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,988] ERROR Conditional update of path
/brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
and expected version 90 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
[2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
I've tried a rolling restart of the cluster to see if that fixed it, but it
did not.
Can someone please help me out here? I'm not sure how I can get things
back in sync.
Thank you so much for your time.
Chris
