I am running a single node cluster in KRaft mode on
confluentinc/cp-kafka:7.3.0-1-ubi8
aka kafka 3.3.x
I cannot recreate a topic 'base.m3.ocusma' (but I can create,delete,create
other topics just fine.
the topic is deleted:
ls /opt/kafka/data-0/logs/base.m3.ocusma
ls: cannot access '/opt/kafka/data-0/logs/base.m3.ocusma': No such file or
directory
When I try to create the topic I see the following in the kafka log:
[2022-12-20 09:38:15,882] WARN [Controller 0] createTopics: failed with
unknown server exception NoSuchElementException at epoch 93 in 1311 us.
Renouncing leadership and reverting to the last committed offset 2037058.
(org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:15 am java.util.NoSuchElementException
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:167)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:139)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.timeline.TimelineHashSet$ValueIterator.next(TimelineHashSet.java:120)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.controller.ReplicationControlManager.validateNewTopicNames(ReplicationControlManager.java:799)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.controller.ReplicationControlManager.createTopics(ReplicationControlManager.java:567)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.controller.QuorumController.lambda$createTopics$7(QuorumController.java:1832)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:767)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
Tue, Dec 20 2022 11:38:15 am at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
Tue, Dec 20 2022 11:38:15 am at
java.base/java.lang.Thread.run(Thread.java:829)
Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,884] INFO [RaftManager
nodeId=0] Received user request to resign from the current epoch 93
(org.apache.kafka.raft.KafkaRaftClient)
Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,889] INFO [RaftManager
nodeId=0] Completed transition to ResignedState(localId=0, epoch=93,
voters=[0], electionTimeoutMs=1515, unackedVoters=[],
preferredSuccessors=[]) (org.apache.kafka.raft.QuorumState)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,837] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,892] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,924] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,926] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,927] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,978] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,021] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,021] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,022] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,072] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,109] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,110] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,111] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,164] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,183] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,184] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,184] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,235] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,252] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,253] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,254] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,303] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,324] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,325] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,326] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,378] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,399] ERROR [Controller 0]
processBrokerHeartbeat: unable to start processing because of
NotControllerException. (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,399] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,403] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,410] INFO [RaftManager
nodeId=0] Completed transition to CandidateState(localId=0, epoch=94,
retries=1, electionTimeoutMs=1303) (org.apache.kafka.raft.QuorumState)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,410] INFO [Controller 0]
In the new epoch 94, the leader is (none).
(org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,418] INFO [RaftManager
nodeId=0] Completed transition to Leader(localId=0, epoch=94,
epochStartOffset=2037059, highWatermark=Optional.empty,
voterStates={0=ReplicaState(nodeId=0, endOffset=Optional.empty,
lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1,
hasAcknowledgedLeader=true)}) (org.apache.kafka.raft.QuorumState)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,419] INFO [Controller 0]
Becoming the active controller at epoch 94, committed offset 2037058,
committed epoch 93 (org.apache.kafka.controller.QuorumController)
Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,450] INFO
[BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
controller, from now on will use broker
kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
(kafka.server.BrokerToControllerRequestThread)
What has happened and is there a way of fixing?
regards,
Simon
