Hi Team,
Running kafka & zookeeper in kubernetes cluster.
No of brokers : 3
No of partitions per topic : 3
creating topic with 3 partitions, and looks like all the partitions are up.
Below is the snapshot to confirm the same,
Topic:applestore PartitionCount:3 ReplicationFactor:3 Configs:
Topic: applestore Partition: 0 Leader: 1001 Replicas: 1001,1003,1002
Isr: 1001,1003,1002
Topic: applestore Partition: 1 Leader: 1002 Replicas: 1002,1001,1003
Isr: 1002,1001,1003
Topic: applestore Partition: 2 Leader: 1003 Replicas: 1003,1002,1001
Isr: 1003,1002,1001
But, we see in the brokers as soon as the topics are created below stack traces
appears,
error 1:
[2018-05-28 08:00:31,875] ERROR [ReplicaFetcher replicaId=1001, leaderId=1003,
fetcherId=7] Error for partition applestore-2 to broker
1003:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
error 2 :
[2018-05-28 00:43:20,993] ERROR [ReplicaFetcher replicaId=1003, leaderId=1001,
fetcherId=0] Error for partition apple-6 to broker
1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
When we tries producing records to each specific partition, it works fine and
also log size across the replicated brokers appears to be equal, which means
replication is happening fine.
Attaching the two stack trace files.
Why are these stack traces appearing ? can we ignore these stack traces if its
some spam messages ?
Best regards,
kaushik
[2018-05-28 00:39:46,195] TRACE [Broker id=1003] Cached leader info
PartitionState(controllerEpoch=1, leader=1001, leaderEpoch=5, isr=[1001, 1002,
1003], zkVersion=8, replicas=[1003, 1001, 1002], offlineReplicas=[]) for
partition steve-0 in response to UpdateMetadata request sent by controller 1001
epoch 1 with correlation id 6 (state.change.logger)
[2018-05-28 00:39:46,195] TRACE [Broker id=1003] Cached leader info
PartitionState(controllerEpoch=1, leader=1001, leaderEpoch=3, isr=[1001, 1002,
1003], zkVersion=6, replicas=[1001, 1002, 1003], offlineReplicas=[]) for
partition apple-7 in response to UpdateMetadata request sent by controller 1001
epoch 1 with correlation id 6 (state.change.logger)
[2018-05-28 00:43:20,093] TRACE [Broker id=1003] Received LeaderAndIsr request
PartitionState(controllerEpoch=1, leader=1003, leaderEpoch=6,
isr=1001,1002,1003, zkVersion=9, replicas=1003,1001,1002, isNew=false)
correlation id 7 from controller 1001 epoch 1 for partition apple-0
(state.change.logger)
[2018-05-28 00:43:20,093] TRACE [Broker id=1003] Handling LeaderAndIsr request
correlationId 7 from controller 1001 epoch 1 starting the become-leader
transition for partition apple-0 (state.change.logger)
[2018-05-28 00:43:20,193] INFO [ReplicaFetcherManager on broker 1003] Removed
fetcher for partitions apple-0 (kafka.server.ReplicaFetcherManager)
[2018-05-28 00:43:20,193] INFO [Partition apple-0 broker=1003] apple-0 starts
at Leader Epoch 6 from offset 0. Previous Leader Epoch was: 5
(kafka.cluster.Partition)
[2018-05-28 00:43:20,193] TRACE [Broker id=1003] Stopped fetchers as part of
become-leader request from controller 1001 epoch 1 with correlation id 7 for
partition apple-0 (last update controller epoch 1) (state.change.logger)
[2018-05-28 00:43:20,194] TRACE [Broker id=1003] Completed LeaderAndIsr request
correlationId 7 from controller 1001 epoch 1 for the become-leader transition
for partition apple-0 (state.change.logger)
[2018-05-28 00:43:20,993] ERROR [ReplicaFetcher replicaId=1003, leaderId=1001,
fetcherId=0] Error for partition apple-6 to broker
1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2018-05-28 00:43:20,993] ERROR [ReplicaFetcher replicaId=1003, leaderId=1001,
fetcherId=0] Error for partition steve-0 to broker
1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2018-05-28 00:43:20,993] TRACE [Broker id=1003] Cached leader info
PartitionState(controllerEpoch=1, leader=1003, leaderEpoch=6, isr=[1001, 1002,
1003], zkVersion=9, replicas=[1003, 1001, 1002], offlineReplicas=[]) for
partition apple-0 in response to UpdateMetadata request sent by controller 1001
epoch 1 with correlation id 8 (state.change.logger)
[2018-05-28 00:43:20,993] ERROR [ReplicaFetcher replicaId=1003, leaderId=1001,
fetcherId=0] Error for partition mango2-0 to broker
1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2018-05-28 00:43:21,268] ERROR [ReplicaFetcher replicaId=1003, leaderId=1001,
fetcherId=0] Error for partition apple-12 to broker
1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2018-05-28 00:43:21,268] ERROR [ReplicaFetcher replicaId=1003, leaderId=1001,
fetcherId=0] Error for partition mango2-6 to broker
1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2018-05-28 00:43:21,268] ERROR [ReplicaFetcher replicaId=1003, leaderId=1001,
fetcherId=0] Error for partition mango2-12 to broker
1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2018-05-28 00:43:21,298] TRACE [Broker id=1003] Received LeaderAndIsr request
PartitionState(controllerEpoch=1, leader=1003, leaderEpoch=6, isr=1001,1003,
zkVersion=9, replicas=1003,1001, isNew=false) correlation id 9 from controller
1001 epoch 1 for partition mango2-0 (state.change.logger)
[2018-05-28 00:43:21,298] TRACE [Broker id=1003] Handling LeaderAndIsr request
correlationId 9 from controller 1001 epoch 1 starting the become-leader
transition for partition mango2-0 (state.change.logger)
[2018-05-28 00:43:21,298] INFO [ReplicaFetcherManager on broker 1003] Removed
fetcher for partitions mango2-0 (kafka.server.ReplicaFetcherManager)
[2018-05-28 00:43:21,298] INFO [Partition mango2-0 broker=1003] mango2-0 starts
at Leader Epoch 6 from offset 0. Previous Leader Epoch was: 5
(kafka.cluster.Partition)
[2018-05-28 00:43:21,298] TRACE [Broker id=1003] Stopped fetchers as part of
become-leader request from controller 1001 epoch 1 with correlation id 9 for
partition mango2-0 (last update controller epoch 1) (state.change.logger)
[2018-05-28 08:00:10,672] TRACE [Controller id=1001] Checking need to trigger
auto leader balancing (kafka.controller.KafkaController)
[2018-05-28 08:00:10,673] DEBUG [Controller id=1001] Preferred replicas by
broker Map() (kafka.controller.KafkaController)
[2018-05-28 08:00:26,671] INFO [Controller id=1001] New topics:
[Set(applestore)], deleted topics: [Set()], new partition replica assignment
[Map(applestore-2 -> Vector(1003, 1002, 1001), applestore-0 -> Vector(1001,
1003, 1002), applestore-1 -> Vector(1002, 1001, 1003))]
(kafka.controller.KafkaController)
[2018-05-28 08:00:26,674] INFO [Controller id=1001] New topic creation callback
for applestore-2,applestore-0,applestore-1 (kafka.controller.KafkaController)
[2018-05-28 08:00:26,777] INFO [Controller id=1001] New partition creation
callback for applestore-2,applestore-0,applestore-1
(kafka.controller.KafkaController)
[2018-05-28 08:00:26,777] INFO [PartitionStateMachine controllerId=1001]
Invoking state change to NewPartition for partitions
applestore-2,applestore-0,applestore-1 (kafka.controller.PartitionStateMachine)
1002,1001 (state.change.logger)
[2018-05-28 08:00:26,869] TRACE [Controller id=1001 epoch=1] Changed partition
applestore-0 state from NonExistentPartition to NewPartition with assigned
replicas 1001,1003,1002 (state.change.logger)
[2018-05-28 08:00:26,869] TRACE [Controller id=1001 epoch=1] Changed partition
applestore-1 state from NonExistentPartition to NewPartition with assigned
replicas 1002,1001,1003 (state.change.logger)
[2018-05-28 08:00:26,880] INFO [ReplicaStateMachine controllerId=1001] Invoking
state change to NewReplica for replicas
[Topic=applestore,Partition=1,Replica=1001],[Topic=applestore,Partition=2,Replica=1003],[Topic=applestore,Partition=0,Replica=1003],[Topic=applestore,Partition=0,Replica=1001],[Topic=applestore,Partition=0,Replica=1002],[Topic=applestore,Partition=2,Replica=1001],[Topic=applestore,Partition=2,Replica=1002],[Topic=applestore,Partition=1,Replica=1003],[Topic=applestore,Partition=1,Replica=1002]
(kafka.controller.ReplicaStateMachine)
[2018-05-28 08:00:27,075] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1001 for partition applestore-1 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,077] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1003 for partition applestore-2 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,078] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1003 for partition applestore-0 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,078] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1001 for partition applestore-0 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,079] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1002 for partition applestore-0 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,168] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1001 for partition applestore-2 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,171] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1002 for partition applestore-2 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,173] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1003 for partition applestore-1 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,173] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1002 for partition applestore-1 from NonExistentReplica to NewReplica
(state.change.logger)
[2018-05-28 08:00:27,174] INFO [PartitionStateMachine controllerId=1001]
Invoking state change to OnlinePartition for partitions
applestore-2,applestore-0,applestore-1 (kafka.controller.PartitionStateMachine)
[2018-05-28 08:00:27,176] DEBUG [PartitionStateMachine controllerId=1001] Live
assigned replicas for partition applestore-2 are: [List(1003, 1002, 1001)]
(kafka.controller.PartitionStateMachine)
[2018-05-28 08:00:27,179] DEBUG [PartitionStateMachine controllerId=1001]
Initializing leader and isr for partition applestore-2 to
(Leader:1003,ISR:1003,1002,1001,LeaderEpoch:0,ControllerEpoch:1)
(kafka.controller.PartitionStateMachine)
[2018-05-28 08:00:27,376] TRACE [Controller id=1001 epoch=1] Changed partition
applestore-2 from NewPartition to OnlinePartition with leader 1003
(state.change.logger)
[2018-05-28 08:00:27,376] DEBUG [PartitionStateMachine controllerId=1001] Live
assigned replicas for partition applestore-0 are: [List(1001, 1003, 1002)]
(kafka.controller.PartitionStateMachine)
[2018-05-28 08:00:27,376] DEBUG [PartitionStateMachine controllerId=1001]
Initializing leader and isr for partition applestore-0 to
(Leader:1001,ISR:1001,1003,1002,LeaderEpoch:0,ControllerEpoch:1)
(kafka.controller.PartitionStateMachine)
[2018-05-28 08:00:27,469] TRACE [Controller id=1001 epoch=1] Changed partition
applestore-0 from NewPartition to OnlinePartition with leader 1001
(state.change.logger)
[2018-05-28 08:00:27,471] DEBUG [PartitionStateMachine controllerId=1001] Live
assigned replicas for partition applestore-1 are: [List(1002, 1001, 1003)]
(kafka.controller.PartitionStateMachine)
[2018-05-28 08:00:27,471] DEBUG [PartitionStateMachine controllerId=1001]
Initializing leader and isr for partition applestore-1 to
(Leader:1002,ISR:1002,1001,1003,LeaderEpoch:0,ControllerEpoch:1)
(kafka.controller.PartitionStateMachine)
[2018-05-28 08:00:27,477] TRACE [Controller id=1001 epoch=1] Changed partition
applestore-1 from NewPartition to OnlinePartition with leader 1002
(state.change.logger)
[2018-05-28 08:00:27,479] TRACE [Controller id=1001 epoch=1] Sending
become-follower LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1002, leaderEpoch=0, isr=1002,1001,1003, zkVersion=0,
replicas=1002,1001,1003, isNew=true) to broker 1001 for partition applestore-1
(state.change.logger)
[2018-05-28 08:00:27,479] TRACE [Controller id=1001 epoch=1] Sending
become-leader LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1001, leaderEpoch=0, isr=1001,1003,1002, zkVersion=0,
replicas=1001,1003,1002, isNew=true) to broker 1001 for partition applestore-0
(state.change.logger)
[2018-05-28 08:00:27,479] TRACE [Controller id=1001 epoch=1] Sending
become-follower LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1003, leaderEpoch=0, isr=1003,1002,1001, zkVersion=0,
replicas=1003,1002,1001, isNew=true) to broker 1001 for partition applestore-2
(state.change.logger)
[2018-05-28 08:00:27,482] TRACE [Controller id=1001 epoch=1] Sending
become-follower LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1002, leaderEpoch=0, isr=1002,1001,1003, zkVersion=0,
replicas=1002,1001,1003, isNew=true) to broker 1003 for partition applestore-1
(state.change.logger)
[2018-05-28 08:00:27,482] TRACE [Controller id=1001 epoch=1] Sending
become-follower LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1001, leaderEpoch=0, isr=1001,1003,1002, zkVersion=0,
replicas=1001,1003,1002, isNew=true) to broker 1003 for partition applestore-0
(state.change.logger)
[2018-05-28 08:00:27,482] TRACE [Controller id=1001 epoch=1] Sending
become-leader LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1003, leaderEpoch=0, isr=1003,1002,1001, zkVersion=0,
replicas=1003,1002,1001, isNew=true) to broker 1003 for partition applestore-2
(state.change.logger)
[2018-05-28 08:00:27,567] TRACE [Controller id=1001 epoch=1] Sending
become-leader LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1002, leaderEpoch=0, isr=1002,1001,1003, zkVersion=0,
replicas=1002,1001,1003, isNew=true) to broker 1002 for partition applestore-1
(state.change.logger)
[2018-05-28 08:00:27,567] TRACE [Controller id=1001 epoch=1] Sending
become-follower LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1001, leaderEpoch=0, isr=1001,1003,1002, zkVersion=0,
replicas=1001,1003,1002, isNew=true) to broker 1002 for partition applestore-0
(state.change.logger)
[2018-05-28 08:00:27,567] TRACE [Controller id=1001 epoch=1] Sending
become-follower LeaderAndIsr request PartitionState(controllerEpoch=1,
leader=1003, leaderEpoch=0, isr=1003,1002,1001, zkVersion=0,
replicas=1003,1002,1001, isNew=true) to broker 1002 for partition applestore-2
(state.change.logger)
[2018-05-28 08:00:27,571] TRACE [Controller id=1001 epoch=1] Sending
UpdateMetadata request PartitionState(controllerEpoch=1, leader=1002,
leaderEpoch=0, isr=[1002, 1001, 1003], zkVersion=0, replicas=[1002, 1001,
1003], offlineReplicas=[]) to brokers Set(1002, 1003, 1001) for partition
applestore-1 (state.change.logger)
[2018-05-28 08:00:27,572] INFO [Controller-1001-to-broker-1001-send-thread]:
Controller 1001 connected to 192.178.1.189:9092 (id: 1001 rack: null) for
sending state change requests (kafka.controller.RequestSendThread)
[2018-05-28 08:00:27,572] INFO [Controller-1001-to-broker-1002-send-thread]:
Controller 1001 connected to 192.178.1.25:9092 (id: 1002 rack: null) for
sending state change requests (kafka.controller.RequestSendThread)
[2018-05-28 08:00:27,572] TRACE [Controller id=1001 epoch=1] Sending
UpdateMetadata request PartitionState(controllerEpoch=1, leader=1001,
leaderEpoch=0, isr=[1001, 1003, 1002], zkVersion=0, replicas=[1001, 1003,
1002], offlineReplicas=[]) to brokers Set(1002, 1003, 1001) for partition
applestore-0 (state.change.logger)
[2018-05-28 08:00:27,572] INFO [Controller-1001-to-broker-1003-send-thread]:
Controller 1001 connected to 192.178.1.120:9092 (id: 1003 rack: null) for
sending state change requests (kafka.controller.RequestSendThread)
[2018-05-28 08:00:27,572] TRACE [Controller id=1001 epoch=1] Sending
UpdateMetadata request PartitionState(controllerEpoch=1, leader=1003,
leaderEpoch=0, isr=[1003, 1002, 1001], zkVersion=0, replicas=[1003, 1002,
1001], offlineReplicas=[]) to brokers Set(1002, 1003, 1001) for partition
applestore-2 (state.change.logger)
[2018-05-28 08:00:27,574] INFO [ReplicaStateMachine controllerId=1001] Invoking
state change to OnlineReplica for replicas
[Topic=applestore,Partition=1,Replica=1001],[Topic=applestore,Partition=2,Replica=1003],[Topic=applestore,Partition=0,Replica=1003],[Topic=applestore,Partition=0,Replica=1001],[Topic=applestore,Partition=0,Replica=1002],[Topic=applestore,Partition=2,Replica=1001],[Topic=applestore,Partition=2,Replica=1002],[Topic=applestore,Partition=1,Replica=1003],[Topic=applestore,Partition=1,Replica=1002]
(kafka.controller.ReplicaStateMachine)
[2018-05-28 08:00:27,575] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1001 for partition applestore-1 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,575] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1003 for partition applestore-2 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,575] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1003 for partition applestore-0 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,575] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1001 for partition applestore-0 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,575] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1002 for partition applestore-0 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,575] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1001 for partition applestore-2 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,575] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1002 for partition applestore-2 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,577] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1003 for partition applestore-1 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,577] TRACE [Controller id=1001 epoch=1] Changed state of
replica 1002 for partition applestore-1 from NewReplica to OnlineReplica
(state.change.logger)
[2018-05-28 08:00:27,667] TRACE [Broker id=1001] Received LeaderAndIsr request
PartitionState(controllerEpoch=1, leader=1001, leaderEpoch=0,
isr=1001,1003,1002, zkVersion=0, replicas=1001,1003,1002, isNew=true)
correlation id 3 from controller 1001 epoch 1 for partition applestore-0
(state.change.logger)
[2018-05-28 08:00:27,667] TRACE [Broker id=1001] Received LeaderAndIsr request
PartitionState(controllerEpoch=1, leader=1002, leaderEpoch=0,
isr=1002,1001,1003, zkVersion=0, replicas=1002,1001,1003, isNew=true)
correlation id 3 from controller 1001 epoch 1 for partition applestore-1
(state.change.logger)
[2018-05-28 08:00:27,667] TRACE [Broker id=1001] Received LeaderAndIsr request
PartitionState(controllerEpoch=1, leader=1003, leaderEpoch=0,
isr=1003,1002,1001, zkVersion=0, replicas=1003,1002,1001, isNew=true)
correlation id 3 from controller 1001 epoch 1 for partition applestore-2
(state.change.logger)
[2018-05-28 08:00:27,677] TRACE [Broker id=1001] Handling LeaderAndIsr request
correlationId 3 from controller 1001 epoch 1 starting the become-leader
transition for partition applestore-0 (state.change.logger)
[2018-05-28 08:00:27,678] INFO [ReplicaFetcherManager on broker 1001] Removed
fetcher for partitions applestore-0 (kafka.server.ReplicaFetcherManager)
[2018-05-28 08:00:28,176] INFO Loading producer state from offset 0 for
partition applestore-0 with message format version 2 (kafka.log.Log)
[2018-05-28 08:00:28,272] INFO Completed load of log applestore-0 with 1 log
segments, log start offset 0 and log end offset 0 in 393 ms (kafka.log.Log)
[2018-05-28 08:00:28,276] INFO Created log for partition [applestore,0] in
/var/lib/kafka/data/topics with properties {compression.type -> producer,
message.format.version -> 1.0-IV0, file.delete.delay.ms -> 60000,
max.message.bytes -> 1000012, min.compaction.lag.ms -> 0,
message.timestamp.type -> CreateTime, min.insync.replicas -> 1,
segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5,
index.interval.bytes -> 4096, unclean.leader.election.enable -> false,
retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy ->
[delete], flush.ms -> 1000, segment.ms -> 604800000, segment.bytes ->
1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms ->
9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 10000}.
(kafka.log.LogManager)
[2018-05-28 08:00:28,278] INFO [Partition applestore-0 broker=1001] No
checkpointed highwatermark is found for partition applestore-0
(kafka.cluster.Partition)
[2018-05-28 08:00:28,769] INFO Replica loaded for partition applestore-0 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:28,771] INFO Replica loaded for partition applestore-0 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:28,771] INFO Replica loaded for partition applestore-0 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:28,775] INFO [Partition applestore-0 broker=1001]
applestore-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was:
-1 (kafka.cluster.Partition)
[2018-05-28 08:00:29,079] TRACE [Broker id=1001] Stopped fetchers as part of
become-leader request from controller 1001 epoch 1 with correlation id 3 for
partition applestore-0 (last update controller epoch 1) (state.change.logger)
[2018-05-28 08:00:29,171] TRACE [Broker id=1001] Completed LeaderAndIsr request
correlationId 3 from controller 1001 epoch 1 for the become-leader transition
for partition applestore-0 (state.change.logger)
[2018-05-28 08:00:29,173] TRACE [Broker id=1001] Handling LeaderAndIsr request
correlationId 3 from controller 1001 epoch 1 starting the become-follower
transition for partition applestore-1 (state.change.logger)
[2018-05-28 08:00:29,173] TRACE [Broker id=1001] Handling LeaderAndIsr request
correlationId 3 from controller 1001 epoch 1 starting the become-follower
transition for partition applestore-2 (state.change.logger)
[2018-05-28 08:00:29,273] INFO Replica loaded for partition applestore-1 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:29,278] INFO Loading producer state from offset 0 for
partition applestore-1 with message format version 2 (kafka.log.Log)
[2018-05-28 08:00:29,279] INFO Completed load of log applestore-1 with 1 log
segments, log start offset 0 and log end offset 0 in 2 ms (kafka.log.Log)
[2018-05-28 08:00:29,281] INFO Created log for partition [applestore,1] in
/var/lib/kafka/data/topics with properties {compression.type -> producer,
message.format.version -> 1.0-IV0, file.delete.delay.ms -> 60000,
max.message.bytes -> 1000012, min.compaction.lag.ms -> 0,
message.timestamp.type -> CreateTime, min.insync.replicas -> 1,
segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5,
index.interval.bytes -> 4096, unclean.leader.election.enable -> false,
retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy ->
[delete], flush.ms -> 1000, segment.ms -> 604800000, segment.bytes ->
1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms ->
9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 10000}.
(kafka.log.LogManager)
[2018-05-28 08:00:29,281] INFO [Partition applestore-1 broker=1001] No
checkpointed highwatermark is found for partition applestore-1
(kafka.cluster.Partition)
[2018-05-28 08:00:29,281] INFO Replica loaded for partition applestore-1 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:29,281] INFO Replica loaded for partition applestore-1 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:29,283] INFO Replica loaded for partition applestore-2 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:29,283] INFO Replica loaded for partition applestore-2 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:29,368] INFO Loading producer state from offset 0 for
partition applestore-2 with message format version 2 (kafka.log.Log)
[2018-05-28 08:00:29,369] INFO Completed load of log applestore-2 with 1 log
segments, log start offset 0 and log end offset 0 in 84 ms (kafka.log.Log)
[2018-05-28 08:00:29,372] INFO Created log for partition [applestore,2] in
/var/lib/kafka/data/topics with properties {compression.type -> producer,
message.format.version -> 1.0-IV0, file.delete.delay.ms -> 60000,
max.message.bytes -> 1000012, min.compaction.lag.ms -> 0,
message.timestamp.type -> CreateTime, min.insync.replicas -> 1,
segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5,
index.interval.bytes -> 4096, unclean.leader.election.enable -> false,
retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy ->
[delete], flush.ms -> 1000, segment.ms -> 604800000, segment.bytes ->
1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms ->
9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 10000}.
(kafka.log.LogManager)
[2018-05-28 08:00:29,372] INFO [Partition applestore-2 broker=1001] No
checkpointed highwatermark is found for partition applestore-2
(kafka.cluster.Partition)
[2018-05-28 08:00:29,372] INFO Replica loaded for partition applestore-2 with
initial high watermark 0 (kafka.cluster.Replica)
[2018-05-28 08:00:29,373] INFO [ReplicaFetcherManager on broker 1001] Removed
fetcher for partitions applestore-1,applestore-2
(kafka.server.ReplicaFetcherManager)
[2018-05-28 08:00:29,374] TRACE [Broker id=1001] Stopped fetchers as part of
become-follower request from controller 1001 epoch 1 with correlation id 3 for
partition applestore-2 (state.change.logger)
[2018-05-28 08:00:29,374] TRACE [Broker id=1001] Stopped fetchers as part of
become-follower request from controller 1001 epoch 1 with correlation id 3 for
partition applestore-1 (state.change.logger)
[2018-05-28 08:00:29,377] TRACE [Broker id=1001] Truncated logs and
checkpointed recovery boundaries for partition applestore-2 as part of
become-follower request with correlation id 3 from controller 1001 epoch 1
(state.change.logger)
[2018-05-28 08:00:29,377] TRACE [Broker id=1001] Truncated logs and
checkpointed recovery boundaries for partition applestore-1 as part of
become-follower request with correlation id 3 from controller 1001 epoch 1
(state.change.logger)
[2018-05-28 08:00:29,673] INFO [ReplicaFetcher replicaId=1001, leaderId=1002,
fetcherId=8] Starting (kafka.server.ReplicaFetcherThread)
[2018-05-28 08:00:29,872] INFO [ReplicaFetcher replicaId=1001, leaderId=1003,
fetcherId=7] Starting (kafka.server.ReplicaFetcherThread)
[2018-05-28 08:00:29,969] INFO [ReplicaFetcherManager on broker 1001] Added
fetcher for partitions List([applestore-2, initOffset 0 to broker
BrokerEndPoint(1003,192.178.1.120,9092)] , [applestore-1, initOffset 0 to
broker BrokerEndPoint(1002,192.178.1.25,9092)] )
(kafka.server.ReplicaFetcherManager)
[2018-05-28 08:00:30,072] TRACE [Broker id=1001] Started fetcher to new leader
as part of become-follower request from controller 1001 epoch 1 with
correlation id 3 for partition applestore-2 (state.change.logger)
[2018-05-28 08:00:30,072] TRACE [Broker id=1001] Started fetcher to new leader
as part of become-follower request from controller 1001 epoch 1 with
correlation id 3 for partition applestore-1 (state.change.logger)
[2018-05-28 08:00:30,074] TRACE [Broker id=1001] Completed LeaderAndIsr request
correlationId 3 from controller 1001 epoch 1 for the become-follower transition
for partition applestore-1 (state.change.logger)
[2018-05-28 08:00:30,074] TRACE [Broker id=1001] Completed LeaderAndIsr request
correlationId 3 from controller 1001 epoch 1 for the become-follower transition
for partition applestore-2 (state.change.logger)
[2018-05-28 08:00:30,167] TRACE [Controller id=1001 epoch=1] Received response
{error_code=0,partitions=[{topic=applestore,partition=0,error_code=0},{topic=applestore,partition=1,error_code=0},{topic=applestore,partition=2,error_code=0}]}
for a request sent to broker 192.178.1.25:9092 (id: 1002 rack: null)
(state.change.logger)
[2018-05-28 08:00:30,274] TRACE [Controller id=1001 epoch=1] Received response
{error_code=0,partitions=[{topic=applestore,partition=0,error_code=0},{topic=applestore,partition=1,error_code=0},{topic=applestore,partition=2,error_code=0}]}
for a request sent to broker 192.178.1.189:9092 (id: 1001 rack: null)
(state.change.logger)
[2018-05-28 08:00:30,372] TRACE [Broker id=1001] Cached leader info
PartitionState(controllerEpoch=1, leader=1001, leaderEpoch=0, isr=[1001, 1003,
1002], zkVersion=0, replicas=[1001, 1003, 1002], offlineReplicas=[]) for
partition applestore-0 in response to UpdateMetadata request sent by controller
1001 epoch 1 with correlation id 4 (state.change.logger)
[2018-05-28 08:00:30,372] TRACE [Broker id=1001] Cached leader info
PartitionState(controllerEpoch=1, leader=1002, leaderEpoch=0, isr=[1002, 1001,
1003], zkVersion=0, replicas=[1002, 1001, 1003], offlineReplicas=[]) for
partition applestore-1 in response to UpdateMetadata request sent by controller
1001 epoch 1 with correlation id 4 (state.change.logger)
[2018-05-28 08:00:30,372] TRACE [Broker id=1001] Cached leader info
PartitionState(controllerEpoch=1, leader=1003, leaderEpoch=0, isr=[1003, 1002,
1001], zkVersion=0, replicas=[1003, 1002, 1001], offlineReplicas=[]) for
partition applestore-2 in response to UpdateMetadata request sent by controller
1001 epoch 1 with correlation id 4 (state.change.logger)
[2018-05-28 08:00:30,373] TRACE [Controller id=1001 epoch=1] Received response
{error_code=0} for a request sent to broker 192.178.1.189:9092 (id: 1001 rack:
null) (state.change.logger)
[2018-05-28 08:00:30,498] TRACE [Controller id=1001 epoch=1] Received response
{error_code=0} for a request sent to broker 192.178.1.25:9092 (id: 1002 rack:
null) (state.change.logger)
[2018-05-28 08:00:30,572] INFO [ReplicaFetcher replicaId=1001, leaderId=1002,
fetcherId=8] Based on follower's leader epoch, leader replied with an offset 0
>= the follower's log end offset 0 in applestore-1. No truncation needed.
(kafka.server.ReplicaFetcherThread)
[2018-05-28 08:00:30,575] INFO Truncating applestore-1 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-05-28 08:00:30,614] INFO [ReplicaFetcher replicaId=1001, leaderId=1003,
fetcherId=7] Based on follower's leader epoch, leader replied with an offset 0
>= the follower's log end offset 0 in applestore-2. No truncation needed.
(kafka.server.ReplicaFetcherThread)
[2018-05-28 08:00:30,615] INFO Truncating applestore-2 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-05-28 08:00:31,767] TRACE [Controller id=1001 epoch=1] Received response
{error_code=0,partitions=[{topic=applestore,partition=0,error_code=0},{topic=applestore,partition=1,error_code=0},{topic=applestore,partition=2,error_code=0}]}
for a request sent to broker 192.178.1.120:9092 (id: 1003 rack: null)
(state.change.logger)
[2018-05-28 08:00:31,867] TRACE [Controller id=1001 epoch=1] Received response
{error_code=0} for a request sent to broker 192.178.1.120:9092 (id: 1003 rack:
null) (state.change.logger)
[2018-05-28 08:00:31,875] ERROR [ReplicaFetcher replicaId=1001, leaderId=1003,
fetcherId=7] Error for partition applestore-2 to broker
1003:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
^C
