[
https://issues.apache.org/jira/browse/KAFKA-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17777578#comment-17777578
]
Xi Yang edited comment on KAFKA-15591 at 10/20/23 6:01 AM:
-----------------------------------------------------------
I print out the topic description after creating the topic. It looks like the
partitions are correctly elected before Trogdor starts producing messages.
However, the producer still reports the NOT_LEADER_OR_FOLLOWER error.
Topic desc:(name=foo1, internal=false, partitions=(partition=0,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=1,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=2,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=3,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=4,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=5,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=6,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=7,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=8,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=9,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)), authorizedOperations=null)
Create topics:[foo1-9, foo1-8, foo1-7, foo1-6, foo1-5, foo1-4, foo1-3, foo1-2,
foo1-1, foo1-0]
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-7, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-7 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-6, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-6 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-5, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-5 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-4, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-4 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-3, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-3 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,844] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-2, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,844] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-2 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
was (Author: xyang):
I print out the topic description after creating the topic. It looks like the
partitions are correctly elected before Trogdor starts producing messages.
However, the producer still reports the NOT_LEADER_OR_FOLLOWER error.
```Topic desc:(name=foo1, internal=false, partitions=(partition=0,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=1,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=2,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=3,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=4,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=5,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=6,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=7,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=8,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)),(partition=9,
leader=localhost:9092 (id: 1 rack: null), replicas=localhost:9092 (id: 1 rack:
null), isr=localhost:9092 (id: 1 rack: null)), authorizedOperations=null)
Create topics:[foo1-9, foo1-8, foo1-7, foo1-6, foo1-5, foo1-4, foo1-3, foo1-2,
foo1-1, foo1-0]
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-7, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-7 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-6, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-6 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-5, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-5 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-4, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-4 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-3, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,843] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-3 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,844] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 4 on topic-partition foo1-2, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-20 05:43:42,844] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-2 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)```
> Trogdor produce workload reports errors in KRaft mode
> -----------------------------------------------------
>
> Key: KAFKA-15591
> URL: https://issues.apache.org/jira/browse/KAFKA-15591
> Project: Kafka
> Issue Type: Bug
> Environment: Linux
> Reporter: Xi Yang
> Priority: Blocker
> Attachments: trogdor-kafka-kraft.txt
>
>
> The Kafka benchmark in the Dacapo Benchmark Suite uses the Trogdor's exec
> mode ([https://github.com/dacapobench/dacapobench/pull/224)] to test the
> Kafka broker.
>
> I am trying to update the benchmark to use the KRaft protocol. We use single
> Kafka instant that plays both controller and broker following the guide in
> Kafka README.md
> (https://github.com/apache/kafka#running-a-kafka-broker-in-kraft-mode).
>
> However, the Trogdor producing workload
> (tests/spec/simple_produce_bench.json) reports the NOT_LEADER_OR_FOLLOWER
> error. The errors are gone after many time of retry. Is this caused by that
> in KRaft protocal, Kafka doesn't not elect leaders immediately after a new
> topic created but rather do that on-demand after receiving the first message
> on the topic? If this is the root cause, Is there a way to ask Kafka to elect
> the leader after creating the topic?
> {code:java}
> // code placeholder
> ./bin/trogdor.sh agent -n node0 -c ./config/trogdor.conf --exec
> ./tests/spec/simple_produce_bench.json
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/xyang/code/kafka/tools/build/dependant-libs-2.13.12/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xyang/code/kafka/trogdor/build/dependant-libs-2.13.12/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
> Oct 12, 2023 12:30:50 AM org.glassfish.jersey.server.wadl.WadlFeature
> configure
> WARNING: JAXBContext implementation could not be found. WADL feature is
> disabled.
> Oct 12, 2023 12:30:50 AM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider org.apache.kafka.trogdor.agent.AgentRestResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider org.apache.kafka.trogdor.agent.AgentRestResource will be ignored.
> Waiting for completion of task:{
> "class" : "org.apache.kafka.trogdor.workload.ProduceBenchSpec",
> "startMs" : 1697070650540,
> "durationMs" : 10000000,
> "producerNode" : "node0",
> "bootstrapServers" : "localhost:9092",
> "targetMessagesPerSec" : 10000,
> "maxMessages" : 50000,
> "keyGenerator" : {
> "type" : "sequential",
> "size" : 4,
> "startOffset" : 0
> },
> "valueGenerator" : {
> "type" : "constant",
> "size" : 512,
> "value" :
> "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
> },
> "activeTopics" : {
> "foo[1-3]" : {
> "numPartitions" : 10,
> "replicationFactor" : 1
> }
> },
> "inactiveTopics" : {
> "foo[4-5]" : {
> "numPartitions" : 10,
> "replicationFactor" : 1
> }
> },
> "useConfiguredPartitioner" : false,
> "skipFlush" : false
> }
> [2023-10-12 00:30:50,862] WARN [Producer clientId=producer-1] Got error
> produce response with correlation id 6 on topic-partition foo1-8, retrying
> (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER
> (org.apache.kafka.clients.producer.internals.Sender)
> [2023-10-12 00:30:50,862] WARN [Producer clientId=producer-1] Received
> invalid metadata error in produce request on partition foo1-8 due to
> org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
> intended only for the leader, this error indicates that the broker is not the
> current leader. For requests intended for any replica, this error indicates
> that the broker is not a replica of the topic partition.. Going to request
> metadata update now (org.apache.kafka.clients.producer.internals.Sender)
> [2023-10-12 00:30:50,870] WARN [Producer clientId=producer-1] Got error
> produce response with correlation id 8 on topic-partition foo2-8, retrying
> (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER
> (org.apache.kafka.clients.producer.internals.Sender)
> [2023-10-12 00:30:50,870] WARN [Producer clientId=producer-1] Received
> invalid metadata error in produce request on partition foo2-8 due to
> org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
> intended only for the leader, this error indicates that the broker is not the
> current leader. For requests intended for any replica, this error indicates
> that the broker is not a replica of the topic partition.. Going to request
> metadata update now (org.apache.kafka.clients.producer.internals.Sender)
> [2023-10-12 00:30:50,875] WARN [Producer clientId=producer-1] Got error
> produce response with correlation id 9 on topic-partition foo3-8, retrying
> (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER
> (org.apache.kafka.clients.producer.internals.Sender)
> [2023-10-12 00:30:50,875] WARN [Producer clientId=producer-1] Received
> invalid metadata error in produce request on partition foo3-8 due to
> org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
> intended only for the leader, this error indicates that the broker is not the
> current leader. For requests intended for any replica, this error indicates
> that the broker is not a replica of the topic partition.. Going to request
> metadata update now (org.apache.kafka.clients.producer.internals.Sender)
> [2023-10-12 00:30:50,876] WARN [Producer clientId=producer-1] Got error
> produce response with correlation id 9 on topic-partition foo2-4, retrying
> (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER
> (org.apache.kafka.clients.producer.internals.Sender)
> [2023-10-12 00:30:50,876] WARN [Producer clientId=producer-1] Received
> invalid metadata error in produce request on partition foo2-4 due to
> org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
> intended only for the leader, this error indicates that the broker is not the
> current leader. For requests intended for any replica, this error indicates
> that the broker is not a replica of the topic partition.. Going to request
> metadata update now (org.apache.kafka.clients.producer.internals.Sender)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
