[ https://issues.apache.org/jira/browse/KAFKA-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Stephenson updated KAFKA-8900: ----------------------------------- Summary: Stalled partition for a consumer group (was: Stalled partitions) > Stalled partition for a consumer group > -------------------------------------- > > Key: KAFKA-8900 > URL: https://issues.apache.org/jira/browse/KAFKA-8900 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 2.1.1 > Reporter: Luke Stephenson > Priority: Major > > I'm seeing behaviour where a Scala KafkaConsumer has stalled for 1 partition > for a topic. All other partitions for that topic are successfully being > consumed. > Restarting the consumer process does not resolve the issue. The consumer is > using version 2.3.0 ("org.apache.kafka" % "kafka-clients" % "2.3.0"). > When the consumer starts, I see that it is assigned the partition. However > it then logs: > {code} > [Consumer > clientId=kafka-bus-router-64c88855cf-hxck7.event-bus-router-consumer.1d1ed7ee-5038-4441-84eb-8080ac130e9a, > groupId=event-bus-router] Setting offset for partition > maxwell.transactions-22 to the committed offset > FetchPosition{offset=275413397, offsetEpoch=Optional[271], > currentLeader=LeaderAndEpoch{leader=:-1 (id: -1 rack: null), epoch=271}} > {code} > Note that the leader is logged as "-1". If I search through my application > logs for the past couple of days, the only time I ever see this logged on the > consumer is for this partition. > The kafka broker is running version 2.1.1. On the broker side the logs show: > {code} > {"timeMillis":1568087844876,"thread":"kafka-request-handler-1","level":"WARN","loggerName":"state.change.logger","message":"[Broker > id=5] Ignoring LeaderAndIsr request from controller 4 with correlation id > 15943 epoch 155 for partition maxwell.transactions-22 since its associated > leader epoch 270 is not higher than the current leader epoch > 270","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter","threadId":72,"threadPriority":5} > {"timeMillis":1568087844880,"thread":"kafka-request-handler-1","level":"INFO","loggerName":"kafka.server.ReplicaFetcherManager","message":"[ReplicaFetcherManager > on broker 5] Removed fetcher for partitions > Set(maxwell.transactions-22)","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter","threadId":72,"threadPriority":5} > {"timeMillis":1568087844880,"thread":"kafka-request-handler-1","level":"INFO","loggerName":"kafka.cluster.Partition","message":"[Partition > maxwell.transactions-22 broker=5] maxwell.transactions-22 starts at Leader > Epoch 271 from offset 275403423. Previous Leader Epoch was: > 270","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter","threadId":72,"threadPriority":5} > {"timeMillis":1568087844891,"thread":"kafka-request-handler-1","level":"INFO","loggerName":"state.change.logger","message":"[Broker > id=5] Skipped the become-leader state change after marking its partition as > leader with correlation id 15945 from controller 4 epoch 155 for partition > maxwell.transactions-22 (last update controller epoch 155) since it is > already the leader for the > partition.","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter","threadId":72,"threadPriority":5} > {code} > As soon as I restart the broker which is the leader for that partition, the > messages flow through to the consumer. > Given restarts of the consumer don't help, but restarting the broker allows > the stalled partition to resume, I'm inclined to think this is an issue with > the broker. Please let me know if I can assist further with investigating or > resolving this. -- This message was sent by Atlassian Jira (v8.3.2#803003)