David Dufour created KAFKA-20961:
------------------------------------
Summary: Loop of OffsetOutOfRangeException
Key: KAFKA-20961
URL: https://issues.apache.org/jira/browse/KAFKA-20961
Project: Kafka
Issue Type: Bug
Components: replication
Affects Versions: 4.1.1
Reporter: David Dufour
Hello,
we have identified scenarios where a leader broker returns
OffsetOutOfRangeException error to the leader in loop.
This is probably what was described in KAFKA-15608 but this issue was logged on
an old version and is still in status open.
We are experiencing this issue very regularly as we are using JBOD and we move
partitions from one disk to another inside a same broker.
Here is an example of logs:
{+}Leader:
{+}INFO Attempting to replace current log Log(...) with Log(...) for xxxxxx
(kafka.log.LogManager) [ReplicaAlterLogDirsThread-3]
...{+}
{+}INFO The current replica is successfully replaced with the future replica
for xxxxxx (kafka.log.LogManager) [ReplicaAlterLogDirsThread-3]
+Follower, few ms later:+
INFO [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] Reset fetch offset
for partition xxxxxx from 0 to the current local replica's end offset 0
(kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-3]
INFO [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] Current offset 0 for
partition xxxxxx is out of range, which typically implies a leader change.
Reset fetch offset to 0 (kafka.server.ReplicaFetcherThread)
[ReplicaFetcherThread-0-3]
....repeat every ~12ms
Often, the loop stops by itself after some time (traffic injected, broker
restart,...).
We use Cruise Control to perform cluster rebalance. Recently, a rebalance
triggered a huge number of such loops that never stopped. The logs were flooded
and in the end, the cluster became unresponsive to the clients, causing a
severe outage. A rolling restart recovered the situation.
+How to reproduce:+
1- create new empty topic
2- build some leader epoch (for ex: switch leadership from one broker to
another)
3- move partition on leader from dir1 to dir2
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
