[
https://issues.apache.org/jira/browse/KAFKA-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178672#comment-16178672
]
ASF GitHub Bot commented on KAFKA-5758:
---------------------------------------
GitHub user ijuma opened a pull request:
https://github.com/apache/kafka/pull/3954
KAFKA-5758: Don't fail fetch request if replica is no longer a follower for
a partition
We log a warning instead, which is what we also do if the partition hasn't
been created
yet.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijuma/kafka
kafka-5758-dont-fail-fetch-request-if-replica-is-not-follower
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3954.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3954
----
commit 7e12d408619b41a37cc40f4c294d47875df312ce
Author: Ismael Juma <ism...@juma.me.uk>
Date: 2017-09-25T07:33:46Z
KAFKA-5758: Don't fail fetch request if replica is no longer a follower for
a partition
We log a warning instead, which is what we also do if the partition hasn't
been created
yet.
----
> Reassigning a topic's partitions can adversely impact other topics
> ------------------------------------------------------------------
>
> Key: KAFKA-5758
> URL: https://issues.apache.org/jira/browse/KAFKA-5758
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.10.1.1
> Reporter: David van Geest
> Assignee: Ismael Juma
> Labels: reliability
> Fix For: 1.0.0
>
>
> We've noticed that reassigning a topic's partitions seems to adversely impact
> other topics. Specifically, followers for other topics fall out of the ISR.
> While I'm not 100% sure about why this happens, the scenario seems to be as
> follows:
> 1. Reassignment is manually triggered on topic-partition X-Y, and broker A
> (which used to be a follower for X-Y) is no longer a follower.
> 2. Broker A makes `FetchRequest` including topic-partition X-Y to broker B,
> just after the reassignment.
> 3. Broker B can fulfill the `FetchRequest`, but while trying to do so it
> tries to record the position of "follower" A. This fails, because broker A is
> no longer a follower for X-Y (see exception below).
> 4. The entire `FetchRequest` request fails, and broker A's other followed
> topics start falling behind.
> 5. Depending on the length of the reassignment, this sequence repeats.
> In step 3, we see exceptions like:
> {noformat}
> Error when handling request Name: FetchRequest; Version: 3; CorrelationId:
> 46781859; ClientId: ReplicaFetcherThread-0-1001; ReplicaId: 1006; MaxWait:
> 500 ms; MinBytes: 1 bytes; MaxBytes:10485760 bytes; RequestInfo:
> <LOTS OF PARTITIONS>
> kafka.common.NotAssignedReplicaException: Leader 1001 failed to record
> follower 1006's position -1 since the replica is not recognized to be one of
> the assigned replicas 1001,1004,1005 for partition [topic_being_reassigned,5].
> at kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:249)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:923)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:920)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:920)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:481)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:534)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Does my assessment make sense? If so, this behaviour seems problematic. A few
> changes that might improve matters (assuming I'm on the right track):
> 1. `FetchRequest` should be able to return partial results
> 2. The broker fulfilling the `FetchRequest` could ignore the
> `NotAssignedReplicaException`, and return results without recording the
> not-any-longer-follower position.
> This behaviour was experienced with 0.10.1.1, although looking at the
> changelogs and the code in question, I don't see any reason why it would have
> changed in later versions.
> Am very interested to have some discussion on this. Thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
