[
https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482302#comment-14482302
]
Evan Huus commented on KAFKA-2082:
----------------------------------
OK, that all makes sense. Some follow-up questions:
1. When the cluster heals (at the end of the test) the replication continues to
fail. Based on your explanation, I would have thought that the cluster healing
would have triggered oncontrollerresignation, which would update the metadata
on that node and fix the replication by pointing it to the new leader, but that
does not appear to be the case?
2. The go client currently tries brokers in the order specified by the user on
startup, not randomly. I can see the argument for a random choice (it's on our
todo list) but even that only works around the problem - an unlucky producer
could choose the bad broker enough times in a row to run out of retries and
start dropping messages. I really think that an isolated broker should refuse
to serve metadata requests - it knows that its information is likely stale, and
forcing clients to try another broker is the only way for them to reliably get
fresh metadata. Just like documentation: the only metadata worse than no
metadata is incorrect metadata.
> Kafka Replication ends up in a bad state
> ----------------------------------------
>
> Key: KAFKA-2082
> URL: https://issues.apache.org/jira/browse/KAFKA-2082
> Project: Kafka
> Issue Type: Bug
> Components: replication
> Affects Versions: 0.8.2.1
> Reporter: Evan Huus
> Assignee: Sriharsha Chintalapani
> Priority: Critical
> Attachments: KAFKA-2082.patch
>
>
> While running integration tests for Sarama (the go client) we came across a
> pattern of connection losses that reliably puts kafka into a bad state:
> several of the brokers start spinning, chewing ~30% CPU and spamming the logs
> with hundreds of thousands of lines like:
> {noformat}
> [2015-04-01 13:08:40,070] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111094 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,1] failed due to Leader not local for partition
> [many_partition,1] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,070] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111094 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,6] failed due to Leader not local for partition
> [many_partition,6] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,070] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,21] failed due to Leader not local for partition
> [many_partition,21] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,071] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,26] failed due to Leader not local for partition
> [many_partition,26] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,071] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,1] failed due to Leader not local for partition
> [many_partition,1] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,071] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,6] failed due to Leader not local for partition
> [many_partition,6] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,072] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111096 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,21] failed due to Leader not local for partition
> [many_partition,21] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,072] WARN [Replica Manager on Broker 9093]: Fetch
> request with correlation id 111096 from client ReplicaFetcherThread-0-9093 on
> partition [many_partition,26] failed due to Leader not local for partition
> [many_partition,26] on broker 9093 (kafka.server.ReplicaManager)
> {noformat}
> This can be easily and reliably reproduced using the {{toxiproxy-final}}
> branch of https://github.com/Shopify/sarama which includes a vagrant script
> for provisioning the appropriate cluster:
> - {{git clone https://github.com/Shopify/sarama.git}}
> - {{git checkout test-jira-kafka-2082}}
> - {{vagrant up}}
> - {{TEST_SEED=1427917826425719059 DEBUG=true go test -v}}
> After the test finishes (it fails because the cluster ends up in a bad
> state), you can log into the cluster machine with {{vagrant ssh}} and inspect
> the bad nodes. The vagrant script provisions five zookeepers and five brokers
> in {{/opt/kafka-9091/}} through {{/opt/kafka-9095/}}.
> Additional context: the test produces continually to the cluster while
> randomly cutting and restoring zookeeper connections (all connections to
> zookeeper are run through a simple proxy on the same vm to make this easy).
> The majority of the time this works very well and does a good job exercising
> our producer's retry and failover code. However, under certain patterns of
> connection loss (the {{TEST_SEED}} in the instructions is important), kafka
> gets confused. The test never cuts more than two connections at a time, so
> zookeeper should always have quorum, and the topic (with three replicas)
> should always be writable.
> Completely restarting the cluster via {{vagrant reload}} seems to put it back
> into a sane state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
