[
https://issues.apache.org/jira/browse/KAFKA-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099967#comment-17099967
]
Eric Ward commented on KAFKA-9957:
----------------------------------
That is correct. xfs_freeze will not fail the writes, the writes will just
hang indefinitely.
In the case of our production failures we saw similar behavior, which is what
lead us to using xfs_freeze to reproduce. Any attempt to interact with the
affected filesystem would hang indefinitely, both reads and writes.
> Kafka Controller doesn't failover during hardware failure
> ---------------------------------------------------------
>
> Key: KAFKA-9957
> URL: https://issues.apache.org/jira/browse/KAFKA-9957
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 2.2.0, 2.5.0
> Reporter: Eric Ward
> Priority: Critical
> Attachments: kafka-threaddump.out
>
>
> On a couple different production environments we've run into an issue where a
> hardware failure has hung up the controller and prevented controller and
> topic leadership from changing to a healthy broker. When the issue happens
> we see this repeated in the logs at regular intervals for the other brokers
> (the affected broker can’t write to disk, so no logging occurs there):
> {noformat}
> [2020-04-26 01:12:30,613] WARN [ReplicaFetcher replicaId=0, leaderId=2,
> fetcherId=0] Error in response for fetch request (type=FetchRequest,
> replicaId=0, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={*snip*},
> isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=1962806970,
> epoch=INITIAL)) (kafka.server.ReplicaFetcherThread)
> java.io.IOException: Connection to 2 was disconnected before the response was
> read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:100)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:193)
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:280)
> at
> kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:132)
> at
> kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:131)
> at scala.Option.foreach(Option.scala:274)
> at
> kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:131)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:113)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> {noformat}
> This issue appears to be similar to KAFKA-7870, though that issue was
> purportedly fixed by KAFKA-7697.
> Once we encounter this error any partitions whose leadership is on the
> affected node are unavailable until we force that broker out of the cluster –
> that is to say, kill the node.
> When we initially hit the issue we were running on version 2.2.0, though I've
> been able to reproduce this in an environment running 2.5.0 as well. To
> simulate the hardware failure I'm using the xfs_freeze utility to suspend
> access to the filesystem. Zookeeper failover is also part of the mix. In
> all instances where we’ve seen this the ZK leader and Kafka Controller were
> on the same node and both affected by the hardware issue. Zookeeper is able
> to successfully failover, which it does rather quickly.
> Reproduction steps are pretty straightforward:
> # Spin up a 3 node cluster
> # Ensure that the Kafka Controller and Zookeeper Leader are on the same node.
> # xfs_freeze the filesystem on the node that the controller is running on
> This reproduces 100% of the time for me. I’ve left it running for well over
> an hour without any Kafka failover happening. Unfreezing the node will allow
> the cluster to heal itself.
> I’ve attached a thread dump from an environment running 2.5.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
