[ https://issues.apache.org/jira/browse/KAFKA-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366753#comment-16366753 ]
Rajini Sivaram edited comment on KAFKA-6565 at 2/16/18 9:12 AM: ---------------------------------------------------------------- [~berman] Looks like this could be due to the deadlocks KAFKA-4399 or KAFKA-3994 which have been fixed in 0.10.1.1. To verify, would you be able to run `_kill -QUIT <brokerPid>_` on an unresponsive broker when this happens again and attach the stack trace? Thank you! was (Author: rsivaram): [~berman] Looks like this could be due to the deadlocks KAFKA-4399 or KAFKA-3994 which have been fixed in 0.10.1.1. To verify, would you be able to run `_kill -QUIT <brokerPid>_` on an unresponsive broker when this happens again? Thank you! > Sudden unresponsiveness from broker + file descriptor leak > ---------------------------------------------------------- > > Key: KAFKA-6565 > URL: https://issues.apache.org/jira/browse/KAFKA-6565 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 0.10.1.0 > Environment: 3 separate clusters running 0.10.1.0 with 5 nodes + 5 > zookeepers. > clients are mostly java 0.10.2.1, and a few nodejs clients > (https://github.com/oleksiyk/kafka). > throughput per broker: ~3.5MB per second. each broker is a leader of ~2500 > partitions. > Debian GNU/Linux 8.8 (jessie) > java version “1.8.0_65” > c4.4xlarge , 1500gb gp2 disks > Reporter: Noam Berman > Priority: Blocker > > Running 3 kafka clusters v0.10.1.0 on aws, 5 brokers each (+5 zookeepers), > in the past week about 2-3 times a day (on all clusters, no specific brokers) > we encounter a situation in which a single broker stops responding to all > requests from clients/other brokers. at this moment all produced messages > fail, and other brokers start writing this to log repeatedly: > [2018-02-15 17:56:08,949] WARN [ReplicaFetcherThread-0-878], Error in fetch > kafka.server.ReplicaFetcherThread$FetchRequest@772ad40c > (kafka.server.ReplicaFetcherThread) > java.io.IOException: Connection to 878 was disconnected before the response > was read > at > kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:115) > at > kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:112) > at scala.Option.foreach(Option.scala:257) > at > kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:112) > at > kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:108) > at > kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:137) > at > kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143) > at > kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108) > at > kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:253) > at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238) > at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42) > at > kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) > at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) > > producers (examples): > 2018-02-13 21:30:51:010 thread=[kafka-producer-network-thread | > catalog-index-0] WARN Got error produce response with correlation id 8147 on > topic-partition catalog-1, retrying (49 attempts left). Error: > REQUEST_TIMED_OUT > 2018-02-13 21:31:06:221 thread=[kafka-producer-network-thread | > catalog-index-0] WARN Got error produce response with correlation id 8166 on > topic-partition catalog-index-18, retrying (48 attempts left). Error: > NOT_LEADER_FOR_PARTITION > > Once this happens, file descriptors on the affected broker start increasing > at a high rate until it reaches its maximum. > no relevant info logs on the affected broker, we'll try to gather debug logs > and attach them to the bug next time it happens. > the big issue here is that although the broker is unresponsive, it stays in > the cluster and its zookeeper node isn't cleared, so there's no rebalance > whatsoever - all clients start failing until a manual shutdown occurs. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)