[ https://issues.apache.org/jira/browse/KAFKA-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008699#comment-16008699 ]
Colin P. McCabe commented on KAFKA-5007: ---------------------------------------- {code} java 24739 cyclone 1216u unix 0xffff880035b35000 0t0 42713601 socket {code} If I'm reading this right, this is a UNIX domain socket that is being leaked. Can you run {{netstat --unix}} and see if the sockets show up in that output? Also, I assume that 24739 was the Kafka process ID? A UNIX domain socket leak in Java code could indicate a problem with how the code uses NIO.... > Kafka Replica Fetcher Thread- Resource Leak > ------------------------------------------- > > Key: KAFKA-5007 > URL: https://issues.apache.org/jira/browse/KAFKA-5007 > Project: Kafka > Issue Type: Bug > Components: core, network > Affects Versions: 0.10.0.0, 0.10.1.1, 0.10.2.0 > Environment: Centos 7 > Jave 8 > Reporter: Joseph Aliase > Priority: Critical > Labels: reliability > Attachments: jstack-kafka.out, jstack-zoo.out, lsofkafka.txt, > lsofzookeeper.txt > > > Kafka is running out of open file descriptor when system network interface is > done. > Issue description: > We have a Kafka Cluster of 5 node running on version 0.10.1.1. The open file > descriptor for the account running Kafka is set to 100000. > During an upgrade, network interface went down. Outage continued for 12 hours > eventually all the broker crashed with java.io.IOException: Too many open > files error. > We repeated the test in a lower environment and observed that Open Socket > count keeps on increasing while the NIC is down. > We have around 13 topics with max partition size of 120 and number of replica > fetcher thread is set to 8. > Using an internal monitoring tool we observed that Open Socket descriptor > for the broker pid continued to increase although NIC was down leading to > Open File descriptor error. -- This message was sent by Atlassian JIRA (v6.3.15#6346)