[ https://issues.apache.org/jira/browse/KAFKA-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17354180#comment-17354180 ]
Duong Pham commented on KAFKA-9135: ----------------------------------- you can upgrade Java version, prefer 1.8.0.291 > Kafka producer/consumer are creating too many open file > ------------------------------------------------------- > > Key: KAFKA-9135 > URL: https://issues.apache.org/jira/browse/KAFKA-9135 > Project: Kafka > Issue Type: Bug > Components: admin, consumer, producer > Affects Versions: 1.0.1 > Environment: apache kafka client :- 1.0.1 > Kafka version :- 1.0.1 > Open JDK :- java-1.8.0-openjdk-1.8.0.222.b10-1 > CentOS version :- CentOS Linux release 7.6.1810 > Reporter: Dhirendra Singh > Priority: Critical > > We have a 3 node Kafka cluster deployment with 5 topic and 6 partition per > topic . we have configured the replication factor =3 , we are seeing very > strange problem that number of file descriptor have been crossed the ulimit ( > what is 50K for our application) > As per the lsof command and our ananlsys > 1. there are 15K established connection from kafka producer/consumer towards > broker and at the same time in thread dump we have observed thousands of > entry for kafka 'admin-client-network-thread' > admin-client-network-thread" #224398 daemon prio=5 os_prio=0 > tid=0x00007f12ca119800 nid=0x5363 runnable [0x00007f12c4db8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000005e0603238> (a sun.nio.ch.Util$3) > - locked <0x00000005e0603228> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000005e0602f08> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:672) > at org.apache.kafka.common.network.Selector.poll(Selector.java:396) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238) > - locked <0x00000005e0602dc0> (a > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:205) > at kafka.admin.AdminClient$$anon$1.run(AdminClient.scala:61) > at java.lang.Thread.run(Thread.java:748) > 2. As per the lsof output , We have observed 35K entry for pipe and event poll > java 5441 app 374r FIFO 0,9 0t0 22415240 pipe > java 5441 app 375w FIFO 0,9 0t0 22415240 pipe > java 5441 app 376u a_inode 0,10 0 6379 [eventpoll] > java 5441 app 377r FIFO 0,9 0t0 22473333 pipe > java 5441 app 378r FIFO 0,9 0t0 28054726 pipe > java 5441 app 379r FIFO 0,9 0t0 22415241 pipe > java 5441 app 380w FIFO 0,9 0t0 22415241 pipe > java 5441 app 381u a_inode 0,10 0 6379 [eventpoll] > java 5441 app 382w FIFO 0,9 0t0 22473333 pipe > java 5441 app 383u a_inode 0,10 0 6379 [eventpoll] > java 5441 app 384u a_inode 0,10 0 6379 [eventpoll] > java 5441 app 385r FIFO 0,9 0t0 40216087 pipe > java 5441 app 386r FIFO 0,9 0t0 22483470 pipe > Setup details :- > apache kafka client :- 1.0.1 > Kafka version :- 1.0.1 > Open JDK :- java-1.8.0-openjdk-1.8.0.222.b10-1 > CentOS version :- CentOS Linux release 7.6.1810 > Note :- After restarted VM file descriptor count was able to clear and come > to normal count as 1000 then after a few second file descriptor count started > to increase and it will reach to 50K (ulimit) after 1 week inIdle scenarion. -- This message was sent by Atlassian Jira (v8.3.4#803005)