[ https://issues.apache.org/jira/browse/SSHD-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lyor Goldstein updated SSHD-1025: --------------------------------- Labels: user-response-required (was: ) > "java.net.SocketException: Too many open files" exception on long run of > application with Apache Mina SSHD 2.1.0 > ---------------------------------------------------------------------------------------------------------------- > > Key: SSHD-1025 > URL: https://issues.apache.org/jira/browse/SSHD-1025 > Project: MINA SSHD > Issue Type: Bug > Affects Versions: 2.1.0 > Environment: Oracle Linux 7 > Reporter: Binay Paul > Priority: Major > Labels: user-response-required > > This issue is observed during the performance testing of Oracle > Communications Sessions Delivery Manager (OCSDM) release 8.2.1 with Apache > Mina SSHD 2.1.0 version, after a long continuous run for about 15 to 16 hours > of OCSDM with processing REST/SOAP API calls in a 3 node cluster setup. > Root Cause Analysis: > When OCSDMis running continuously with REST/SOAP consecutive requests, list > of opened file descriptors count increases rapidly like some where around 23 > Lakh and this leads to an exception "java.net.SocketException: Too many open > files". > As a result of this DB read/write operations fails and OCSDM shutdown task is > triggered by the application. > The lsof command shows FD named "PIPE" and "EventPoll" are been opened by SDM > process in large number. > These FDs are opened by third party library Apache Mina which is used to make > SSH/SFTP connections to 297 devices which are present in SDM (Used for SDM > performance testing). > Walk around : As of now for OCSDM release 8.2.1 library Apache Mina is > replaced with JSCH 0.1.55 which is been used in previous versions of SDM and > application works as expected. > > Console.out : > [Wed Nov 13 23:49:36 EST 2019] WARNING: RMI TCP Accept-1099: accept loop for > ServerSocket[addr=0.0.0.0/0.0.0.0,localport=1099] throws > [Wed Nov 13 23:49:36 EST 2019] java.net.SocketException: Too many open files > [Wed Nov 13 23:49:36 EST 2019] at > java.net.PlainSocketImpl.socketAccept(Native Method) > [Wed Nov 13 23:49:36 EST 2019] at > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) > [Wed Nov 13 23:49:36 EST 2019] at > java.net.ServerSocket.implAccept(ServerSocket.java:545) > [Wed Nov 13 23:49:36 EST 2019] at > java.net.ServerSocket.accept(ServerSocket.java:513) > [Wed Nov 13 23:49:36 EST 2019] at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400) > [Wed Nov 13 23:49:36 EST 2019] at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372) > [Wed Nov 13 23:49:36 EST 2019] at java.lang.Thread.run(Thread.java:745) > BEServer.log : > > 11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: > [shutdown] Thread: [Thread-176494:334181] Msg:[##### BEServer shutdown was > called ##### : Starting shutdown of BE server. ] > 11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: > [shutDownApacheServer] Thread: [Thread-176494:334181] Msg:[##### WebServer > shutdown was called ##### : Starting shutdown of http server] > 11/13 23:49:41.300 ERROR [com.acmepacket.ems.server.main.BEServer] - Method: > [shutDownApacheServer] Thread: [Thread-176494:334181] Msg:[Exception thrown > in attempting to shutdown Web server. Reason : Cannot run program > "../Apache/httpse > rver/bin/stopHttpServer.sh": error=24, Too many open files] > java.io.IOException: Cannot run program > "../Apache/httpserver/bin/stopHttpServer.sh": error=24, Too many open files > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_74] > at java.lang.Runtime.exec(Runtime.java:620) ~[?:1.8.0_74] > at java.lang.Runtime.exec(Runtime.java:450) ~[?:1.8.0_74] > at java.lang.Runtime.exec(Runtime.java:347) ~[?:1.8.0_74] > at > com.acmepacket.ems.server.main.BEServer.shutDownApacheServer(BEServer.java:989) > [ACMEPacketBEServer.jar:?] > at > com.acmepacket.ems.server.main.BEServer.shutdownWebServers(BEServer.java:954) > [ACMEPacketBEServer.jar:?] > at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:433) > [ACMEPacketBEServer.jar:?] > at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:403) > [ACMEPacketBEServer.jar:?] > at > com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.shutDownAfterFault(ReplicatedXMLDatabaseManagerImpl.java:2536) > [ACMEPacketBEServer.jar:?] > at > com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.handlePanicEvent(ReplicatedXMLDatabaseManagerImpl.java:2568) > [ACMEPacketBEServer.jar:?] > at com.sleepycat.db.internal.DbEnv.handle_panic_event_notify(DbEnv.java:115) > [db.jar:?] > Caused by: java.io.IOException: error=24, Too many open files > at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_74] > at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) ~[?:1.8.0_74] > at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[?:1.8.0_74] > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_74] > ... 10 more -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org