Hi Tom,
you may have a too low ulimit setup. It defaults to 1024, and each time
you open a socket, that eats a file descriptor.
You have options, though :
- either you increase this ulimit, or set it to unlimited,
- or you lower the tcp-time-wait value so that the sockets are closed
faster.
Socket usually are kept open for a little while (I think it's 300
seconds on linux, but not sure), waiting for the client to close the
connection. Going down to 30 seconds should be OK.
Hope it helps
Tom Wieczorek wrote:
Hi all,
I encountered an interesting problem today, in which MINA didn't
accept any incoming connections any longer, because we had too many
open files on the server machine.
The application in question uses MINA version 2.0.0-M1 with 3
different IoAcceptor services sharing the same Executor and
IoProcessor thread pool. The first two services are designed to have
typically 5-10 permanent IoSessions connected per service. These two
services don't have an ExecutionFilter in their filter chain and are
driven by the IoProcessor theads. The third service is a HTTP
Request/Response service based on AsyncWeb, using an ExecutorFilter
with a dedicated HTTP thread pool.
After running for several weeks without any problems, the app stooped
responding to new connections and the log file was spammed with the
Exception Monitor log entries:
WARN [NioSocketAcceptor-3]
org.apache.mina.common.DefaultExceptionMonitor: java.io.IOException:
Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSocketAcceptor.java:153)
at
org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSocketAcceptor.java:47)
at
org.apache.mina.common.AbstractPollingIoAcceptor$Worker.processHandles(AbstractPollingIoAcceptor.java:305)
at
org.apache.mina.common.AbstractPollingIoAcceptor$Worker.run(AbstractPollingIoAcceptor.java:243)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
After further investigation using "lsof -p", I really found approx.
1000 sockets in CLOSE_WAIT state that didn't get released. The sockets
were mainly bound to the HTTP service. Note that the already open
sessions on the ohter two services worked without any problems
(messages were processed normally). Also interesting: The app has a
Connection Limit by IP address enforced by an IoFilter. The filter
prints the managed session count for the IoService in its DEBUG log
messages when sessionClosed() is invoked. The last DEBUG entry of this
kind close before the max file limit was reached said that the managed
sessions count of the HTTP service was 0. I couldn't detect a missing
coseSession() event in the Logs either.
What circumstances could lead to the effect that MINA doesn't
close/release sockets that have been colsed remotely? The processors
seemed to work (the other sessions were still active), the acceptors
spammed the exceptions. Any ideas?
Some enviromental information:
Linux 2.6.16.21-0.25-smp #1 SMP Tue Sep 19 07:26:15 UTC 2006 x86_64
x86_64 x86_64 GNU/Linux
java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_11-b03, mixed mode)
Thanks for your help.
Cheers,
Tom.
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
