I am using Tomcat 7.0.53 with Spring 4 websockets (spring's SockJS support).
I notice that tomcat opens thousands of files.
Just running for a few hours I noticed, it opened 16,552 files:
root@dripcast:~# lsof -p 22691 | wc -l
16552
Our logs are full of this exception thrown tons of times every second:
org.apache.tomcat.util.net.AprEndpoint$Acceptor run
SEVERE: Socket accept failed
org.apache.tomcat.jni.Error: 24: Too many open files
at org.apache.tomcat.jni.Socket.accept(Native Method)
at
org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:992)
at java.lang.Thread.run(Thread.java:745)
Here is a sample of some of the open files, using lsof.
What can we do to fix this?
java 2056 root *904u sock 0,7 0t0 77846 can't
identify protocol
java 2056 root *905u IPv6 76695 0t0 TCP
dripcast:https->pppoe.95-55-165-198.dynamic.avangarddsl.ru:1376
(ESTABLISHED)
java 2056 root *906u IPv6 74369 0t0 TCP
dripcast:https->fibre.backbonetechnology.com:46415 (ESTABLISHED)
java 2056 root *907u IPv6 76696 0t0 TCP
dripcast:https->outbound.kenshoo.com:14292 (ESTABLISHED)
java 2056 root *908u sock 0,7 0t0 76697 can't
identify protocol
java 2056 root *909u sock 0,7 0t0 76698 can't
identify protocol
java 2056 root *910u 0000 0,9 0 7351 anon_inode
java 2056 root *911u 0000 0,9 0 7351 anon_inode
java 2056 root *912u 0000 0,9 0 7351 anon_inode
java 2056 root *913u 0000 0,9 0 7351 anon_inode
java 2056 root *914u 0000 0,9 0 7351 anon_inode
java 2056 root *915u sock 0,7 0t0 76699 can't
identify protocol
java 2056 root *916u sock 0,7 0t0 76700 can't
identify protocol
java 2056 root *917u sock 0,7 0t0 76701 can't
identify protocol
java 2056 root *918u sock 0,7 0t0 76702 can't
identify protocol
java 2056 root *919u sock 0,7 0t0 77845 can't
identify protocol
java 2056 root *920u sock 0,7 0t0 76705 can't
identify protocol
java 2056 root *921u IPv6 76906 0t0 TCP
dripcast:https->c-67-185-79-130.hsd1.wa.comcast.net:61280 (ESTABLISHED)
java 2056 root *922u 0000 0,9 0 7351 anon_inode
java 2056 root *923u sock 0,7 0t0 77847 can't
identify protocol
java 2056 root *924u sock 0,7 0t0 75196 can't
identify protocol
java 2056 root *925u sock 0,7 0t0 76907 can't
identify protocol
java 2056 root *926u IPv6 76908 0t0 TCP
dripcast:https->cdm-75-108-116-63-nixa.mid.suddenlink.net:56996
(ESTABLISHED)
java 2056 root *927u 0000 0,9 0 7351 anon_inode
java 2056 root *928u IPv6 76909 0t0 TCP
dripcast:https->fibre.backbonetechnology.com:46416 (ESTABLISHED)
java 2056 root *929u sock 0,7 0t0 76910 can't
identify protocol
java 2056 root *930u sock 0,7 0t0 76911 can't
identify protocol
java 2056 root *931u IPv6 76912 0t0 TCP
dripcast:https->outbound.kenshoo.com:14309 (ESTABLISHED)
java 2056 root *932u 0000 0,9 0 7351 anon_inode
java 2056 root *933u 0000 0,9 0 7351 anon_inode
java 2056 root *934u 0000 0,9 0 7351 anon_inode
java 2056 root *935u 0000 0,9 0 7351 anon_inode
java 2056 root *936u 0000 0,9 0 7351 anon_inode
java 2056 root *937u sock 0,7 0t0 73405 can't
identify protocol
java 2056 root *938u sock 0,7 0t0 73406 can't
identify protocol
java 2056 root *939u sock 0,7 0t0 73407 can't
identify protocol
java 2056 root *940u sock 0,7 0t0 71475 can't
identify protocol
java 2056 root *941u sock 0,7 0t0 74371 can't
identify protocol
java 2056 root *942u 0000 0,9 0 7351 anon_inode
java 2056 root *943u sock 0,7 0t0 74372 can't
identify protocol
java 2056 root *944u sock 0,7 0t0 74373 can't
identify protocol
java 2056 root *945u sock 0,7 0t0 74378 can't
identify protocol
java 2056 root *946u 0000 0,9 0 7351 anon_inode
java 2056 root *947u sock 0,7 0t0 71476 can't
identify protocol
java 2056 root *948u sock 0,7 0t0 74379 can't
identify protocol
java 2056 root *949u sock 0,7 0t0 76929 can't
identify protocol
java 2056 root *950u 0000 0,9 0 7351 anon_inode
java 2056 root *951u IPv6 76922 0t0 TCP
dripcast:https->c-67-185-79-130.hsd1.wa.comcast.net:61281 (ESTABLISHED)
java 2056 root *952u sock 0,7 0t0 76923 can't
identify protocol
java 2056 root *953u sock 0,7 0t0 76930 can't
identify protocol
java 2056 root *954u IPv6 76924 0t0 TCP
dripcast:https->outbound.kenshoo.com:14337 (ESTABLISHED)
java 2056 root *955u sock 0,7 0t0 76925 can't
identify protocol
java 2056 root *956u IPv6 76926 0t0 TCP
dripcast:https->kpersfw.state.ks.us:62427 (CLOSE_WAIT)
java 2056 root *957u sock 0,7 0t0 76931 can't
identify protocol
java 2056 root *958u 0000 0,9 0 7351 anon_inode
java 2056 root *959u 0000 0,9 0 7351 anon_inode
java 2056 root *960u 0000 0,9 0 7351 anon_inode
java 2056 root *961u 0000 0,9 0 7351 anon_inode
java 2056 root *962u 0000 0,9 0 7351 anon_inode
Prashant
