Hi Chris, Thanks for your email. Below are replies to your questions: 1. Did you upgrade anything recently (like Java VM)? [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
2. What is error you are seeing? A full stack trace would be helpful. [YJ] The application error can occur in any place with the (Too many open files) error when the limit of open files for tomcat has reached (262144 in my environment), for example, I can get a connectivity error when pulling info from a server as below : [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - Error : The XML configuration file failed to be retrieved for server 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for server=10.1.6.25 error=java.io.FileNotFoundException: /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.2.10.RELEASE.jar (Too many open files) restatus=-1 output= 3. What is your <Connector> configuration? [YJ] It is as below in server.xml: <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1" maxThreads="150" minSpareThreads="25" maxSpareThreads="75" enableLookups="false" acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" compression="on" compressionMinSize="2048" noCompressionUserAgents="gozilla, traviata" compressableMimeType="text/html,text/xml,application/json" redirectPort="443" /> ........... <Connector address="0.0.0.0" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" sslEnabledProtocols="TLSv1.2" ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA" keystoreFile="conf/freestor" keystorePass="freestor"/> <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3" redirectPort="443" secretRequired="false" /> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process? Different kernel versions count things differently. [YJ] I am just getting the process ID of tomcat and count open files for that process 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)? [YJ] Here is the netstat output: Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED tcp 0 0 localhost.loca:postgres localhost.localdo:36850 ESTABLISHED tcp 0 0 localhost.localdo:11753 localhost.localdo:51302 ESTABLISHED tcp 0 0 localhost.loca:postgres localhost.localdo:36844 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48922 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48964 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48968 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60817 TIME_WAIT tcp6 0 0 Yeggy-F8-FMSVA:48968 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48974 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60771 TIME_WAIT tcp6 0 0 Yeggy-F8-FMSVA:48934 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48936 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48954 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48970 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48932 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48938 ESTABLISHED tcp6 0 0 localhost.localdo:51302 localhost.localdo:11753 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48956 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48928 ESTABLISHED tcp6 0 0 localhost.localdo:36844 localhost.loca:postgres ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48930 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 localhost.localdo:35202 localhost.localdo:vrace ESTABLISHED tcp6 0 0 localhost.localdo:36850 localhost.loca:postgres ESTABLISHED tcp6 0 0 localhost.localdo:vrace localhost.localdo:35202 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48966 ESTABLISHED tcp6 0 0 localhost.localdo:51298 localhost.localdo:11753 TIME_WAIT tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48954 ESTABLISHED tcp6 0 1045 Yeggy-F8-FMSVA:54246 172.22.22.192:https ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48970 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48918 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48972 ESTABLISHED tcp6 0 0 localhost.localdo:36846 localhost.loca:postgres ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48960 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48960 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48974 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60714 TIME_WAIT tcp6 0 0 Yeggy-F8-FMSVA:48924 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48924 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48924 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48972 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48926 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48920 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60772 TIME_WAIT tcp6 0 0 Yeggy-F8-FMSVA:48976 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60864 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48916 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48920 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60818 TIME_WAIT tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48976 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48964 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:36321 Yeggy-F8-FMSVA:35998 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48966 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60865 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48952 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60713 TIME_WAIT tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48934 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48958 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48916 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48936 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48962 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48938 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:35998 Yeggy-F8-FMSVA:36321 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48918 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48962 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48926 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48922 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48928 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48930 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48952 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48940 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48958 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48956 ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:48940 Yeggy-F8-FMSVA:vrace ESTABLISHED tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48932 ESTABLISHED udp 0 0 localhost.localdo:41912 localhost.localdo:41912 ESTABLISHED Active UNIX domain sockets (w/o servers) Proto RefCnt Flags Type State I-Node Path unix 2 [ ] DGRAM 520 /var/run/chrony/chronyd.sock unix 3 [ ] DGRAM 14708 /run/systemd/notify unix 2 [ ] DGRAM 14710 /run/systemd/cgroups-agent unix 12 [ ] DGRAM 14720 /run/systemd/journal/dev-log unix 6 [ ] DGRAM 14734 /run/systemd/journal/socket unix 2 [ ] DGRAM 31223 /run/user/0/systemd/notify unix 2 [ ] STREAM CONNECTED 11157518 unix 2 [ ] STREAM CONNECTED 11152644 unix 3 [ ] STREAM CONNECTED 11657 unix 3 [ ] STREAM CONNECTED 22048 /var/lib/sss/pipes/private/sbus-monitor unix 3 [ ] STREAM CONNECTED 545 /run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 11974 /var/lib/sss/pipes/private/sbus-dp_implicit_files.946 unix 2 [ ] STREAM CONNECTED 11367562 unix 3 [ ] DGRAM 31226 unix 3 [ ] STREAM CONNECTED 22051 /var/lib/sss/pipes/private/sbus-dp_implicit_files.946 unix 3 [ ] STREAM CONNECTED 31229 unix 2 [ ] DGRAM 551 unix 3 [ ] STREAM CONNECTED 23869 /run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 11879 unix 2 [ ] STREAM CONNECTED 800347 unix 2 [ ] DGRAM 23301 unix 3 [ ] STREAM CONNECTED 11819 unix 2 [ ] DGRAM 246 unix 3 [ ] STREAM CONNECTED 543 unix 3 [ ] STREAM CONNECTED 23109 /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 11365631 unix 2 [ ] DGRAM 31213 unix 3 [ ] STREAM CONNECTED 11363838 unix 3 [ ] STREAM CONNECTED 12063 unix 3 [ ] STREAM CONNECTED 797101 unix 2 [ ] DGRAM 11659 unix 3 [ ] DGRAM 14712 unix 3 [ ] STREAM CONNECTED 23548 unix 3 [ ] STREAM CONNECTED 11880 /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 23325 unix 3 [ ] STREAM CONNECTED 248 unix 3 [ ] STREAM CONNECTED 11973 unix 2 [ ] STREAM CONNECTED 32574 unix 3 [ ] STREAM CONNECTED 507 unix 3 [ ] STREAM CONNECTED 21947 /run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 24095 unix 3 [ ] STREAM CONNECTED 18281 /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 11001952 unix 2 [ ] DGRAM 11971 unix 3 [ ] STREAM CONNECTED 12134 /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 11325964 unix 3 [ ] STREAM CONNECTED 23050 unix 2 [ ] STREAM CONNECTED 11365500 unix 3 [ ] STREAM CONNECTED 24096 /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 11143790 unix 2 [ ] STREAM CONNECTED 793273 unix 3 [ ] STREAM CONNECTED 23868 unix 3 [ ] STREAM CONNECTED 11820 /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 11152638 unix 3 [ ] STREAM CONNECTED 24676 /run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 23167 unix 3 [ ] STREAM CONNECTED 21912 unix 3 [ ] STREAM CONNECTED 543420 unix 3 [ ] DGRAM 31225 unix 3 [ ] STREAM CONNECTED 564 unix 3 [ ] STREAM CONNECTED 12133 unix 3 [ ] STREAM CONNECTED 12064 /var/lib/sss/pipes/private/sbus-monitor unix 3 [ ] STREAM CONNECTED 18285 /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 808539 unix 2 [ ] DGRAM 23542 unix 3 [ ] STREAM CONNECTED 22708 unix 2 [ ] DGRAM 11969 unix 3 [ ] STREAM CONNECTED 18279 /run/systemd/journal/stdout unix 3 [ ] DGRAM 14713 unix 2 [ ] DGRAM 34975 unix 2 [ ] STREAM CONNECTED 11353853 unix 3 [ ] STREAM CONNECTED 11156644 /run/systemd/journal/stdout unix 2 [ ] DGRAM 21925 unix 2 [ ] DGRAM 22033 unix 3 [ ] STREAM CONNECTED 23168 /run/systemd/journal/stdout unix 2 [ ] DGRAM 24657 unix 3 [ ] STREAM CONNECTED 543421 unix 2 [ ] STREAM CONNECTED 793268 unix 3 [ ] STREAM CONNECTED 22047 unix 3 [ ] STREAM CONNECTED 782556 /run/systemd/journal/stdout unix 2 [ ] DGRAM 21490 unix 3 [ ] STREAM CONNECTED 23698 unix 2 [ ] DGRAM 31210 unix 3 [ ] STREAM CONNECTED 11141980 unix 2 [ ] STREAM CONNECTED 11344541 unix 2 [ ] STREAM CONNECTED 11353876 unix 2 [ ] STREAM CONNECTED 11367557 unix 3 [ ] STREAM CONNECTED 247 unix 3 [ ] STREAM CONNECTED 544 unix 3 [ ] STREAM CONNECTED 34094 /run/dbus/system_bus_socket unix 2 [ ] STREAM CONNECTED 25346 unix 3 [ ] DGRAM 11664 unix 3 [ ] STREAM CONNECTED 21861 unix 3 [ ] STREAM CONNECTED 13539 /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 565 /var/lib/sss/pipes/private/sbus-monitor unix 2 [ ] DGRAM 24744 unix 3 [ ] STREAM CONNECTED 18283 /run/systemd/journal/stdout unix 2 [ ] DGRAM 13499 unix 2 [ ] STREAM CONNECTED 31209 unix 2 [ ] STREAM CONNECTED 32883 unix 3 [ ] DGRAM 11663 unix 3 [ ] STREAM CONNECTED 28573 /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 561 /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 11363839 unix 3 [ ] STREAM CONNECTED 11370377 /var/lib/sss/pipes/nss unix 3 [ ] STREAM CONNECTED 22007 unix 2 [ ] DGRAM 28344 unix 2 [ ] DGRAM 23311 unix 3 [ ] STREAM CONNECTED 11932 unix 2 [ ] STREAM CONNECTED 11143887 unix 3 [ ] STREAM CONNECTED 31205 unix 2 [ ] DGRAM 32811 unix 2 [ ] DGRAM 513 Active Bluetooth connections (w/o servers) Proto Destination Source State PSM DCID SCID IMTU OMTU Security Proto Destination Source State Channel Thanks -----Original Message----- From: Christopher Schultz <ch...@christopherschultz.net> Sent: Monday, May 17, 2021 5:18 PM To: users@tomcat.apache.org Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 Yeggy, On 5/17/21 15:17, Yeggy Javadi wrote: > Hi, > I am seeing too many TCP sockets open by tomcat on Oracle Linux 8: > java 51798 root 6601u sock 0,9 0t0 3959342 > protocol: TCPv6 > > I was running the same application on Linux 7.6 with the same > apache-tomcat version without seeing this issue. I saw that RedHat has > removed apache-tomcat fom their Linux 8 distribution. Please help with > throubleshooting and indicating if there is any changes in Linux 8 > that can explain this > > Configuration: > apache-tomcat-8.5.59 > OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application > using Postgres and elastic search databases > > > # ps -ef | grep tomcat > root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java > -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 > -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dignore.endorsed.dirs= -classpath > /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar > -Dcatalina.base=/usr/local/apache-tomcat > -Dcatalina.home=/usr/local/apache-tomcat > -Djava.io.tmpdir=/usr/local/apache-tomcat/temp > org.apache.catalina.startup.Bootstrap start > root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat > > > # cat /proc/51798/limits > Limit Soft Limit Hard Limit Units > Max cpu time unlimited unlimited seconds > Max file size unlimited unlimited bytes > Max data size unlimited unlimited bytes > Max stack size 8388608 unlimited bytes > Max core file size unlimited unlimited bytes > Max resident set unlimited unlimited bytes > Max processes 256518 256518 processes > Max open files 262144 262144 files > Max locked memory 65536 65536 bytes > Max address space unlimited unlimited bytes > Max file locks unlimited unlimited locks > Max pending signals 256518 256518 signals > Max msgqueue size 819200 819200 bytes > Max nice priority 0 0 > Max realtime priority 0 0 > Max realtime timeout unlimited unlimited us > > > Too may open files: > # lsof -p 51798 | wc -l > 6679 > > > Most of open files are TCP sockets: > java 51798 root 6601u sock 0,9 0t0 3959342 > protocol: TCPv6 > > > # lsof -p 51798 | grep "protocol: TCPv6" | wc -l > 6413 Some potentially dumb questions: 1. Did you upgrade anything recently (like Java VM)? 2. What is error you are seeing? A full stack trace would be helpful. 3. What is your <Connector> configuration? 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process? Different kernel versions count things differently. 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)? -chris --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org