Re: apache-tomcat-8.5.59 too many open files on Linux 8

[Christopher Schultz]

1835053 /apps/apache-tomcat/logs/fms.log
java    165217 root  111w  REG    8,6   919042  
1835019 /apps/apache-tomcat/logs/obd.log
java    165217 root  112u sock    0,9  0t0 
16823821 protocol: TCPv6
java    165217 root  113u sock    0,9  0t0 
16823822 protocol: TCPv6
java    165217 root  114u sock    0,9  0t0 
16823633 protocol: TCPv6
java    165217 root  115u sock    0,9  0t0 
16823527 protocol: TCPv6
java    165217 root  116u sock    0,9  0t0 
16823489 protocol: TCPv6
java    165217 root  117u sock    0,9  0t0 
16823490 protocol: TCPv6
java    165217 root  118u sock    0,9  0t0 
16823491 protocol: TCPv6

..

Thanks

-Original Message-
From: André Warnier (tomcat/perl) 
Sent: Tuesday, May 25, 2021 12:42 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.
The point is to try to figure out what these thousands of apparently 
"TCPv6" sockets
belonging to the tomcat process actually are, so that we can maybe 
begin to look at where

they may be coming from.
The trouble is, the lsof output so far did not really tell us what 
these "sock" things

might be.

But there may be a clue here :
https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines 


(about when things run in a container).
Is that your case ?
And if so, could you run the lsof command in the container, as they 
suggest ?


And the point of forcing a tomcat/JVM GC was this :
When you restart tomcat (actually the JVM which runs tomcat), the OS 
will clean up *all*
the file descriptors belonging to that process, including the 
"legitimate" ones shown by

netstat, and the "unknown" ones shown in addition by lsof.
Doing a GC, without stopping the JVM, would clean up *only* such 
sockets/fd that are held
by objects which are discarded, but still sit on the heap awaiting a 
GC to really destroy
them.  If your heap is very large, it may otherwise take a long while 
before such a GC

happens, and such sockets could accumulate.
One way to trigger a GC is through JMX, but it takes a bit of 
additional setup to make

that work. That's why I was asking if you had some method to do that.
(see : https://code.google.com/archive/p/jmxsh/)
But let's look at the lsof part first.



On 24.05.2021 16:09, Yeggy Javadi wrote:

Hi,
I restarted tomcat so PID has changed to 143152; I do not know how to 
trigger tomcat GC, I just restart it to reset the lsof to 0.

Please see outputs below:

# ps -ef | grep tomcat
root  143152   1  0 May22 ?    00:26:44 
/usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
-d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar 
-Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root  153962  153912  0 10:00 pts/1    00:00:00 grep --color=auto 
tomcat


# lsof -p 143152 | wc -l
41043

# lsof -p 143152 | grep "protocol: TCPv6"| wc -l
40487

# netstat -p -a -n --inet6 | grep 143152
tcp6   0  0 :::8443 :::*
LISTEN  143152/java
tcp6   0  0 :::443  :::*
LISTEN  143152/java
tcp6   0  0 127.0.0.1:8005  :::*
LISTEN  143152/java
tcp6   0  0 :::1099 :::*
LISTEN  143152/java
tcp6   0  0 :::80   :::*
LISTEN  143152/java
tcp6   0  0 :::36081    :::*
LISTEN  143152/java
tcp6   0  0 10.4.3.55:60736 10.4.3.55:9300  
ESTABLISHED 143152/java
tcp6   0  0 10.4.3.55:60732 10.4.3.55:9300  
ESTABLISHED 143152/java
tcp6   0  0 10.4.3.55:60728 10.4.3.55:9300  
ESTABLISHED 143152/java
tcp6   0  0 10.4.3.55:80    10.197.255.10:55446 
ESTABLISHED 143152/java
tcp6   1  0 10.4.3.55:55958 10.4.3.55:11576 
CLOSE_WAIT  143152/java
tcp6   0  0 10.4.3.55:53682 172.22.21.48:443
ESTABLISHED 143152/java
tcp6   0  0 127.0.0.1:48622 127.0.0.1:5432  
ESTABLISHED 143152/java
tcp6   0  0 10.4.3.55:60748 10.4.3.55:9300  
ESTABLISHED 143152/java
tcp6   1  0 10.4.3.55:55956 1

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[tomcat/perl]

 0t0  TCP 127.0.0.1:mxi 
(LISTEN)

java    165217 root   94u sock    0,9  0t0 16963075 
protocol: TCPv6
java    165217 root   96u sock    0,9  0t0 16824649 
protocol: TCPv6
java    165217 root   97u IPv6   17051898  0t0  TCP 
10.4.3.55:60204->172.22.22.192:https (ESTABLISHED)

java    165217 root   98u sock    0,9  0t0 16824673 
protocol: TCPv6
java    165217 root   99u sock    0,9  0t0 16824711 
protocol: TCPv6
java    165217 root  100u sock    0,9  0t0 16893245 
protocol: TCPv6
java    165217 root  101u sock    0,9  0t0 16824385 
protocol: TCPv6
java    165217 root  102u sock    0,9  0t0 16824264 
protocol: TCPv6
java    165217 root  103u sock    0,9  0t0 16824278 
protocol: TCPv6
java    165217 root  104u sock    0,9  0t0 16824279 
protocol: TCPv6
java    165217 root  105u sock    0,9  0t0 16824063 
protocol: TCPv6
java    165217 root  106u sock    0,9  0t0 16823982 
protocol: TCPv6
java    165217 root  107u sock    0,9  0t0 16823959 
protocol: TCPv6
java    165217 root  108r  CHR    1,8  0t0 1041 
/dev/random
java    165217 root  109u sock    0,9  0t0 16778709 
protocol: TCPv6
java    165217 root  110w  REG    8,6 25506176  1835053 
/apps/apache-tomcat/logs/fms.log
java    165217 root  111w  REG    8,6   919042  1835019 
/apps/apache-tomcat/logs/obd.log

java    165217 root  112u sock    0,9  0t0 16823821 
protocol: TCPv6
java    165217 root  113u sock    0,9  0t0 16823822 
protocol: TCPv6
java    165217 root  114u sock    0,9  0t0 16823633 
protocol: TCPv6
java    165217 root  115u sock    0,9  0t0 16823527 
protocol: TCPv6
java    165217 root  116u sock    0,9  0t0 16823489 
protocol: TCPv6
java    165217 root  117u sock    0,9  0t0 16823490 
protocol: TCPv6
java    165217 root  118u sock    0,9  0t0 16823491 
protocol: TCPv6
..

Thanks

-Original Message-
From: André Warnier (tomcat/perl) 
Sent: Tuesday, May 25, 2021 12:42 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.
The point is to try to figure out what these thousands of apparently "TCPv6" 
sockets
belonging to the tomcat process actually are, so that we can maybe begin to 
look at where
they may be coming from.
The trouble is, the lsof output so far did not really tell us what these "sock" 
things
might be.

But there may be a clue here :
https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines
(about when things run in a container).
Is that your case ?
And if so, could you run the lsof command in the container, as they suggest ?

And the point of forcing a tomcat/JVM GC was this :
When you restart tomcat (actually the JVM which runs tomcat), the OS will clean 
up *all*
the file descriptors belonging to that process, including the "legitimate" ones 
shown by
netstat, and the "unknown" ones shown in addition by lsof.
Doing a GC, without stopping the JVM, would clean up *only* such sockets/fd 
that are held
by objects which are discarded, but still sit on the heap awaiting a GC to 
really destroy
them.  If your heap is very large, it may otherwise take a long while before 
such a GC
happens, and such sockets could accumulate.
One way to trigger a GC is through JMX, but it takes a bit of additional setup 
to make
that work. That's why I was asking if you had some method to do that.
(see : https://code.google.com/archive/p/jmxsh/)
But let's look at the lsof part first.



On 24.05.2021 16:09, Yeggy Javadi wrote:

Hi,
I restarted tomcat so PID has changed to 143152; I do not know how to trigger tomcat 
GC, I just restart it to reset the lsof to 0.

Please see outputs below:

# ps -ef | grep tomcat
root  143152   1  0 May22 ?    00:26:44 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= 
-classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar 
-Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

root  153962  153912  0 10:00 pts/1    00:00:00 grep --color=auto tomcat

# lsof -p 143152 | wc -l
41043

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[tomcat/perl]

 
protocol: TCPv6
java165217 root  106u sock0,9  0t0 16823982 
protocol: TCPv6
java165217 root  107u sock0,9  0t0 16823959 
protocol: TCPv6
java165217 root  108r  CHR1,8  0t0 1041 
/dev/random
java165217 root  109u sock0,9  0t0 16778709 
protocol: TCPv6
java165217 root  110w  REG8,6 25506176  1835053 
/apps/apache-tomcat/logs/fms.log
java165217 root  111w  REG8,6   919042  1835019 
/apps/apache-tomcat/logs/obd.log
java165217 root  112u sock0,9  0t0 16823821 
protocol: TCPv6
java165217 root  113u sock0,9  0t0 16823822 
protocol: TCPv6
java165217 root  114u sock0,9  0t0 16823633 
protocol: TCPv6
java165217 root  115u sock0,9  0t0 16823527 
protocol: TCPv6
java165217 root  116u sock0,9  0t0 16823489 
protocol: TCPv6
java165217 root  117u sock0,9  0t0 16823490 
protocol: TCPv6
java165217 root  118u sock0,9  0t0 16823491 
protocol: TCPv6
..

Thanks

-Original Message-
From: André Warnier (tomcat/perl) 
Sent: Tuesday, May 25, 2021 12:42 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.
The point is to try to figure out what these thousands of apparently "TCPv6" 
sockets
belonging to the tomcat process actually are, so that we can maybe begin to 
look at where
they may be coming from.
The trouble is, the lsof output so far did not really tell us what these "sock" 
things
might be.

But there may be a clue here :
https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines
(about when things run in a container).
Is that your case ?
And if so, could you run the lsof command in the container, as they suggest ?

And the point of forcing a tomcat/JVM GC was this :
When you restart tomcat (actually the JVM which runs tomcat), the OS will clean 
up *all*
the file descriptors belonging to that process, including the "legitimate" ones 
shown by
netstat, and the "unknown" ones shown in addition by lsof.
Doing a GC, without stopping the JVM, would clean up *only* such sockets/fd 
that are held
by objects which are discarded, but still sit on the heap awaiting a GC to 
really destroy
them.  If your heap is very large, it may otherwise take a long while before 
such a GC
happens, and such sockets could accumulate.
One way to trigger a GC is through JMX, but it takes a bit of additional setup 
to make
that work. That's why I was asking if you had some method to do that.
(see : https://code.google.com/archive/p/jmxsh/)
But let's look at the lsof part first.



On 24.05.2021 16:09, Yeggy Javadi wrote:

Hi,
I restarted tomcat so PID has changed to 143152; I do not know how to trigger 
tomcat GC, I just restart it to reset the lsof to 0.
Please see outputs below:

# ps -ef | grep tomcat
root  143152   1  0 May22 ?00:26:44 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root  153962  153912  0 10:00 pts/100:00:00 grep --color=auto tomcat

# lsof -p 143152 | wc -l
41043

# lsof -p 143152 | grep "protocol: TCPv6"| wc -l
40487

# netstat -p -a -n --inet6 | grep 143152
tcp6   0  0 :::8443 :::*LISTEN  
143152/java
tcp6   0  0 :::443  :::*LISTEN  
143152/java
tcp6   0  0 127.0.0.1:8005  :::*LISTEN  
143152/java
tcp6   0  0 :::1099 :::*LISTEN  
143152/java
tcp6   0  0 :::80   :::*LISTEN  
143152/java
tcp6   0  0 :::36081:::*LISTEN  
143152/java
tcp6   0  0 10.4.3.55:60736 10.4.3.55:9300  ESTABLISHED 
143152/java
tcp6   0  0 10.4.3.55:60732 10.4.3.55:9300  ESTABLISHED 
143152/java
tcp6   0  0 10.4.3.55:60728 10.4.3.55:9300  ESTABLISHED 
143152/java
tcp6   0  0 10.4.3.55:8010.197.255.10:55446 ESTABLISHED 
143152/java
tcp6

RE: apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

-INF/lib/spring-jms-3.2.17.RELEASE.jar
java165217 root  mem   REG8,264021   160837 
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-join-4.10.4.jar
java165217 root  mem   REG8,2   714194   160828 
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/javassist-3.18.1-GA.jar
java165217 root  mem   REG8,2   279193   160876 
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang-2.5.jar
java165217 root  mem   REG8,275311   160826 
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-commons-annotations-4.0.4.Final.jar
java165217 root  mem   REG8,2   436303   160745 
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/avro-1.7.7.jar

java165217 root   73w FIFO   0,13  0t0 16786125 pipe
java165217 root   74u  a_inode   0,14010970 
[eventpoll]
java165217 root   75r FIFO   0,13  0t0 16786126 pipe
java165217 root   76w FIFO   0,13  0t0 16786126 pipe
java165217 root   77u  a_inode   0,14010970 
[eventpoll]
java165217 root   78u sock0,9  0t0 16961335 
protocol: TCPv6
java165217 root   79r FIFO   0,13  0t0 16786127 pipe
java165217 root   80w FIFO   0,13  0t0 16786127 pipe
java165217 root   81u  a_inode   0,14010970 
[eventpoll]
java165217 root   82r FIFO   0,13  0t0 16786128 pipe
java165217 root   83w FIFO   0,13  0t0 16786128 pipe
java165217 root   84u  a_inode   0,14010970 
[eventpoll]
java165217 root   86r FIFO   0,13  0t0 16786129 pipe
java165217 root   87w FIFO   0,13  0t0 16786129 pipe
java165217 root   88u  a_inode   0,14010970 
[eventpoll]
java165217 root   89r FIFO   0,13  0t0 16786130 pipe
java165217 root   90w FIFO   0,13  0t0 16786130 pipe
java165217 root   91u  a_inode   0,14010970 
[eventpoll]
java165217 root   93u IPv6   16786133  0t0  TCP 
127.0.0.1:mxi (LISTEN)
java165217 root   94u sock0,9  0t0 16963075 
protocol: TCPv6
java165217 root   96u sock0,9  0t0 16824649 
protocol: TCPv6
java165217 root   97u IPv6   17051898  0t0  TCP 
10.4.3.55:60204->172.22.22.192:https (ESTABLISHED)
java165217 root   98u sock0,9  0t0 16824673 
protocol: TCPv6
java165217 root   99u sock0,9  0t0 16824711 
protocol: TCPv6
java165217 root  100u sock0,9  0t0 16893245 
protocol: TCPv6
java165217 root  101u sock0,9  0t0 16824385 
protocol: TCPv6
java165217 root  102u sock0,9  0t0 16824264 
protocol: TCPv6
java165217 root  103u sock0,9  0t0 16824278 
protocol: TCPv6
java165217 root  104u sock0,9  0t0 16824279 
protocol: TCPv6
java165217 root  105u sock0,9  0t0 16824063 
protocol: TCPv6
java165217 root  106u sock0,9  0t0 16823982 
protocol: TCPv6
java165217 root  107u sock0,9  0t0 16823959 
protocol: TCPv6
java165217 root  108r  CHR1,8  0t0 1041 
/dev/random
java165217 root  109u sock0,9  0t0 16778709 
protocol: TCPv6
java165217 root  110w  REG8,6 25506176  1835053 
/apps/apache-tomcat/logs/fms.log
java165217 root  111w  REG8,6   919042  1835019 
/apps/apache-tomcat/logs/obd.log
java165217 root  112u sock0,9  0t0 16823821 
protocol: TCPv6
java165217 root  113u sock0,9  0t0 16823822 
protocol: TCPv6
java165217 root  114u sock0,9  0t0 16823633 
protocol: TCPv6
java165217 root  115u sock0,9  0t0 16823527 
protocol: TCPv6
java165217 root  116u sock0,9  0t0 16823489 
protocol: TCPv6
java165217 root  117u sock0,9  0t0 16823490 
protocol: TCPv6
java165217 root  118u sock0,9  0t0 16823491 
protocol: TCPv6
..

Thanks

-Original Message-
From: André Warnier (tomcat/perl)  
Sent: Tuesday, May 25, 2021 12:42 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.
The point is to try to figure out what these thousands of apparently "TCPv6" 
sockets 
belonging to the tomcat process actually are, so that we can maybe begin to 
look at where 
they 

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[tomcat/perl]

HED)
java143152 root *818u  IPv6 15602584  0t0  TCP 
Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)

Thank

-Original Message-----
From: André Warnier (tomcat/perl) 
Sent: Sunday, May 23, 2021 6:30 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.

I have no idea what lines like this are :

java130244 root   78u sock0,9  0t0 12154374 
protocol: TCPv6


There are obviously too many of them, for them to match with the sockets listed 
by netstat.

The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond 
to the
ones marked as "LISTEN" in the netstat output.
But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.

Could you redo a netstat as follows :
# netstat -p -a -n --inet6 | grep 130244

and can you also try this form of lsof :
# lsof -a -p 130244 -T s -i6

And finally (after copying the result of the above) : do you know how to 
trigger a GC
(Garbage Collection) in your tomcat JVM ?
(the point is to see if when a GC happens, these things disappear).


On 22.05.2021 18:03, Yeggy Javadi wrote:

Here it is:

# netstat -p -a  --tcp | grep 130244
tcp6   0  0 [::]:pcsync-https   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:https  [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:37537  [::]:*  LISTEN  
130244/java
tcp6   0  0 localhost.localdoma:mxi [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:8009   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:rmiregistry[::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:http   [::]:*  LISTEN  
130244/java
tcp6  86  0 Yeggy-F8-FMSVA:39680172.22.22.192:https CLOSE_WAIT  
130244/java
tcp6   0  1 Yeggy-F8-FMSVA:5361810.12.3.78:httpsSYN_SENT
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54772Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54782Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54766Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54778Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54788Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54770Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54790Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54776Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54786Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54780Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54768Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54784Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  1 Yeggy-F8-FMSVA:4292210.12.3.77:httpsSYN_SENT
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:35794172.22.22.192:https ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54774Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 
130244/java


# lsof -p 130244
COMMANDPID USER   FD  TYPE DEVICE SIZE/OFF NODE NAME
java130244 root  cwd   DIR8,2 4096   157664 
/usr/local/freestor/bin
java130244 root  rtd   DIR8,3 40962 /
java130244 root  txt   REG8,2 8712 8913 
/usr/local/jdk/jre1.8.0_271/bin/java
java130244 root  mem   REG8,2   498864 9007 
/usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
java130244 root  mem   REG8,239176 9006 
/usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
java130244 root  mem   REG8,2   759184 8996 
/usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
java130244 root  mem   REG8,2  3559360 9139 
/usr/local/jdk/jre1.8.0_271/lib/resources.jar
java130244 root  mem   REG8,299680   133076

RE: apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

 0t0  TCP 
localhost.localdomain:52312->localhost.localdomain:postgres (ESTABLISHED)
java143152 root  215u  IPv6 13822102  0t0  TCP 
localhost.localdomain:48620->localhost.localdomain:postgres (ESTABLISHED)
java143152 root  216u  IPv6 13823064  0t0  TCP 
localhost.localdomain:48624->localhost.localdomain:postgres (ESTABLISHED)
java143152 root  217u  IPv6 13821107  0t0  TCP 
localhost.localdomain:48622->localhost.localdomain:postgres (ESTABLISHED)
java143152 root  269u  IPv6 13824001  0t0  TCP 
localhost.localdomain:46976->localhost.localdomain:vrace (ESTABLISHED)
java143152 root  270u  IPv6 13787907  0t0  TCP 
Yeggy-F8-FMSVA:60726->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  271u  IPv6 13787908  0t0  TCP 
Yeggy-F8-FMSVA:60728->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  272u  IPv6 13787909  0t0  TCP 
Yeggy-F8-FMSVA:60730->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  273u  IPv6 13787910  0t0  TCP 
Yeggy-F8-FMSVA:60732->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  274u  IPv6 13787911  0t0  TCP 
Yeggy-F8-FMSVA:60734->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  275u  IPv6 13787912  0t0  TCP 
Yeggy-F8-FMSVA:60736->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  276u  IPv6 13787913  0t0  TCP 
Yeggy-F8-FMSVA:60738->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  277u  IPv6 13787914  0t0  TCP 
Yeggy-F8-FMSVA:60740->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  278u  IPv6 13787915  0t0  TCP 
Yeggy-F8-FMSVA:60742->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  279u  IPv6 13787916  0t0  TCP 
Yeggy-F8-FMSVA:60744->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  280u  IPv6 13787917  0t0  TCP 
Yeggy-F8-FMSVA:60746->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  281u  IPv6 13787918  0t0  TCP 
Yeggy-F8-FMSVA:60748->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  282u  IPv6 13787919  0t0  TCP 
Yeggy-F8-FMSVA:60750->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java143152 root  293u  IPv6 13821122  0t0  TCP *:rmiregistry (LISTEN)
java143152 root  294u  IPv6 13816318  0t0  TCP *:36081 (LISTEN)
java143152 root  304u  IPv6 13787923  0t0  TCP *:pcsync-https (LISTEN)
java143152 root *740u  IPv6 15607907  0t0  TCP 
Yeggy-F8-FMSVA:53366->172.22.21.48:https (ESTABLISHED)
java143152 root *876u  IPv6 15608779  0t0  TCP 
Yeggy-F8-FMSVA:40328->172.22.21.47:https (ESTABLISHED)
java143152 root *881u  IPv6 15614275  0t0  TCP 
Yeggy-F8-FMSVA:59970->172.22.22.192:https (ESTABLISHED)
java143152 root *882u  IPv6 15558639  0t0  TCP 
Yeggy-F8-FMSVA:53526->172.22.21.48:https (ESTABLISHED)
java143152 root *067u  IPv6 15602583  0t0  TCP 
Yeggy-F8-FMSVA:http->10.197.255.10:55037 (ESTABLISHED)
java143152 root *070u  IPv6 15602586  0t0  TCP 
Yeggy-F8-FMSVA:40318->Yeggy-F8-FMSVA:36081 (ESTABLISHED)
java143152 root *073u  IPv6 15602582  0t0  TCP 
Yeggy-F8-FMSVA:http->10.197.255.10:55036 (ESTABLISHED)
java143152 root *080u  IPv6 15512947  0t0  TCP 
Yeggy-F8-FMSVA:55956->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
java143152 root *081u  IPv6 15512948  0t0  TCP 
Yeggy-F8-FMSVA:55958->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
java143152 root *817u  IPv6 15611323  0t0  TCP 
Yeggy-F8-FMSVA:36081->Yeggy-F8-FMSVA:40318 (ESTABLISHED)
java143152 root *818u  IPv6 15602584  0t0  TCP 
Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)

Thank

-Original Message-
From: André Warnier (tomcat/perl)  
Sent: Sunday, May 23, 2021 6:30 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.

I have no idea what lines like this are :
> java130244 root   78u sock0,9  0t0 12154374 
> protocol: TCPv6

There are obviously too many of them, for them to match with the sockets listed 
by netstat.

The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond 
to the 
ones marked as "LISTEN" in the netstat output.
But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.

Could you redo a netstat as follows :
# netstat -p -a -n --inet6 | grep 130244

and can you also try this form of lsof :
# lsof -a -p 130244 -T s -i6

And finally (after copying the result of the above) : do you know how to 
trigger a GC 
(Garbage Collection) in your tomcat JVM ?
(the point is to see if when a GC happens, these things disappear).


On 22.05.2021 18:03, Yeggy Javadi wrote:
> Here it is:
> 
> # netstat -p -a  --tcp | grep 130244
> tcp6   0  0 [::]:pcsync-https   [::]:*  LISTEN
>   130244/java
> tcp6   0  0 [::]:https  [::]:*  LISTEN
>   130244/java
> tcp6  

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[tomcat/perl]

30244 root  115u sock0,9  0t0 11987351 
protocol: TCPv6
java130244 root  116u sock0,9  0t0 11987352 
protocol: TCPv6
java130244 root  117u sock0,9  0t0 11987353 
protocol: TCPv6
java130244 root  118u sock0,9  0t0 11987354 
protocol: TCPv6
java130244 root  119u sock0,9  0t0 11987355 
protocol: TCPv6
java130244 root  120u sock0,9  0t0 11987356 
protocol: TCPv6
java130244 root  121u sock0,9  0t0 11987357 
protocol: TCPv6
java130244 root  122u sock0,9  0t0 11987358 
protocol: TCPv6
java130244 root  123u sock0,9  0t0 11987359 
protocol: TCPv6
java130244 root  124u sock0,9  0t0 11987360 
protocol: TCPv6
...
Keep seeing the same line

-Original Message-
From: André Warnier (tomcat/perl) 
Sent: Saturday, May 22, 2021 10:22 AM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Mmm. Nothing very special in that netstat output.
The sockets seen there look quite normal for tomcat, and there are not a lot.
What about the IPv4 sockets ? (remove the -6 in your netstat command)

Looks like lsof is counting things which are not IPv6 TCP sockets belonging to 
the tomcat JVM process.
Where is the difference between the lsof count (19948) and the netstat count 
(25) ?
(genuine question, I'm puzzled too)

Can you give an example of the "lsof -p 130244" output lines ?
(not all 19948 please, but enough to see some variety)

On 21.05.2021 16:11, Yeggy Javadi wrote:

Hi,
Here its is:
# lsof -p 130244 | grep "protocol: TCPv6"| wc -l
19948

# netstat -p -a -6 --tcp | grep 130244
tcp6   0  0 [::]:pcsync-https   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:https  [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:37537  [::]:*  LISTEN  
130244/java
tcp6   0  0 localhost.localdoma:mxi [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:8009   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:rmiregistry[::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:http   [::]:*  LISTEN  
130244/java
tcp6  86  0 Yeggy-F8-FMSVA:39680172.22.22.192:https CLOSE_WAIT  
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54772Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54782Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54766Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54778Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54788Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54770Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54790Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54776Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54786Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54780Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54768Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54784Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54774Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 
130244/java
Thanks
-Original Message-
From: Noelette Stout 
Sent: Friday, May 21, 2021 8:28 AM
To: Tomcat Users List 
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

ss has all the same options as netstat

On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)

wrote:


Soyrry to top-post, but it's getting cluttered down there.

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[Felix Schumacher]

   0,13  0t0 11968848 pipe
>> java130244 root   83w FIFO   0,13  0t0 11968848 pipe
>> java130244 root   84u  a_inode   0,14010970 
>> [eventpoll]
>> java130244 root   86r FIFO   0,13  0t0 11968849 pipe
>> java130244 root   87w FIFO   0,13  0t0 11968849 pipe
>> java130244 root   88u  a_inode   0,14010970 
>> [eventpoll]
>> java130244 root   89r FIFO   0,13  0t0 11968850 pipe
>> java130244 root   90w FIFO   0,13  0t0 11968850 pipe
>> java130244 root   91u  a_inode   0,14010970 
>> [eventpoll]
>> java130244 root   93u IPv6   11968853  0t0  TCP 
>> localhost.localdomain:mxi (LISTEN)
>> java130244 root   94u IPv6   12298916  0t0  TCP 
>> localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
>> java130244 root   95u sock0,9  0t0 12017983 
>> protocol: TCPv6
>> java130244 root   96u sock0,9  0t0 11983086 
>> protocol: TCPv6
>> java130244 root   97u sock0,9  0t0 11983087 
>> protocol: TCPv6
>> java130244 root   98u sock0,9  0t0 11983088 
>> protocol: TCPv6
>> java130244 root   99u sock0,9  0t0 11983089 
>> protocol: TCPv6
>> java130244 root  100u sock0,9  0t0 11983090 
>> protocol: TCPv6
>> java130244 root  101u sock0,9  0t0 11983091 
>> protocol: TCPv6
>> java130244 root  102u sock0,9  0t0 11983092 
>> protocol: TCPv6
>> java130244 root  103u sock0,9  0t0 11969375 
>> protocol: TCPv6
>> java130244 root  104u     sock            0,9      0t0 11987339 
>> protocol: TCPv6
>> java130244 root  105u sock0,9  0t0 11987347 
>> protocol: TCPv6
>> java130244 root  106u sock0,9  0t0 11969384 
>> protocol: TCPv6
>> java130244 root  107u sock0,9  0t0 11987348 
>> protocol: TCPv6
>> java130244 root  108r  CHR1,8  0t0 1041 
>> /dev/random
>> java130244 root  109w  REG8,6 19536380  1835053 
>> /apps/apache-tomcat/logs/fms.log
>> java130244 root  110u sock0,9  0t0 11979291 
>> protocol: TCPv6
>> java130244 root  111u sock0,9  0t0 11979292 
>> protocol: TCPv6
>> java130244 root  112w  REG8,6   835408  1835019 
>> /apps/apache-tomcat/logs/obd.log
>> java130244 root  113u sock0,9  0t0 11987349 
>> protocol: TCPv6
>> java130244 root  114u sock0,9  0t0 11987350 
>> protocol: TCPv6
>> java130244 root  115u sock0,9  0t0 11987351 
>> protocol: TCPv6
>> java130244 root  116u sock0,9  0t0 11987352 
>> protocol: TCPv6
>> java130244 root  117u sock0,9  0t0 11987353 
>> protocol: TCPv6
>> java130244 root  118u sock0,9  0t0 11987354 
>> protocol: TCPv6
>> java130244 root  119u sock0,9  0t0 11987355 
>> protocol: TCPv6
>> java130244 root  120u sock0,9  0t0 11987356 
>> protocol: TCPv6
>> java130244 root  121u sock0,9  0t0 11987357 
>> protocol: TCPv6
>> java130244 root  122u sock0,9  0t0 11987358 
>> protocol: TCPv6
>> java130244 root  123u sock0,9  0t0 11987359 
>> protocol: TCPv6
>> java130244 root  124u sock0,9  0t0 11987360 
>> protocol: TCPv6
>> ...
>> Keep seeing the same line
>
> How many of those lines are there? (lsof -p ... | wc -l)
>
> Are you sure there are more than before? Maybe your ulimits where higher
> on your older system and are now
> bothering you?
>
> What is the setting of ulimit for Tomcats process? (cat /proc//limits)
>
> Felix
>
>
>> -Original Message-
>> From: André Warnier (tomcat/perl)  
>> Sent: Saturday, May 22, 2021 10:22 AM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Mmm. Nothing very special in that netstat output.
>

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[Felix Schumacher]

6
> java130244 root  102u sock0,9  0t0 11983092 
> protocol: TCPv6
> java130244 root  103u sock0,9  0t0 11969375 
> protocol: TCPv6
> java130244 root  104u sock0,9  0t0 11987339 
> protocol: TCPv6
> java130244 root  105u sock0,9  0t0 11987347 
> protocol: TCPv6
> java130244 root  106u sock0,9  0t0 11969384 
> protocol: TCPv6
> java130244 root  107u sock0,9  0t0 11987348 
> protocol: TCPv6
> java130244 root  108r  CHR1,8  0t0 1041 
> /dev/random
> java130244 root  109w  REG8,6 19536380  1835053 
> /apps/apache-tomcat/logs/fms.log
> java130244 root  110u sock0,9  0t0 11979291 
> protocol: TCPv6
> java130244 root  111u sock0,9  0t0 11979292 
> protocol: TCPv6
> java130244 root  112w  REG8,6   835408  1835019 
> /apps/apache-tomcat/logs/obd.log
> java130244 root  113u sock0,9  0t0 11987349 
> protocol: TCPv6
> java130244 root  114u sock0,9  0t0 11987350 
> protocol: TCPv6
> java130244 root  115u sock0,9  0t0 11987351 
> protocol: TCPv6
> java130244 root  116u sock    0,9  0t0 11987352 
> protocol: TCPv6
> java130244 root  117u sock0,9  0t0 11987353 
> protocol: TCPv6
> java130244 root  118u sock0,9  0t0 11987354 
> protocol: TCPv6
> java130244 root  119u sock0,9  0t0 11987355 
> protocol: TCPv6
> java130244 root  120u sock0,9  0t0 11987356 
> protocol: TCPv6
> java130244 root  121u sock0,9  0t0 11987357 
> protocol: TCPv6
> java130244 root  122u sock0,9  0t0 11987358 
> protocol: TCPv6
> java130244 root  123u sock0,9  0t0 11987359 
> protocol: TCPv6
> java130244 root  124u sock0,9  0t0 11987360 
> protocol: TCPv6
> ...
> Keep seeing the same line


How many of those lines are there? (lsof -p ... | wc -l)

Are you sure there are more than before? Maybe your ulimits where higher
on your older system and are now
bothering you?

What is the setting of ulimit for Tomcats process? (cat /proc//limits)

Felix


>
> -Original Message-
> From: André Warnier (tomcat/perl)  
> Sent: Saturday, May 22, 2021 10:22 AM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Mmm. Nothing very special in that netstat output.
> The sockets seen there look quite normal for tomcat, and there are not a lot.
> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>
> Looks like lsof is counting things which are not IPv6 TCP sockets belonging 
> to the tomcat JVM process.
> Where is the difference between the lsof count (19948) and the netstat count 
> (25) ?
> (genuine question, I'm puzzled too)
>
> Can you give an example of the "lsof -p 130244" output lines ?
> (not all 19948 please, but enough to see some variety)
>
> On 21.05.2021 16:11, Yeggy Javadi wrote:
>> Hi,
>> Here its is:
>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>> 19948
>>
>> # netstat -p -a -6 --tcp | grep 130244
>> tcp6   0  0 [::]:pcsync-https   [::]:*  LISTEN   
>>130244/java
>> tcp6   0  0 [::]:https  [::]:*  LISTEN   
>>130244/java
>> tcp6   0  0 [::]:37537  [::]:*  LISTEN   
>>130244/java
>> tcp6   0  0 localhost.localdoma:mxi [::]:*  LISTEN   
>>130244/java
>> tcp6   0  0 [::]:8009   [::]:*  LISTEN   
>>130244/java
>> tcp6   0  0 [::]:rmiregistry[::]:*  LISTEN   
>>130244/java
>> tcp6   0  0 [::]:http   [::]:*  LISTEN   
>>130244/java
>> tcp6  86  0 Yeggy-F8-FMSVA:39680172.22.22.192:https 
>> CLOSE_WAIT  130244/java
>> tcp6   0  0 Yeggy-F8-FMSVA:54772Yeggy-F8-FMSVA:vrace
>> ESTABLISHED 130244/java
>> tcp6   0  0 localhost.localdo:42664 localhost.loca:postgres 
>> ESTABLISHED 130244/java
>> tcp6   0  0 Yeggy-F8-FMSVA:54782Yeggy-F8-FMSVA:vrace
>> ESTABLISHED 130244/java
>> tcp6   0  0 Yeggy-F8-FMSVA:54766Yeggy-F8-FMSVA:vrace

RE: apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

Pv6
...
Keep seeing the same line

-Original Message-
From: André Warnier (tomcat/perl)  
Sent: Saturday, May 22, 2021 10:22 AM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Mmm. Nothing very special in that netstat output.
The sockets seen there look quite normal for tomcat, and there are not a lot.
What about the IPv4 sockets ? (remove the -6 in your netstat command)

Looks like lsof is counting things which are not IPv6 TCP sockets belonging to 
the tomcat JVM process.
Where is the difference between the lsof count (19948) and the netstat count 
(25) ?
(genuine question, I'm puzzled too)

Can you give an example of the "lsof -p 130244" output lines ?
(not all 19948 please, but enough to see some variety)

On 21.05.2021 16:11, Yeggy Javadi wrote:
> Hi,
> Here its is:
> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
> 19948
> 
> # netstat -p -a -6 --tcp | grep 130244
> tcp6   0  0 [::]:pcsync-https   [::]:*  LISTEN
>   130244/java
> tcp6   0  0 [::]:https  [::]:*  LISTEN
>   130244/java
> tcp6   0  0 [::]:37537  [::]:*  LISTEN
>   130244/java
> tcp6   0  0 localhost.localdoma:mxi [::]:*  LISTEN
>   130244/java
> tcp6   0  0 [::]:8009   [::]:*  LISTEN
>   130244/java
> tcp6   0  0 [::]:rmiregistry[::]:*  LISTEN
>   130244/java
> tcp6   0  0 [::]:http   [::]:*  LISTEN
>   130244/java
> tcp6  86  0 Yeggy-F8-FMSVA:39680172.22.22.192:https 
> CLOSE_WAIT  130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54772Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 localhost.localdo:42664 localhost.loca:postgres 
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54782Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54766Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 localhost.localdo:42662 localhost.loca:postgres 
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54778Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 localhost.localdo:46966 localhost.localdo:11753 
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54788Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54770Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54790Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54776Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54786Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54780Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 localhost.localdo:45736 localhost.loca:postgres 
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54768Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54784Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 localhost.localdo:42660 localhost.loca:postgres 
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:54774Yeggy-F8-FMSVA:vrace
> ESTABLISHED 130244/java
> tcp6   0  0 localhost.localdo:45734 localhost.loca:postgres 
> ESTABLISHED 130244/java
> tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 
> ESTABLISHED 130244/java
> tcp6   0  0 localhost.localdo:41016 localhost.localdo:vrace 
> ESTABLISHED 130244/java
> Thanks
> -Original Message-
> From: Noelette Stout 
> Sent: Friday, May 21, 2021 8:28 AM
> To: Tomcat Users List 
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> 
> ss has all the same options as netstat
> 
> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) 
> 
> wrote:
> 
>> Soyrry to top-post, but it's getting cluttered down there..
>>
>> The next thing that you may want to do :
>>
>>> netstat -p -a -6 --tcp
>>
>> That is an alternative list of sockets, which also shows the "tcp state"
>> of the sockets.
>> To get only the ones of the tomcat JVM PID, filter with grep based of 
>> the last column.
>> The type of thing you are looking for is the column which should show 
>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT"

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[tomcat/perl]

Mmm. Nothing very special in that netstat output.
The sockets seen there look quite normal for tomcat, and there are not a lot.
What about the IPv4 sockets ? (remove the -6 in your netstat command)

Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat 
JVM process.

Where is the difference between the lsof count (19948) and the netstat count 
(25) ?
(genuine question, I'm puzzled too)

Can you give an example of the "lsof -p 130244" output lines ?
(not all 19948 please, but enough to see some variety)

On 21.05.2021 16:11, Yeggy Javadi wrote:

Hi,
Here its is:
# lsof -p 130244 | grep "protocol: TCPv6"| wc -l
19948

# netstat -p -a -6 --tcp | grep 130244
tcp6   0  0 [::]:pcsync-https   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:https  [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:37537  [::]:*  LISTEN  
130244/java
tcp6   0  0 localhost.localdoma:mxi [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:8009   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:rmiregistry[::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:http   [::]:*  LISTEN  
130244/java
tcp6  86  0 Yeggy-F8-FMSVA:39680172.22.22.192:https CLOSE_WAIT  
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54772Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54782Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54766Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54778Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54788Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54770Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54790Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54776Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54786Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54780Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54768Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54784Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54774Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 
130244/java
Thanks
-Original Message-
From: Noelette Stout 
Sent: Friday, May 21, 2021 8:28 AM
To: Tomcat Users List 
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

ss has all the same options as netstat

On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) 
wrote:


Soyrry to top-post, but it's getting cluttered down there..

The next thing that you may want to do :

   > netstat -p -a -6 --tcp

That is an alternative list of sockets, which also shows the "tcp state"
of the sockets.
To get only the ones of the tomcat JVM PID, filter with grep based of
the last column.
The type of thing you are looking for is the column which should show
"LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..

The options above :
-p : show PID and program
-a : show all sockets states
-6 : only inet v6
--tcp : only TCP sockets

"netstat" may not be on your system by default, and you may need to
install it.
An alternative is "ss", but I don't know the options.


On 21.05.2021 02:14, Yeggy Javadi wrote:

Hi,
Yes; that is what I get and as you can see among 8028 open files,
7474

are for TCPv6 sockets:

java130244 root 7805u sock0,9  0t0 12294251

protocol: TCPv6


# ps -ef | grep tomcat
root  130244   1  1 11:01 ?00:06:20

/usr/local/jre/bin/java
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
properties
-Djava.util.logging.manager=org.apac

RE: apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

Hi,
Here its is:
# lsof -p 130244 | grep "protocol: TCPv6"| wc -l
19948

# netstat -p -a -6 --tcp | grep 130244
tcp6   0  0 [::]:pcsync-https   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:https  [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:37537  [::]:*  LISTEN  
130244/java
tcp6   0  0 localhost.localdoma:mxi [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:8009   [::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:rmiregistry[::]:*  LISTEN  
130244/java
tcp6   0  0 [::]:http   [::]:*  LISTEN  
130244/java
tcp6  86  0 Yeggy-F8-FMSVA:39680172.22.22.192:https CLOSE_WAIT  
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54772Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54782Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54766Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54778Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54788Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54770Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54790Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54776Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54786Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54780Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54768Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54784Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:54774Yeggy-F8-FMSVA:vraceESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 
130244/java
tcp6   0  0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 
130244/java
tcp6   0  0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 
130244/java
Thanks
-Original Message-
From: Noelette Stout  
Sent: Friday, May 21, 2021 8:28 AM
To: Tomcat Users List 
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

ss has all the same options as netstat

On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) 
wrote:

> Soyrry to top-post, but it's getting cluttered down there..
>
> The next thing that you may want to do :
>
>   > netstat -p -a -6 --tcp
>
> That is an alternative list of sockets, which also shows the "tcp state"
> of the sockets.
> To get only the ones of the tomcat JVM PID, filter with grep based of 
> the last column.
> The type of thing you are looking for is the column which should show 
> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>
> The options above :
> -p : show PID and program
> -a : show all sockets states
> -6 : only inet v6
> --tcp : only TCP sockets
>
> "netstat" may not be on your system by default, and you may need to 
> install it.
> An alternative is "ss", but I don't know the options.
>
>
> On 21.05.2021 02:14, Yeggy Javadi wrote:
> > Hi,
> > Yes; that is what I get and as you can see among 8028 open files, 
> > 7474
> are for TCPv6 sockets:
> > java130244 root 7805u sock0,9  0t0 12294251
> protocol: TCPv6
> >
> > # ps -ef | grep tomcat
> > root  130244   1  1 11:01 ?00:06:20
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
> properties 
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi
&

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[Noelette Stout]

ss has all the same options as netstat

On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) 
wrote:

> Soyrry to top-post, but it's getting cluttered down there..
>
> The next thing that you may want to do :
>
>   > netstat -p -a -6 --tcp
>
> That is an alternative list of sockets, which also shows the "tcp state"
> of the sockets.
> To get only the ones of the tomcat JVM PID, filter with grep based of the
> last column.
> The type of thing you are looking for is the column which should show
> "LISTEN", or
> "ESTABLISHED" or "CLOSE_WAIT" etc..
>
> The options above :
> -p : show PID and program
> -a : show all sockets states
> -6 : only inet v6
> --tcp : only TCP sockets
>
> "netstat" may not be on your system by default, and you may need to
> install it.
> An alternative is "ss", but I don't know the options.
>
>
> On 21.05.2021 02:14, Yeggy Javadi wrote:
> > Hi,
> > Yes; that is what I get and as you can see among 8028 open files, 7474
> are for TCPv6 sockets:
> > java130244 root 7805u sock0,9  0t0 12294251
> protocol: TCPv6
> >
> > # ps -ef | grep tomcat
> > root  130244   1  1 11:01 ?00:06:20
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
> -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> > root  132566  132492  0 20:10 pts/100:00:00 grep --color=auto
> tomcat
> >
> > #lsof -p 130244 | wc -l
> > 8028
> >
> > #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
> > 7474
> >
> > Thanks
> >
> > -Original Message-
> > From: André Warnier (tomcat/perl) 
> > Sent: Thursday, May 20, 2021 4:19 PM
> > To: users@tomcat.apache.org
> > Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> >
> > Hi.
> > According to the list below, you have 2 java (JVM) processes running on
> your system.
> > One (PID = 130244) is the JVM which runs tomcat.  This is visible when
> you look at the whole command-line.
> >
> > The other (PID = 130516) runs ElasticSearch, which I believe is not
> relevant here.
> >
> > So you should run lsof with the "-p 130244" option, to show only the
> files opened by the tomcat JVM.
> > To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
> "apache-tomcat"
> > (or anything else unique in that line)
> >
> >
> > On 20.05.2021 21:00, Yeggy Javadi wrote:
> >> Hi Chris,
> >> Please indicate how to show lsof or netstat to *just the JVM* process.
> >>
> >> Below is the list of running processes on my server:
> >> UID  PIDPPID  C STIME TTY  TIME CMD
> >> root   1   0  0 May07 ?00:00:14
> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
> >> root   2   0  0 May07 ?00:00:00 [kthreadd]
> >> root   3   2  0 May07 ?00:00:00 [rcu_gp]
> >> root   4   2  0 May07 ?00:00:00 [rcu_par_gp]
> >> root   6   2  0 May07 ?00:00:00
> [kworker/0:0H-kblockd]
> >> root   8   2  0 May07 ?00:00:00 [mm_percpu_wq]
> >> root   9   2  0 May07 ?00:00:00 [ksoftirqd/0]
> >> root  10   2  0 May07 ?00:02:18 [rcu_sched]
> >> root  11   2  0 May07 ?00:00:00 [migration/0]
> >> root  12   2  0 May07 ?00:00:00 [watchdog/0]
> >> root  13   2  0 May07 ?00:00:00 [cpuhp/0]
> >> root  14   2  0 May07 ?00:00:00 [cpuhp/1]
> >> root  15   2  0 May07 ?00:00:00 [watchdog/1]
> >> root  16   2  0 May07 ?00:00:00 [migration/1]
> >> root  17   2  0 May07 ?00:00:00 [ksoftirqd/1]
> >> root  19   2  0 May07 ?00:00:00
> [kworker/1:0H-kblockd]
> >> root  20   2  0 May07 ?00:

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[tomcat/perl]

Soyrry to top-post, but it's getting cluttered down there..

The next thing that you may want to do :

 > netstat -p -a -6 --tcp

That is an alternative list of sockets, which also shows the "tcp state" of the 
sockets.
To get only the ones of the tomcat JVM PID, filter with grep based of the last 
column.
The type of thing you are looking for is the column which should show "LISTEN", or 
"ESTABLISHED" or "CLOSE_WAIT" etc..


The options above :
-p : show PID and program
-a : show all sockets states
-6 : only inet v6
--tcp : only TCP sockets

"netstat" may not be on your system by default, and you may need to install it.
An alternative is "ss", but I don't know the options.


On 21.05.2021 02:14, Yeggy Javadi wrote:

Hi,
Yes; that is what I get and as you can see among 8028 open files, 7474 are for 
TCPv6 sockets:
java130244 root 7805u sock0,9  0t0 12294251 
protocol: TCPv6

# ps -ef | grep tomcat
root  130244   1  1 11:01 ?00:06:20 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root  132566  132492  0 20:10 pts/100:00:00 grep --color=auto tomcat

#lsof -p 130244 | wc -l
8028

#lsof -p 130244 | grep "protocol: TCPv6"| wc -l
7474

Thanks

-Original Message-
From: André Warnier (tomcat/perl) 
Sent: Thursday, May 20, 2021 4:19 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.
According to the list below, you have 2 java (JVM) processes running on your 
system.
One (PID = 130244) is the JVM which runs tomcat.  This is visible when you look 
at the whole command-line.

The other (PID = 130516) runs ElasticSearch, which I believe is not relevant 
here.

So you should run lsof with the "-p 130244" option, to show only the files 
opened by the tomcat JVM.
To show the current Tomcat JVM PID, do e.g. : ps -ef | grep "apache-tomcat"
(or anything else unique in that line)


On 20.05.2021 21:00, Yeggy Javadi wrote:

Hi Chris,
Please indicate how to show lsof or netstat to *just the JVM* process.

Below is the list of running processes on my server:
UID  PIDPPID  C STIME TTY  TIME CMD
root   1   0  0 May07 ?00:00:14 /usr/lib/systemd/systemd 
--switched-root --system --deserialize 17
root   2   0  0 May07 ?00:00:00 [kthreadd]
root   3   2  0 May07 ?00:00:00 [rcu_gp]
root   4   2  0 May07 ?00:00:00 [rcu_par_gp]
root   6   2  0 May07 ?00:00:00 [kworker/0:0H-kblockd]
root   8   2  0 May07 ?00:00:00 [mm_percpu_wq]
root   9   2  0 May07 ?00:00:00 [ksoftirqd/0]
root  10   2  0 May07 ?00:02:18 [rcu_sched]
root  11   2  0 May07 ?00:00:00 [migration/0]
root  12   2  0 May07 ?00:00:00 [watchdog/0]
root  13   2  0 May07 ?00:00:00 [cpuhp/0]
root  14   2  0 May07 ?00:00:00 [cpuhp/1]
root  15   2  0 May07 ?00:00:00 [watchdog/1]
root  16   2  0 May07 ?00:00:00 [migration/1]
root  17   2  0 May07 ?00:00:00 [ksoftirqd/1]
root  19   2  0 May07 ?00:00:00 [kworker/1:0H-kblockd]
root  20   2  0 May07 ?00:00:00 [cpuhp/2]
root  21   2  0 May07 ?00:00:00 [watchdog/2]
root  22   2  0 May07 ?00:00:00 [migration/2]
root  23   2  0 May07 ?00:00:00 [ksoftirqd/2]
root  25   2  0 May07 ?00:00:00 [kworker/2:0H-kblockd]
root  26   2  0 May07 ?00:00:00 [cpuhp/3]
root  27   2  0 May07 ?00:00:00 [watchdog/3]
root  28   2  0 May07 ?00:00:00 [migration/3]
root  29   2  0 May07 ?00:00:00 [ksoftirqd/3]
root  31   2  0 May07 ?00:00:00 [kworker/3:0H-kblockd]
root  32   2  0 May07 ?00:00:00 [cpuhp/4]
root  33   2  0 May07 ?00:00:00 [watchdog/4]
root  34   2  0 May07 ?00:00:00 [migration/4]
root  35   2  0 May07 ?00:00:00 [ksoftirqd/4]
root  37   2  0 May07 ?00:00:00 [kworker/4:0H-kblockd]
ro

RE: apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

Hi,
Yes; that is what I get and as you can see among 8028 open files, 7474 are for 
TCPv6 sockets:
java130244 root 7805u sock0,9  0t0 12294251 
protocol: TCPv6

# ps -ef | grep tomcat
root  130244   1  1 11:01 ?00:06:20 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root  132566  132492  0 20:10 pts/100:00:00 grep --color=auto tomcat

#lsof -p 130244 | wc -l
8028

#lsof -p 130244 | grep "protocol: TCPv6"| wc -l
7474

Thanks

-Original Message-
From: André Warnier (tomcat/perl)  
Sent: Thursday, May 20, 2021 4:19 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Hi.
According to the list below, you have 2 java (JVM) processes running on your 
system.
One (PID = 130244) is the JVM which runs tomcat.  This is visible when you look 
at the whole command-line.

The other (PID = 130516) runs ElasticSearch, which I believe is not relevant 
here.

So you should run lsof with the "-p 130244" option, to show only the files 
opened by the tomcat JVM.
To show the current Tomcat JVM PID, do e.g. : ps -ef | grep "apache-tomcat"
(or anything else unique in that line)


On 20.05.2021 21:00, Yeggy Javadi wrote:
> Hi Chris,
> Please indicate how to show lsof or netstat to *just the JVM* process.
> 
> Below is the list of running processes on my server:
> UID  PIDPPID  C STIME TTY  TIME CMD
> root   1   0  0 May07 ?00:00:14 /usr/lib/systemd/systemd 
> --switched-root --system --deserialize 17
> root   2   0  0 May07 ?00:00:00 [kthreadd]
> root   3   2  0 May07 ?00:00:00 [rcu_gp]
> root   4   2  0 May07 ?00:00:00 [rcu_par_gp]
> root   6   2  0 May07 ?00:00:00 [kworker/0:0H-kblockd]
> root   8   2  0 May07 ?00:00:00 [mm_percpu_wq]
> root   9   2  0 May07 ?00:00:00 [ksoftirqd/0]
> root  10   2  0 May07 ?00:02:18 [rcu_sched]
> root  11   2  0 May07 ?00:00:00 [migration/0]
> root  12   2  0 May07 ?00:00:00 [watchdog/0]
> root  13   2  0 May07 ?00:00:00 [cpuhp/0]
> root  14   2  0 May07 ?00:00:00 [cpuhp/1]
> root  15   2  0 May07 ?00:00:00 [watchdog/1]
> root  16   2  0 May07 ?00:00:00 [migration/1]
> root  17   2  0 May07 ?00:00:00 [ksoftirqd/1]
> root  19   2  0 May07 ?00:00:00 [kworker/1:0H-kblockd]
> root  20   2  0 May07 ?00:00:00 [cpuhp/2]
> root  21   2  0 May07 ?00:00:00 [watchdog/2]
> root  22   2  0 May07 ?00:00:00 [migration/2]
> root  23   2  0 May07 ?00:00:00 [ksoftirqd/2]
> root  25   2  0 May07 ?00:00:00 [kworker/2:0H-kblockd]
> root  26   2  0 May07 ?00:00:00 [cpuhp/3]
> root  27   2  0 May07 ?00:00:00 [watchdog/3]
> root  28   2  0 May07 ?00:00:00 [migration/3]
> root  29   2  0 May07 ?00:00:00 [ksoftirqd/3]
> root  31   2  0 May07 ?00:00:00 [kworker/3:0H-kblockd]
> root  32   2  0 May07 ?00:00:00 [cpuhp/4]
> root  33   2  0 May07 ?00:00:00 [watchdog/4]
> root  34   2  0 May07 ?00:00:00 [migration/4]
> root  35   2  0 May07 ?00:00:00 [ksoftirqd/4]
> root  37   2  0 May07 ?00:00:00 [kworker/4:0H-kblockd]
> root  38   2  0 May07 ?00:00:00 [cpuhp/5]
> root  39   2  0 May07 ?00:00:00 [watchdog/5]
> root  40   2  0 May07 ?00:00:00 [migration/5]
> root  41   2  0 May07 ?00:00:00 [ksoftirqd/5]
> root  43   2  0 May07 ?00:00:00 [kworker/5:0H-kblockd]
> root  44   2  0 May07 ?00:00:00 [cpuhp/6]
> root  45   2  0 May07 ?00:00:00 [watchdog/6]
> root  46   2  0 May07 ?00:00:00 [migration/6]
> root  47   2  0 May07 ?  

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[tomcat/perl]

00:00 ps -ef

Thanks
-Original Message-
From: Christopher Schultz 
Sent: Thursday, May 20, 2021 12:49 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Yeggy,

On 5/19/21 19:37, Yeggy Javadi wrote:

Hi Chris,
Thanks for your email. Below are replies to your questions:

1. Did you upgrade anything recently (like Java VM)?
[YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.

2. What is error you are seeing? A full stack trace would be helpful.
[YJ] The application error can occur in any place with the (Too many open 
files) error when the limit of open files for tomcat has reached (262144 in my 
environment), for example, I can get a connectivity error when pulling info 
from a server as below :

   [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService -
Error : The XML configuration file failed to be retrieved for server
10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for
server=10.1.6.25 error=java.io.FileNotFoundException:
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
2.10.RELEASE.jar (Too many open files) restatus=-1 output=

3. What is your  configuration?
[YJ] It is as below in server.xml:

  
...
  

  
  


You have lots of settings on the HTTP connector but not the HTTPS one.
Is that intentional? Which one are you using? Or both?

Are you using AJP? If not, disable the connetor entirely. Definitely never 
enable it on 0.0.0.0 unless you really know what you are doing.


4. Are you counting all the open files for a single process or all of the 
sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
[YJ] I am just getting the process ID of tomcat and count open files
for that process


Ok.


5. Running lsof, netstat, etc. can you see if some large number of those 
sockets are bound to any specific port (listen or connect)?
[YJ] Here is the netstat output:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 Yeggy-F8-FMSVA:ssh  10.12.3.33:55236ESTABLISHED
tcp0 64 Yeggy-F8-FMSVA:ssh  10.197.255.10:60378 ESTABLISHED
tcp0  0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED


That's not helpful. I want to see what the JVM process is holding, not what the 
whole OS is handling.

My guess is that lsof is going to be a better bet for inspecting the process. But if you get 
"too many open files" being thrown inside the JVM and you don't see a 1/4 million lines 
of "lsof" output on your process, then something else is going on. But remember to limit 
your lsof to *just the JVM* process.

-chris


-Original Message-
From: Christopher Schultz 
Sent: Monday, May 17, 2021 5:18 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Yeggy,

On 5/17/21 15:17, Yeggy Javadi wrote:

Hi,
I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
java51798 root 6601u sock0,9  0t0 3959342 protocol: 
TCPv6

I was running the same application on Linux 7.6 with the same
apache-tomcat version without seeing this issue. I saw that RedHat
has removed apache-tomcat fom their Linux 8 distribution. Please help
with throubleshooting and indicating if there is any changes in Linux
8 that can explain this

Configuration:
apache-tomcat-8.5.59
OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
using Postgres and elastic search databases


# ps -ef | grep tomcat
root   51798   1  1 11:16 ?00:06:01 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root   54122   54037  0 19:43 pts/100:00:00 grep --color=auto tomcat


# cat /proc/51798/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack size8388608  unlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max resident set  unlimited

RE: apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

-256-cert-...@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-...@openssh.com,ssh-rsa,ssh-rsa-cert-...@openssh.com
 
-oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
root  122905   2  0 May19 ?00:00:27 [kworker/7:0-events]
root  126037   2  0 05:30 ?00:00:00 [kworker/1:2-events]
root  128108   2  0 08:40 ?00:00:00 
[kworker/4:1-events_power_efficient]
root  128360   2  0 09:10 ?00:00:00 [kworker/4:2-cgroup_destroy]
root  128430   2  0 09:30 ?00:00:00 [kworker/6:0-mm_percpu_wq]
root  128738   2  0 11:00 ?00:00:00 [kworker/3:3-mm_percpu_wq]
root  128761   2  0 11:00 ?00:00:00 [kworker/2:1-mm_percpu_wq]
root  130244   1  1 11:01 ?00:03:50 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
elastic+  130516   1  1 11:01 ?00:03:26 /bin/java -Xms8g -Xmx8g 
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 
-Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp 
:/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
 -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid 
-Des.default.path.home=/usr/share/elasticsearch 
-Des.default.path.logs=/var/log/elasticsearch 
-Des.default.path.data=/var/lib/elasticsearch 
-Des.default.config=/etc/elasticsearch/elasticsearch.yml 
-Des.default.path.conf=/etc/elasticsearch 
org.elasticsearch.bootstrap.Elasticsearch
postgres  130548   1  0 11:01 ?00:00:00 /usr/pgsql-9.6/bin/postgres 
-D /DB/postgres/data
postgres  130561  130548  0 11:01 ?00:00:00 postgres: logger process
postgres  130563  130548  0 11:01 ?00:00:00 postgres: checkpointer 
process
postgres  130564  130548  0 11:01 ?00:00:00 postgres: writer process
postgres  130565  130548  0 11:01 ?00:00:00 postgres: wal writer process
postgres  130566  130548  0 11:01 ?00:00:00 postgres: autovacuum 
launcher process
postgres  130567  130548  0 11:01 ?00:00:00 postgres: stats collector 
process
root  130570   1  0 11:01 ?00:00:01 /usr/local/freestor/bin/fmsd
postgres  130675  130548  0 11:01 ?00:00:03 postgres: postgres freestor 
127.0.0.1(42660) idle
postgres  130676  130548  0 11:01 ?00:00:04 postgres: postgres freestor 
127.0.0.1(42662) idle
postgres  130677  130548  0 11:01 ?00:00:03 postgres: postgres freestor 
127.0.0.1(42664) idle
root  131206   2  0 13:00 ?00:00:00 [kworker/7:2]
root  131269   2  0 13:20 ?00:00:00 [kworker/6:2]
root  131347   2  0 13:49 ?00:00:00 [kworker/5:1-mm_percpu_wq]
root  131364   2  0 13:56 ?00:00:00 
[kworker/u16:2-events_unbound]
root  131401   2  0 14:05 ?00:00:00 
[kworker/u16:1-events_unbound]
root  131438   2  0 14:20 ?00:00:00 [kworker/2:2]
root  131465   2  0 14:30 ?00:00:00 [kworker/0:2-events]
root  131490   2  0 14:40 ?00:00:00 [kworker/3:0]
root  131491   2  0 14:40 ?00:00:00 [kworker/0:1-events]
root  131515   2  0 14:49 ?00:00:00 
[kworker/1:0-events_freezable]
root  131520   2  0 14:50 ?00:00:00 [kworker/5:0]
root  131525   14261  0 14:52 ?00:00:00 sshd: root [priv]
root  131529  131525  0 14:52 ?00:00:00 sshd: root@pts/1
root  131530  131529  0 14:52 pts/100:00:00 -bash
root  131600  131530  0 14:57 pts/100:00:00 ps -ef

Thanks
-Original Message-
From: Christopher Schultz  
Sent: Thursday, May 20, 2021 12:49 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Yeggy,

On 5/19/21 19:37, Yeggy Javadi wrote:
> Hi Chris,
> Thanks for your email. Below are replies to your questions:
> 
> 1. Did you upgrade anything recently (like Java VM)?
> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
> 
> 2. What is error you are seeing? A full stack trace would be helpful.
> [YJ] The app

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[Christopher Schultz]

Yeggy,

On 5/19/21 19:37, Yeggy Javadi wrote:

Hi Chris,
Thanks for your email. Below are replies to your questions:

1. Did you upgrade anything recently (like Java VM)?
[YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.

2. What is error you are seeing? A full stack trace would be helpful.
[YJ] The application error can occur in any place with the (Too many open 
files) error when the limit of open files for tomcat has reached (262144 in my 
environment), for example, I can get a connectivity error when pulling info 
from a server as below :

  [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - Error : The 
XML configuration file failed to be retrieved for server 10.1.6.25. Check 
server connectivity.Error retrieving IPStorConfg for server=10.1.6.25 
error=java.io.FileNotFoundException: 
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.2.10.RELEASE.jar
 (Too many open files) restatus=-1 output=

3. What is your  configuration?
[YJ] It is as below in server.xml:

 
...
 

 
 


You have lots of settings on the HTTP connector but not the HTTPS one. 
Is that intentional? Which one are you using? Or both?


Are you using AJP? If not, disable the connetor entirely. Definitely 
never enable it on 0.0.0.0 unless you really know what you are doing.



4. Are you counting all the open files for a single process or all of the 
sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
[YJ] I am just getting the process ID of tomcat and count open files for that 
process


Ok.


5. Running lsof, netstat, etc. can you see if some large number of those 
sockets are bound to any specific port (listen or connect)?
[YJ] Here is the netstat output:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 Yeggy-F8-FMSVA:ssh  10.12.3.33:55236ESTABLISHED
tcp0 64 Yeggy-F8-FMSVA:ssh  10.197.255.10:60378 ESTABLISHED
tcp0  0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED


That's not helpful. I want to see what the JVM process is holding, not 
what the whole OS is handling.


My guess is that lsof is going to be a better bet for inspecting the 
process. But if you get "too many open files" being thrown inside the 
JVM and you don't see a 1/4 million lines of "lsof" output on your 
process, then something else is going on. But remember to limit your 
lsof to *just the JVM* process.


-chris


-Original Message-
From: Christopher Schultz 
Sent: Monday, May 17, 2021 5:18 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Yeggy,

On 5/17/21 15:17, Yeggy Javadi wrote:

Hi,
I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
java51798 root 6601u sock0,9  0t0 3959342 protocol: 
TCPv6

I was running the same application on Linux 7.6 with the same
apache-tomcat version without seeing this issue. I saw that RedHat has
removed apache-tomcat fom their Linux 8 distribution. Please help with
throubleshooting and indicating if there is any changes in Linux 8
that can explain this

Configuration:
apache-tomcat-8.5.59
OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
using Postgres and elastic search databases


# ps -ef | grep tomcat
root   51798   1  1 11:16 ?00:06:01 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root   54122   54037  0 19:43 pts/100:00:00 grep --color=auto tomcat


# cat /proc/51798/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack size8388608  unlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max resident set  unlimitedunlimitedbytes
Max processes 256518   256518   processes
Max open files262144   262144   fil

RE: apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

  2  [ ] DGRAM28344
unix  2  [ ] DGRAM23311
unix  3  [ ] STREAM CONNECTED 11932
unix  2  [ ] STREAM CONNECTED 11143887
unix  3  [ ] STREAM CONNECTED 31205
unix  2  [ ] DGRAM32811
unix  2  [ ] DGRAM513
Active Bluetooth connections (w/o servers)
Proto  Destination   SourceState PSM DCID   SCID  
IMTUOMTU Security
Proto  Destination   SourceState Channel

Thanks

-Original Message-
From: Christopher Schultz  
Sent: Monday, May 17, 2021 5:18 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8

Yeggy,

On 5/17/21 15:17, Yeggy Javadi wrote:
> Hi,
> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
> java51798 root 6601u sock0,9  0t0 3959342 
> protocol: TCPv6
> 
> I was running the same application on Linux 7.6 with the same 
> apache-tomcat version without seeing this issue. I saw that RedHat has 
> removed apache-tomcat fom their Linux 8 distribution. Please help with 
> throubleshooting and indicating if there is any changes in Linux 8 
> that can explain this
> 
> Configuration:
> apache-tomcat-8.5.59
> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application 
> using Postgres and elastic search databases
> 
> 
> # ps -ef | grep tomcat
> root   51798   1  1 11:16 ?00:06:01 /usr/local/jre/bin/java 
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
>  -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
> -Djdk.tls.ephemeralDHKeySize=2048 
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
> -Dignore.endorsed.dirs= -classpath 
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
>  -Dcatalina.base=/usr/local/apache-tomcat 
> -Dcatalina.home=/usr/local/apache-tomcat 
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
> org.apache.catalina.startup.Bootstrap start
> root   54122   54037  0 19:43 pts/100:00:00 grep --color=auto tomcat
> 
> 
> # cat /proc/51798/limits
> Limit Soft Limit   Hard Limit   Units
> Max cpu time  unlimitedunlimitedseconds
> Max file size unlimitedunlimitedbytes
> Max data size unlimitedunlimitedbytes
> Max stack size8388608  unlimitedbytes
> Max core file sizeunlimitedunlimitedbytes
> Max resident set  unlimitedunlimitedbytes
> Max processes 256518   256518   processes
> Max open files262144   262144   files
> Max locked memory 6553665536bytes
> Max address space unlimitedunlimitedbytes
> Max file locksunlimitedunlimitedlocks
> Max pending signals   256518   256518   signals
> Max msgqueue size 819200   819200   bytes
> Max nice priority 00
> Max realtime priority 00
> Max realtime timeout  unlimitedunlimitedus
> 
> 
> Too may open files:
> # lsof -p 51798 | wc -l
> 6679
> 
> 
> Most of open files are TCP sockets:
> java51798 root 6601u sock0,9  0t0 3959342 
> protocol: TCPv6
> 
> 
> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
> 6413

Some potentially dumb questions:

1. Did you upgrade anything recently (like Java VM)?

2. What is error you are seeing? A full stack trace would be helpful.

3. What is your  configuration?

4. Are you counting all the open files for a single process or all of the 
sub-processes which represent the threads of the main process? 
Different kernel versions count things differently.

5. Running lsof, netstat, etc. can you see if some large number of those 
sockets are bound to any specific port (listen or connect)?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: apache-tomcat-8.5.59 too many open files on Linux 8

[Christopher Schultz]

Yeggy,

On 5/17/21 15:17, Yeggy Javadi wrote:

Hi,
I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
java51798 root 6601u sock0,9  0t0 3959342 protocol: 
TCPv6

I was running the same application on Linux 7.6 with the same apache-tomcat 
version without seeing this issue. I saw that RedHat has remove apache-tomcat 
fom their Linux 8 distribution. Please help with throubleshooting and 
indicating if there is any changes in Linux 8 that can explain this

Configuration:
apache-tomcat-8.5.59
OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64
Web-based application using Postgres and elastic search databases


# ps -ef | grep tomcat
root   51798   1  1 11:16 ?00:06:01 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root   54122   54037  0 19:43 pts/100:00:00 grep --color=auto tomcat


# cat /proc/51798/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack size8388608  unlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max resident set  unlimitedunlimitedbytes
Max processes 256518   256518   processes
Max open files262144   262144   files
Max locked memory 6553665536bytes
Max address space unlimitedunlimitedbytes
Max file locksunlimitedunlimitedlocks
Max pending signals   256518   256518   signals
Max msgqueue size 819200   819200   bytes
Max nice priority 00
Max realtime priority 00
Max realtime timeout  unlimitedunlimitedus


Too may open files:
# lsof -p 51798 | wc -l
6679


Most of open files are TCP sockets:
java51798 root 6601u sock0,9  0t0 3959342 protocol: 
TCPv6


# lsof -p 51798 | grep "protocol: TCPv6" | wc -l
6413


Some potentially dumb questions:

1. Did you upgrade anything recently (like Java VM)?

2. What is error you are seeing? A full stack trace would be helpful.

3. What is your  configuration?

4. Are you counting all the open files for a single process or all of 
the sub-processes which represent the threads of the main process? 
Different kernel versions count things differently.


5. Running lsof, netstat, etc. can you see if some large number of those 
sockets are bound to any specific port (listen or connect)?


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

apache-tomcat-8.5.59 too many open files on Linux 8

[Yeggy Javadi]

Hi,
I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
java51798 root 6601u sock0,9  0t0 3959342 protocol: 
TCPv6

I was running the same application on Linux 7.6 with the same apache-tomcat 
version without seeing this issue. I saw that RedHat has remove apache-tomcat 
fom their Linux 8 distribution. Please help with throubleshooting and 
indicating if there is any changes in Linux 8 that can explain this

Configuration:
apache-tomcat-8.5.59
OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64
Web-based application using Postgres and elastic search databases


# ps -ef | grep tomcat
root   51798   1  1 11:16 ?00:06:01 /usr/local/jre/bin/java 
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server 
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 
-Dignore.endorsed.dirs= -classpath 
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
 -Dcatalina.base=/usr/local/apache-tomcat 
-Dcatalina.home=/usr/local/apache-tomcat 
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp 
org.apache.catalina.startup.Bootstrap start
root   54122   54037  0 19:43 pts/100:00:00 grep --color=auto tomcat


# cat /proc/51798/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack size8388608  unlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max resident set  unlimitedunlimitedbytes
Max processes 256518   256518   processes
Max open files262144   262144   files
Max locked memory 6553665536bytes
Max address space unlimitedunlimitedbytes
Max file locksunlimitedunlimitedlocks
Max pending signals   256518   256518   signals
Max msgqueue size 819200   819200   bytes
Max nice priority 00
Max realtime priority 00
Max realtime timeout  unlimitedunlimitedus


Too may open files:
# lsof -p 51798 | wc -l
6679


Most of open files are TCP sockets:
java51798 root 6601u sock0,9  0t0 3959342 protocol: 
TCPv6


# lsof -p 51798 | grep "protocol: TCPv6" | wc -l
6413


Thanks