Soyrry to top-post, but it's getting cluttered down there..
The next thing that you may want to do :
> netstat -p -a -6 --tcp
That is an alternative list of sockets, which also shows the "tcp state"
of the sockets.
To get only the ones of the tomcat JVM PID, filter with grep based of
the last column.
The type of thing you are looking for is the column which should show
"LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
The options above :
-p : show PID and program
-a : show all sockets states
-6 : only inet v6
--tcp : only TCP sockets
"netstat" may not be on your system by default, and you may need to
install it.
An alternative is "ss", but I don't know the options.
On 21.05.2021 02:14, Yeggy Javadi wrote:
Hi,
Yes; that is what I get and as you can see among 8028 open files,
7474
are for TCPv6 sockets:
java 130244 root 7805u sock 0,9 0t0 12294251
protocol: TCPv6
# ps -ef | grep tomcat
root 130244 1 1 11:01 ? 00:06:20
/usr/local/jre/bin/java
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Dignore.endorsed.dirs= -classpath
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
-Dcatalina.home=/usr/local/apache-tomcat
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp
org.apache.catalina.startup.Bootstrap start
root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
tomcat
#lsof -p 130244 | wc -l
8028
#lsof -p 130244 | grep "protocol: TCPv6"| wc -l
7474
Thanks
-----Original Message-----
From: André Warnier (tomcat/perl) <a...@ice-sa.com>
Sent: Thursday, May 20, 2021 4:19 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Hi.
According to the list below, you have 2 java (JVM) processes running
on
your system.
One (PID = 130244) is the JVM which runs tomcat. This is visible
when
you look at the whole command-line.
The other (PID = 130516) runs ElasticSearch, which I believe is not
relevant here.
So you should run lsof with the "-p 130244" option, to show only the
files opened by the tomcat JVM.
To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
"apache-tomcat"
(or anything else unique in that line)
On 20.05.2021 21:00, Yeggy Javadi wrote:
Hi Chris,
Please indicate how to show lsof or netstat to *just the JVM* process.
Below is the list of running processes on my server:
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May07 ? 00:00:14
/usr/lib/systemd/systemd --switched-root --system --deserialize 17
root 2 0 0 May07 ? 00:00:00 [kthreadd]
root 3 2 0 May07 ? 00:00:00 [rcu_gp]
root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
root 6 2 0 May07 ? 00:00:00
[kworker/0:0H-kblockd]
root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
root 10 2 0 May07 ? 00:02:18 [rcu_sched]
root 11 2 0 May07 ? 00:00:00 [migration/0]
root 12 2 0 May07 ? 00:00:00 [watchdog/0]
root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
root 15 2 0 May07 ? 00:00:00 [watchdog/1]
root 16 2 0 May07 ? 00:00:00 [migration/1]
root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
root 19 2 0 May07 ? 00:00:00
[kworker/1:0H-kblockd]
root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
root 21 2 0 May07 ? 00:00:00 [watchdog/2]
root 22 2 0 May07 ? 00:00:00 [migration/2]
root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
root 25 2 0 May07 ? 00:00:00
[kworker/2:0H-kblockd]
root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
root 27 2 0 May07 ? 00:00:00 [watchdog/3]
root 28 2 0 May07 ? 00:00:00 [migration/3]
root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
root 31 2 0 May07 ? 00:00:00
[kworker/3:0H-kblockd]
root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
root 33 2 0 May07 ? 00:00:00 [watchdog/4]
root 34 2 0 May07 ? 00:00:00 [migration/4]
root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
root 37 2 0 May07 ? 00:00:00
[kworker/4:0H-kblockd]
root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
root 39 2 0 May07 ? 00:00:00 [watchdog/5]
root 40 2 0 May07 ? 00:00:00 [migration/5]
root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
root 43 2 0 May07 ? 00:00:00
[kworker/5:0H-kblockd]
root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
root 45 2 0 May07 ? 00:00:00 [watchdog/6]
root 46 2 0 May07 ? 00:00:00 [migration/6]
root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
root 49 2 0 May07 ? 00:00:00
[kworker/6:0H-kblockd]
root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
root 51 2 0 May07 ? 00:00:00 [watchdog/7]
root 52 2 0 May07 ? 00:00:00 [migration/7]
root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
root 55 2 0 May07 ? 00:00:00
[kworker/7:0H-kblockd]
root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
root 58 2 0 May07 ? 00:00:00 [netns]
root 59 2 0 May07 ? 00:00:00 [kauditd]
root 62 2 0 May07 ? 00:00:00 [khungtaskd]
root 63 2 0 May07 ? 00:00:00 [oom_reaper]
root 64 2 0 May07 ? 00:00:00 [writeback]
root 65 2 0 May07 ? 00:00:00 [kcompactd0]
root 66 2 0 May07 ? 00:00:00 [ksmd]
root 67 2 0 May07 ? 00:00:02 [khugepaged]
root 68 2 0 May07 ? 00:00:00 [crypto]
root 69 2 0 May07 ? 00:00:00 [kintegrityd]
root 70 2 0 May07 ? 00:00:00 [kblockd]
root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
root 72 2 0 May07 ? 00:00:00 [md]
root 73 2 0 May07 ? 00:00:00 [edac-poller]
root 74 2 0 May07 ? 00:00:00 [watchdogd]
root 92 2 0 May07 ? 00:00:00 [kswapd0]
root 188 2 0 May07 ? 00:00:00 [kthrotld]
root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
root 224 2 0 May07 ? 00:00:00 [kaluad]
root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
root 227 2 0 May07 ? 00:00:00 [kstrp]
root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
root 509 2 0 May07 ? 00:00:00 [ata_sff]
root 510 2 0 May07 ? 00:00:00 [mpt/0]
root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
root 525 2 0 May07 ? 00:00:00 [ttm_swap]
root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
root 540 2 0 May07 ? 00:00:01
[kworker/7:1H-kblockd]
root 541 2 0 May07 ? 00:00:01
[kworker/5:1H-kblockd]
root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 613 2 0 May07 ? 00:00:01
[kworker/6:1H-kblockd]
root 650 2 0 May07 ? 00:00:01
[kworker/4:1H-kblockd]
root 659 2 0 May07 ? 00:00:02
[kworker/3:1H-kblockd]
root 660 2 0 May07 ? 00:00:02
[kworker/2:1H-kblockd]
root 662 2 0 May07 ? 00:00:01
[kworker/0:1H-kblockd]
root 679 1 0 May07 ? 00:00:03
/usr/lib/systemd/systemd-journald
root 682 2 0 May07 ? 00:00:02
[kworker/1:1H-kblockd]
root 716 1 0 May07 ? 00:00:01
/usr/lib/systemd/systemd-udevd
root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 776 1 0 May07 ? 00:00:01 /sbin/auditd
root 790 2 0 May07 ? 00:00:00 [rpciod]
root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
root 792 2 0 May07 ? 00:00:00 [xprtiod]
dbus 883 1 0 May07 ? 00:00:03
/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
--systemd-activation --syslog-only
root 885 1 0 May07 ? 00:02:42
/usr/sbin/NetworkManager --no-daemon
root 887 1 0 May07 ? 00:00:44
/usr/sbin/irqbalance --foreground
root 888 1 0 May07 ? 00:00:00
/usr/bin/VGAuthService -s
root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
--logger=files
chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
--fill-watermark=0
root 946 890 0 May07 ? 00:00:06
/usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
--logger=files
root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
-d2
root 962 890 0 May07 ? 00:00:12
/usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
-D
root 970 1 0 May07 ? 00:00:02
/usr/lib/systemd/systemd-logind
root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
-n
root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
root 6223 1 0 May07 ? 00:07:27
/usr/local/vtl/bin/sanidmgr
root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
root 6272 2 0 May07 ? 00:00:03 [bksc_done]
root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
root 6460 2 0 May07 ? 00:00:00 [kdownstream]
root 6461 2 0 May07 ? 00:00:07 [msgfilter]
root 6463 2 0 May07 ? 00:00:12 [iocore]
root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
root 6523 2 0 May07 ? 00:01:50 [trudpd]
root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
root 6551 2 0 May07 ? 00:00:00 [misc_thread]
root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
root 6554 2 0 May07 ? 00:00:00 [listen_handle]
root 6557 2 0 May07 ? 00:00:00 [ipstord]
root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
root 6654 1 0 May07 ? 00:00:00
/usr/local/vtl/bin/ipstorcomm -d 0
root 6655 6654 0 May07 ? 00:05:18
/usr/local/vtl/bin/ipstorcomm -d 0
root 6656 6655 0 May07 ? 00:00:36
/usr/local/vtl/bin/ipstorcomm -d 0
root 6693 1 0 May07 ? 00:00:00 rdesirkey
root 6699 1 0 May07 ? 00:00:12 ikmd
root 6721 1 0 May07 ? 00:00:35
/usr/local/vtl/bin/iscliproxy
root 6747 1 0 May07 ? 00:00:40
/usr/local/vtl/bin/ipstorlog
root 6780 1 0 May07 ? 00:04:50
/usr/local/vtl/bin/snmpd
root 7722 1 0 May07 ? 00:00:00
/usr/local/vtl/bin/ipstorsm
root 7723 7722 0 May07 ? 01:08:12
/usr/local/vtl/bin/ipstorsm
root 7901 1 0 May07 ? 00:00:00
/usr/lib/systemd/systemd --user
root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
-- \u --noclear tty1 linux
root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
root 14202 14201 0 May08 pts/0 00:00:00 -bash
root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
-oCiphers=aes256-...@openssh.com,chacha20-poly1...@openssh.com
,aes256-ctr,aes256-cbc,aes128-...@openssh.com,aes128-ctr,aes128-cbc
-oMACs=hmac-sha2-256-...@openssh.com,hmac-sha1-...@openssh.com,
umac-128-...@openssh.com,hmac-sha2-512-...@openssh.com
,hmac-sha2-256,hmac-sha1,umac-...@openssh.com,hmac-sha2-512
-oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
-
group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
-oKexAlgorithms=curve25519-sha256,curve25519-sha...@libssh.org,ecdh-s
h
a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
p
-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
-
sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
a
1,diffie-hellman-group14-sha1
-oHostKeyAlgorithms=ecdsa-sha2-nistp256,
ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384,
ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521,
ecdsa-sha2-nistp521-cert-...@openssh.com,ssh-ed25519,
ssh-ed25519-cert-...@openssh.com,rsa-sha2-256,
rsa-sha2-256-cert-...@openssh.com,rsa-sha2-512,
rsa-sha2-512-cert-...@openssh.com,ssh-rsa,ssh-rsa-cert-...@openssh.co
m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384,
ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521,
ecdsa-sha2-nistp521-cert-...@openssh.com,ssh-ed25519,
ssh-ed25519-cert-...@openssh.com,rsa-sha2-256,
rsa-sha2-256-cert-...@openssh.com,rsa-sha2-512,
rsa-sha2-512-cert-...@openssh.com,ssh-rsa,ssh-rsa-cert-...@openssh.co
m
-oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
- sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
root 128108 2 0 08:40 ? 00:00:00
[kworker/4:1-events_power_efficient]
root 128360 2 0 09:10 ? 00:00:00
[kworker/4:2-cgroup_destroy]
root 128430 2 0 09:30 ? 00:00:00
[kworker/6:0-mm_percpu_wq]
root 128738 2 0 11:00 ? 00:00:00
[kworker/3:3-mm_percpu_wq]
root 128761 2 0 11:00 ? 00:00:00
[kworker/2:1-mm_percpu_wq]
root 130244 1 1 11:01 ? 00:03:50
/usr/local/jre/bin/java
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Dignore.endorsed.dirs= -classpath
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
-Dcatalina.home=/usr/local/apache-tomcat
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp
org.apache.catalina.startup.Bootstrap start
elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
-Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
-Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
-Des.path.home=/usr/share/elasticsearch -cp
:/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
t
icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
-Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.config=/etc/elasticsearch/elasticsearch.yml
-Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch
postgres 130548 1 0 11:01 ? 00:00:00
/usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
process
postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
checkpointer process
postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
process
postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
writer process
postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
autovacuum launcher process
postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
collector process
root 130570 1 0 11:01 ? 00:00:01
/usr/local/freestor/bin/fmsd
postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
freestor 127.0.0.1(42660) idle
postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
freestor 127.0.0.1(42662) idle
postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
freestor 127.0.0.1(42664) idle
root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
root 131347 2 0 13:49 ? 00:00:00
[kworker/5:1-mm_percpu_wq]
root 131364 2 0 13:56 ? 00:00:00
[kworker/u16:2-events_unbound]
root 131401 2 0 14:05 ? 00:00:00
[kworker/u16:1-events_unbound]
root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
root 131515 2 0 14:49 ? 00:00:00
[kworker/1:0-events_freezable]
root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
root 131530 131529 0 14:52 pts/1 00:00:00 -bash
root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
Thanks
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, May 20, 2021 12:49 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Yeggy,
On 5/19/21 19:37, Yeggy Javadi wrote:
Hi Chris,
Thanks for your email. Below are replies to your questions:
1. Did you upgrade anything recently (like Java VM)?
[YJ] To support Linux 8, only Postgres was upgraded from version
9.3
to 9.6.
2. What is error you are seeing? A full stack trace would be helpful.
[YJ] The application error can occur in any place with the (Too
many
open files) error when the limit of open files for tomcat has reached
(262144 in my environment), for example, I can get a connectivity
error when pulling info from a server as below :
[InventoryPullerTask - 10.1.6.25] ERROR
FSSDataCollectorService - Error : The XML configuration file
failed to be retrieved for server 10.1.6.25. Check server
connectivity.Error retrieving IPStorConfg for
server=10.1.6.25 error=java.io.FileNotFoundException:
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
2.10.RELEASE.jar (Too many open files) restatus=-1 output=
3. What is your <Connector> configuration?
[YJ] It is as below in server.xml:
<Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
maxThreads="150"
minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml,application/json"
redirectPort="443" /> ...........
<Connector address="0.0.0.0" port="443"
protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslEnabledProtocols="TLSv1.2"
ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
keystoreFile="conf/freestor"
keystorePass="freestor"/>
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
redirectPort="443" secretRequired="false" />
You have lots of settings on the HTTP connector but not the HTTPS one.
Is that intentional? Which one are you using? Or both?
Are you using AJP? If not, disable the connetor entirely.
Definitely
never enable it on 0.0.0.0 unless you really know what you are doing.
4. Are you counting all the open files for a single process or all
of
the sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
[YJ] I am just getting the process ID of tomcat and count open
files for that process
Ok.
5. Running lsof, netstat, etc. can you see if some large number of
those sockets are bound to any specific port (listen or connect)?
[YJ] Here is the netstat output:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
ESTABLISHED
tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36846
ESTABLISHED
That's not helpful. I want to see what the JVM process is holding,
not
what the whole OS is handling.
My guess is that lsof is going to be a better bet for inspecting
the
process. But if you get "too many open files" being thrown inside the
JVM and you don't see a 1/4 million lines of "lsof" output on your
process, then something else is going on. But remember to limit your
lsof to *just the JVM* process.
-chris
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, May 17, 2021 5:18 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Yeggy,
On 5/17/21 15:17, Yeggy Javadi wrote:
Hi,
I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
java 51798 root 6601u sock 0,9 0t0 3959342
protocol: TCPv6
I was running the same application on Linux 7.6 with the same
apache-tomcat version without seeing this issue. I saw that
RedHat has removed apache-tomcat fom their Linux 8 distribution.
Please help with throubleshooting and indicating if there is any
changes in Linux
8 that can explain this
Configuration:
apache-tomcat-8.5.59
OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
application using Postgres and elastic search databases
# ps -ef | grep tomcat
root 51798 1 1 11:16 ? 00:06:01
/usr/local/jre/bin/java
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Dignore.endorsed.dirs= -classpath
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
-Dcatalina.home=/usr/local/apache-tomcat
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp
org.apache.catalina.startup.Bootstrap start
root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
tomcat
# cat /proc/51798/limits
Limit Soft Limit Hard Limit
Units
Max cpu time unlimited unlimited
seconds
Max file size unlimited unlimited
bytes
Max data size unlimited unlimited
bytes
Max stack size 8388608 unlimited
bytes
Max core file size unlimited unlimited
bytes
Max resident set unlimited unlimited
bytes
Max processes 256518 256518
processes
Max open files 262144 262144
files
Max locked memory 65536 65536
bytes
Max address space unlimited unlimited
bytes
Max file locks unlimited unlimited
locks
Max pending signals 256518 256518
signals
Max msgqueue size 819200 819200
bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Too may open files:
# lsof -p 51798 | wc -l
6679
Most of open files are TCP sockets:
java 51798 root 6601u sock 0,9 0t0 3959342
protocol: TCPv6
# lsof -p 51798 | grep "protocol: TCPv6" | wc -l
6413
Some potentially dumb questions:
1. Did you upgrade anything recently (like Java VM)?
2. What is error you are seeing? A full stack trace would be helpful.
3. What is your <Connector> configuration?
4. Are you counting all the open files for a single process or all
of
the sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
5. Running lsof, netstat, etc. can you see if some large number of
those sockets are bound to any specific port (listen or connect)?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
