Hi,
We have upgraded RHEL 7.2 to RHEL 7.4. After upgrade when we are running
performance testing of our application we are seeing below error in
catalina.out.
java.lang.OutOfMemoryError: unable to create new native thread Exception in
Catalina.out after upgrading to RHEL 7.4
Attached is the thread dump.
We are using Tomcat 7.0.69.
As per REDHAT we have increased tomcat5 user which used to run Tomcat process
in /etc/security/limits.d/20-nproc.conf like below
* soft nproc 4096
root soft nproc unlimited
tomcat5 soft nproc 8192
All the processes are used by Tomcat. If we increase the value to 16K or 32K
all threads are consumed and they are in Waiting(Parking) state.
"http-bio-8443-exec-56" #973 daemon prio=5 os_prio=0 tid=0x00007efc8c029800
nid=0x10e7b waiting on condition [0x00007ef9bc7ee000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c454c008> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8443-exec-55" #972 daemon prio=5 os_prio=0 tid=0x00007efc0401e000
nid=0x10e7a waiting on condition [0x00007ef9bc82f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c454c008> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
They are not getting release. After some time Tomcat getting stopped
automatically. You may need the output of below from our setup.
[root@iqsvtk05<mailto:root@iqsvtk05> ~]# rpm -qif
/opt/coreservices/tomcat-7.0.69/logs/
Name : tomcat
Version : 7.0.69
Release : AV10
Architecture: x86_64
Install Date: Fri 18 Aug 2017 01:41:54 AM MDT
Group : System Environment/Daemons
Size : 8468465
License : Apache License, Version 2.0
Signature : (none)
Source RPM : tomcat-7.0.69-AV10.src.rpm
Build Date : Fri 24 Jun 2016 01:59:37 AM MDT
Build Host : puiqx3650dev03.apac.avaya.com
Relocations : /usr/share
Packager : Avaya, Inc.
Vendor : Apache Software Foundation
URL : http://tomcat.apache.org/
Summary : Core Services Apache Jakarta Servlet/JSP server
Description : Servlet container for developing Web applications in Java. Used
as basis for the
Reference Implementation of the Servlet and JSP specificiations. This RPM
mirrors the layout used by official Tomcat distributions on all other OSes and
linux variants. If you want a FHS-based package - be warned that tomcat may not
work very well. (TODO: add more warnings !). If you still want FHS-based
package - please use jpackage.org or an RPM that is using the same exact layout.
Can you please help on this.
Regards,
Radhika
