Radhika, On Tue, Sep 19, 2017 at 2:21 PM, Peddi, Radhika (Radhika) <rpe...@avaya.com> wrote: > Hi, > > We have upgraded RHEL 7.2 to RHEL 7.4. After upgrade when we are running > performance testing of our application we are seeing below error in > catalina.out. > > java.lang.OutOfMemoryError: unable to create new native thread Exception in > Catalina.out after upgrading to RHEL 7.4 >
This simply indicates that JVM was trying to create a new thread and OS can't create any new thread simply because native memory space was exhausted. This limit is very much platform dependent. Is your app opening too many threads? On my Windows 10 laptop with 32GB RAM and JDK 1.8, I can open 530000+ threads before run into this OOM problem. You can find it by monitoring Tomcat instance. So far I ran into this problem one time - couple of years ago. We got away after adding some memory and by reducing thread stack size. Another thing can happen, you might have lost some OS setting(which was used to bump up the default OS limit) while upgrading from one version to another; which was preventing this issue from occurring so far - just speculating. And this is not a Tomcat's problem. :) > Attached is the thread dump. > > We are using Tomcat 7.0.69. > > As per REDHAT we have increased tomcat5 user which used to run Tomcat process > in /etc/security/limits.d/20-nproc.conf like below > > * soft nproc 4096 > root soft nproc unlimited > tomcat5 soft nproc 8192 > > All the processes are used by Tomcat. If we increase the value to 16K or 32K > all threads are consumed and they are in Waiting(Parking) state. > > "http-bio-8443-exec-56" #973 daemon prio=5 os_prio=0 tid=0x00007efc8c029800 > nid=0x10e7b waiting on condition [0x00007ef9bc7ee000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000003c454c008> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) > at > org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > > "http-bio-8443-exec-55" #972 daemon prio=5 os_prio=0 tid=0x00007efc0401e000 > nid=0x10e7a waiting on condition [0x00007ef9bc82f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000003c454c008> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) > at > org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > > They are not getting release. After some time Tomcat getting stopped > automatically. You may need the output of below from our setup. > > [root@iqsvtk05<mailto:root@iqsvtk05> ~]# rpm -qif > /opt/coreservices/tomcat-7.0.69/logs/ > Name : tomcat > Version : 7.0.69 > Release : AV10 > Architecture: x86_64 > Install Date: Fri 18 Aug 2017 01:41:54 AM MDT > Group : System Environment/Daemons > Size : 8468465 > License : Apache License, Version 2.0 > Signature : (none) > Source RPM : tomcat-7.0.69-AV10.src.rpm > Build Date : Fri 24 Jun 2016 01:59:37 AM MDT > Build Host : puiqx3650dev03.apac.avaya.com > Relocations : /usr/share > Packager : Avaya, Inc. > Vendor : Apache Software Foundation > URL : http://tomcat.apache.org/ > Summary : Core Services Apache Jakarta Servlet/JSP server > Description : Servlet container for developing Web applications in Java. Used > as basis for the > Reference Implementation of the Servlet and JSP specificiations. This RPM > mirrors the layout used by official Tomcat distributions on all other OSes > and linux variants. If you want a FHS-based package - be warned that tomcat > may not work very well. (TODO: add more warnings !). If you still want > FHS-based package - please use jpackage.org or an RPM that is using the same > exact layout. > > Can you please help on this. > > Regards, > Radhika --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
