vlsi opened a new issue #6035: Pulsar precommit Jenkins jobs consume too much ASF resources and it leads to "unable to create native thread" URL: https://github.com/apache/pulsar/issues/6035 See https://lists.apache.org/thread.html/r9cb0772531814fdf10c82b61fb4bb8d3a187852ddf98ac84754bf778%40%3Cbuilds.apache.org%3E H23 node was unresponsive, and it turned out to have lots of Pulsar Java processes (~14 processes, 9000+ threads): ``` 22058 jenkins 20 0 19.514g 2.156g 33960 S 36.8 2.3 2032:55 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Xmx1G -XX:+UseG1GC -Dpulsar.allocator.pooled=false -Dpulsar.allocator.leak_detection=Advanced -Dpulsar.allocator.exit_on_oom=false -Dlog4j.configurationFile=log4j2.xml -jar /home/jenkins/jenkins-slave/workspace/pulsar_precommit_java8/pulsar-broker/target/surefire/surefirebooter5673414172185975509.jar /home/jenkins/jenkins-slave/workspace/pulsar_precommit_java8/pulsar-broker/target/sur ``` Thread dump includes 1020 threas like ``` "pulsar-9510-20" #73509 prio=5 os_prio=0 tid=0x00007fba40010000 nid=0xa73 waiting on condition [0x00007fb8dd946000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000cd3bf4d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) ``` 733 threads like ``` "bookkeeper-ml-cache-eviction-6747-1" #51441 prio=5 os_prio=0 tid=0x00007fbb1c31c000 nid=0x58be sleeping[0x00007fb8ea6e6000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.cacheEvictionTask(ManagedLedgerFactoryImpl.java:221) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$$Lambda$70/551994588.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) ``` and so on (see the attached threaddump) [pulsar_threaddump.txt.gz](https://github.com/apache/pulsar/files/4050894/pulsar_threaddump.txt.gz)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
