Thanks Clebert - I guess this is related to ARTEMIS-3587 so will go to 2.20 asap.
On Mon, 20 Dec 2021 at 20:05, Clebert Suconic <clebert.suco...@gmail.com> wrote: > You should try 2.20.0 that's being released now > > On Mon, Dec 20, 2021 at 10:11 AM David Martin <dav...@qoritek.com> wrote: > > > > FYI found a similar issue as the OPs after upgrading from 2.16 to 2.18 - > > perhaps 2.18 is better at detecting slow paths, can see some related > > tickets in 2.17 and 2.18 release notes. > > > > Turned out to be MiB/s throttling on cloud storage - increasing the > > allocation increased the threshold at which throttling would occur. > > > > Possibly decreasing the page size configured in broker.xml would help > > reduce overall IOPS consumption in this type of deployment? > > > > > > Regards, > > > > Dave > > > > > > On Fri, 29 Oct 2021 at 23:27, Clebert Suconic <clebert.suco...@gmail.com > > > > wrote: > > > > > The critical analyzer is used to protected against dead locks. > > > > > > if you have a box that's not acting as fast for any other reason, I > > > suggest you turn it off. (Use Log instead of HALT on the Critical > > > Analyzer options). > > > > > > On Wed, Oct 27, 2021 at 11:09 AM Eric Kaplan <ericdotkap...@gmail.com> > > > wrote: > > > > > > > > I am seeing this as well. If this is a load issue, how could I better > > > understand what that load is? I have a cluster of 7 machines with > approx 30 > > > producers/consumers running across them. The producers produce small > > > messages at regular intervals (15 seconds). We are using non-durable > > > subscribers non-persistent topics. The hornetq data directory is on > nfs. I > > > have gotten it a few times though I can't reproduce regularly. > > > > > > > > We are moving from HornetQ (believe it or not) to Artemis 2.19 and > would > > > like to understand this issue. > > > > > > > > My log shows the following: > > > > > > > > 2021-10-24 13:04:03,320 INFO > > > [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting > > > ActiveMQ Artemis Server > > > > 2021-10-24 13:04:03,353 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221000: live Message Broker is starting with configuration Broker > > > Configuration > > > > (clustered=false,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging) > > > > 2021-10-24 13:04:03,415 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221013: Using NIO Journal > > > > 2021-10-24 13:04:03,467 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size > > > (-Xmx). being defined as 1,073,741,824 > > > > 2021-10-24 13:04:03,491 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221043: Protocol module found: [artemis-server]. Adding protocol > support > > > for: CORE > > > > 2021-10-24 13:04:03,491 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding > protocol > > > support for: AMQP > > > > 2021-10-24 13:04:03,492 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding > > > protocol support for: HORNETQ > > > > 2021-10-24 13:04:03,492 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding > protocol > > > support for: MQTT > > > > 2021-10-24 13:04:03,492 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding > > > protocol support for: OPENWIRE > > > > 2021-10-24 13:04:03,492 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding > protocol > > > support for: STOMP > > > > 2021-10-24 13:04:03,536 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221034: Waiting indefinitely to obtain live lock > > > > 2021-10-24 13:04:03,536 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221035: Live Server Obtained live lock > > > > 2021-10-24 13:04:04,037 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221020: Started EPOLL Acceptor at > 10-234-60-76.ssnc-corp.cloud:61616 for > > > protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE] > > > > 2021-10-24 13:04:04,039 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221007: Server is now live > > > > 2021-10-24 13:04:04,040 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.19.0 > [armanta, > > > nodeID=ecbe3206-32c0-11ec-8a24-0050569966af] > > > > 2021-10-24 13:04:04,348 INFO > > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized > > > activemq-branding plugin > > > > 2021-10-24 13:04:04,514 INFO > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized > > > artemis-plugin plugin > > > > 2021-10-24 13:04:05,196 INFO [io.hawt.HawtioContextListener] > > > Initialising hawtio services > > > > 2021-10-24 13:04:05,214 INFO [io.hawt.system.ConfigManager] > > > Configuration will be discovered via system properties > > > > 2021-10-24 13:04:05,217 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to > > > Hawtio 2.14.0 > > > > 2021-10-24 13:04:05,223 INFO > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio > > > authentication filter, JAAS realm: "activemq" authorized role(s): > "guest" > > > role principal classes: > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal" > > > > 2021-10-24 13:04:05,229 INFO [io.hawt.web.auth.LoginRedirectFilter] > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout > > > > 2021-10-24 13:04:05,242 INFO [io.hawt.web.proxy.ProxyServlet] Proxy > > > servlet is disabled > > > > 2021-10-24 13:04:05,247 INFO > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden > > > property: [key=policyLocation, > > > > value=file:/shared_dir/awa_install/awa/awa_3.0.13.4dev.597331/jms/artemis/instance/etc/jolokia-access.xml] > > > > 2021-10-24 13:04:05,340 INFO [org.apache.activemq.artemis] > AMQ241001: > > > HTTP Server started at http://localhost:8161 > > > > 2021-10-24 13:04:05,340 INFO [org.apache.activemq.artemis] > AMQ241002: > > > Artemis Jolokia REST API available at > > > http://localhost:8161/console/jolokia > > > > 2021-10-24 13:04:05,340 INFO [org.apache.activemq.artemis] > AMQ241004: > > > Artemis Console available at http://localhost:8161/console > > > > 2021-10-24 13:14:03,338 WARN > > > [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component > > > org.apache.activemq.artemis.core.server.impl.QueueImpl is expired on > path 2 > > > > 2021-10-24 13:14:03,339 INFO > [org.apache.activemq.artemis.core.server] > > > AMQ224107: The Critical Analyzer detected slow paths on the broker. > It is > > > recommended that you enable trace logs on > > > org.apache.activemq.artemis.utils.critical while you troubleshoot this > > > issue. You should disable the trace logs when you have finished > > > troubleshooting. > > > > 2021-10-24 13:14:03,339 ERROR > [org.apache.activemq.artemis.core.server] > > > AMQ224079: The process for the virtual machine will be killed, as > component > > > QueueImpl[name=d28a82eb-8a73-4148-bcb3-008abad893b5, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=armanta], > > > temp=true]@71707b9b is not responsive > > > > 2021-10-24 13:14:03,376 WARN > [org.apache.activemq.artemis.core.server] > > > AMQ222199: Thread dump: > > > > ******************************************************************************* > > > > Complete Thread dump > > > > "Thread-40 (activemq-netty-threads)" Id=117 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-39 (activemq-netty-threads)" Id=116 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-38 (activemq-netty-threads)" Id=115 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-37 (activemq-netty-threads)" Id=114 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-36 (activemq-netty-threads)" Id=113 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-35 (activemq-netty-threads)" Id=112 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-34 (activemq-netty-threads)" Id=111 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-33 (activemq-netty-threads)" Id=110 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-32 (activemq-netty-threads)" Id=109 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-31 (activemq-netty-threads)" Id=108 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-30 (activemq-netty-threads)" Id=107 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-29 (activemq-netty-threads)" Id=106 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-28 (activemq-netty-threads)" Id=105 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-27 (activemq-netty-threads)" Id=104 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-26 (activemq-netty-threads)" Id=103 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-25 (activemq-netty-threads)" Id=102 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-24 (activemq-netty-threads)" Id=101 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-23 (activemq-netty-threads)" Id=100 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-22 (activemq-netty-threads)" Id=99 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-21 (activemq-netty-threads)" Id=98 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-20 (activemq-netty-threads)" Id=97 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-19 (activemq-netty-threads)" Id=96 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-18 (activemq-netty-threads)" Id=95 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-17 (activemq-netty-threads)" Id=94 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-16 (activemq-netty-threads)" Id=93 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-15 (activemq-netty-threads)" Id=92 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-25 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=91 RUNNABLE > > > > at sun.misc.Unsafe.park(Native Method) > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-26 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=90 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-28 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=88 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-29 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=89 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-27 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=87 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-24 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=86 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-20 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=85 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-23 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=84 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-22 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=83 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-21 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=82 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-15 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=81 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-16 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=80 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-19 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=79 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-18 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=78 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-17 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=77 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-14 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=76 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-13 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=75 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-12 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=74 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-14 (activemq-netty-threads)" Id=73 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-13 (activemq-netty-threads)" Id=72 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-11 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=71 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-10 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=70 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-9 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=69 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-8 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=68 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-12 (activemq-netty-threads)" Id=67 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-11 (activemq-netty-threads)" Id=66 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-10 (activemq-netty-threads)" Id=65 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-7 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=64 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-9 (activemq-netty-threads)" Id=63 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-6 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=62 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-5 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=61 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-4 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=60 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-8 (activemq-netty-threads)" Id=59 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-7 (activemq-netty-threads)" Id=58 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-6 (activemq-netty-threads)" Id=57 RUNNABLE > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-5 (activemq-netty-threads)" Id=56 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-4 (activemq-netty-threads)" Id=55 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-3 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=53 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-3 (activemq-netty-threads)" Id=52 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-2 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=49 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-1 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=48 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-0 > > > > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7 > )" > > > Id=47 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-2 (activemq-netty-threads)" Id=46 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-1 (activemq-netty-threads)" Id=45 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "DestroyJavaVM" Id=44 RUNNABLE > > > > > > > > > > > > "qtp553759818-43-acceptor-1@500b3746-ServerConnector > @5c153b9e{HTTP/1.1, > > > (http/1.1)}{localhost:8161}" Id=43 RUNNABLE (in native) > > > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > > > at sun.nio.ch > > > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) > > > > at sun.nio.ch > > > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249) > > > > - locked java.lang.Object@4cd49917 > > > > at > > > > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388) > > > > at > > > > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-42-acceptor-0@7c31424f-ServerConnector > @5c153b9e{HTTP/1.1, > > > (http/1.1)}{localhost:8161}" Id=42 BLOCKED on java.lang.Object@4cd49917 > > > owned by "qtp553759818-43-acceptor-1@500b3746-ServerConnector > @5c153b9e{HTTP/1.1, > > > (http/1.1)}{localhost:8161}" Id=43 > > > > at sun.nio.ch > > > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233) > > > > - blocked on java.lang.Object@4cd49917 > > > > at > > > > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388) > > > > at > > > > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "Session-HouseKeeper-7e985ce9-1" Id=41 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7edaeb > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7edaeb > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-40" Id=40 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@34ca9ca7 > > > > - locked java.util.Collections$UnmodifiableSet@5ea57eb1 > > > > - locked sun.nio.ch.EPollSelectorImpl@36514853 > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-39" Id=39 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@4f0175f0 > > > > - locked java.util.Collections$UnmodifiableSet@2fcef713 > > > > - locked sun.nio.ch.EPollSelectorImpl@5aeb44ad > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-38" Id=38 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@66c49c73 > > > > - locked java.util.Collections$UnmodifiableSet@222edbae > > > > - locked sun.nio.ch.EPollSelectorImpl@7a80d697 > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-37" Id=37 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@7ace3b46 > > > > - locked java.util.Collections$UnmodifiableSet@382b0768 > > > > - locked sun.nio.ch.EPollSelectorImpl@6c0c5868 > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-36" Id=36 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@10441f10 > > > > - locked java.util.Collections$UnmodifiableSet@1bdb1255 > > > > - locked sun.nio.ch.EPollSelectorImpl@7bab4601 > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-35" Id=35 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@75ea9e04 > > > > - locked java.util.Collections$UnmodifiableSet@45e6a336 > > > > - locked sun.nio.ch.EPollSelectorImpl@66c5865 > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-34" Id=34 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@27d69de7 > > > > - locked java.util.Collections$UnmodifiableSet@66d2da7 > > > > - locked sun.nio.ch.EPollSelectorImpl@76e2c14f > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "qtp553759818-33" Id=33 RUNNABLE (in native) > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked sun.nio.ch.Util$3@12a91086 > > > > - locked java.util.Collections$UnmodifiableSet@1839e91 > > > > - locked sun.nio.ch.EPollSelectorImpl@6eec9b13 > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.nioSelect(ManagedSelector.java:183) > > > > at org.eclipse.jetty.io > > > .ManagedSelector.select(ManagedSelector.java:190) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) > > > > at org.eclipse.jetty.io > > > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > at org.eclipse.jetty.io > .ManagedSelector$$Lambda$113/968957548.run(Unknown > > > Source) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "Thread-0 (activemq-netty-threads)" Id=32 RUNNABLE (in native) > > > > at io.netty.channel.epoll.Native.epollWait(Native Method) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:192) > > > > at io.netty.channel.epoll.Native.epollWait(Native.java:185) > > > > at > > > > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) > > > > at > > > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > > > > at > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-4 (ActiveMQ-scheduled-threads)" Id=31 WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-3 (ActiveMQ-scheduled-threads)" Id=30 TIMED_WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-2 (ActiveMQ-scheduled-threads)" Id=29 WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "activemq-failure-check-thread" Id=28 TIMED_WAITING on > > > java.util.concurrent.CountDownLatch$Sync@5493cb52 > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > java.util.concurrent.CountDownLatch$Sync@5493cb52 > > > > at > > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) > > > > at > > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) > > > > at > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:787) > > > > > > > > > > > > "Thread-1 (ActiveMQ-scheduled-threads)" Id=25 WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "Thread-0 (ActiveMQ-scheduled-threads)" Id=24 WAITING on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738 > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > > > > > "activemq-buffer-timeout" Id=23 WAITING on > > > java.util.concurrent.Semaphore$NonfairSync@4979a047 > > > > at sun.misc.Unsafe.park(Native Method) > > > > - waiting on > java.util.concurrent.Semaphore$NonfairSync@4979a047 > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > > > > at > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > > > > at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) > > > > at > > > > org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:479) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > "Thread-0 (-scheduled-threads)" Id=22 RUNNABLE > > > > at sun.management.ThreadImpl.dumpThreads0(Native Method) > > > > at > sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:496) > > > > at > sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:484) > > > > at > > > > org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47) > > > > at > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:1140) > > > > at > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$0(ActiveMQServerImpl.java:734) > > > > at > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$35/1117519786.run(Unknown > > > Source) > > > > at > > > > org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireActions(CriticalAnalyzerImpl.java:155) > > > > at > > > > org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:140) > > > > at > > > > org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53) > > > > at > > > > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:313) > > > > at > > > > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.bookedRunForScheduler(ActiveMQScheduledComponent.java:328) > > > > at > > > > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForScheduler(ActiveMQScheduledComponent.java:339) > > > > at > > > > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$start$0(ActiveMQScheduledComponent.java:166) > > > > at > > > > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$$Lambda$34/341138954.run(Unknown > > > Source) > > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > at > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > > > Number of locked synchronizers = 1 > > > > - java.util.concurrent.ThreadPoolExecutor$Worker@3d5c822d > > > > > > > > > > > > "ActiveMQ Artemis Server Shutdown Timer" Id=20 TIMED_WAITING on > > > java.util.TaskQueue@43c78e78 > > > > at java.lang.Object.wait(Native Method) > > > > - waiting on java.util.TaskQueue@43c78e78 > > > > at java.util.TimerThread.mainLoop(Timer.java:552) > > > > at java.util.TimerThread.run(Timer.java:505) > > > > > > > > > > > > "Signal Dispatcher" Id=5 RUNNABLE > > > > > > > > > > > > "Finalizer" Id=3 WAITING on > java.lang.ref.ReferenceQueue$Lock@6bb24c2f > > > > at java.lang.Object.wait(Native Method) > > > > - waiting on java.lang.ref.ReferenceQueue$Lock@6bb24c2f > > > > at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) > > > > at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) > > > > at > > > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216) > > > > > > > > > > > > "Reference Handler" Id=2 WAITING on > java.lang.ref.Reference$Lock@21b1c350 > > > > at java.lang.Object.wait(Native Method) > > > > - waiting on java.lang.ref.Reference$Lock@21b1c350 > > > > at java.lang.Object.wait(Object.java:502) > > > > at > java.lang.ref.Reference.tryHandlePending(Reference.java:191) > > > > at > > > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > > > > > > > > > > > > > > > > =============================================================================== > > > > End Thread dump > > > > > > > > ******************************************************************************* > > > > > > > > > > > > > > > > On 2021/09/16 14:46:19, Justin Bertram <jbert...@apache.org> wrote: > > > > > I think perhaps you've misunderstood the issue reported via > > > ARTEMIS-3271 > > > > > [1]. The issue was that the critical analyzer [2] would sometimes > > > report > > > > > false positives. The fix eliminated the false positives, but it > didn't > > > > > eliminate *all* reports. You may be hitting an actual issue. In > order > > > to > > > > > investigate further you need to provide the thread dump logged by > the > > > > > critical analyzer. > > > > > > > > > > For what it's worth, you can configure the critical analyzer to > just > > > log > > > > > the issue and not shutdown the broker. The documentation [2] has > the > > > > > configuration details. > > > > > > > > > > > > > > > Justin > > > > > > > > > > [1] https://issues.apache.org/jira/browse/ARTEMIS-3271 > > > > > [2] > > > > > > > > > https://activemq.apache.org/components/artemis/documentation/latest/critical-analysis.html > > > > > > > > > > On Thu, Sep 16, 2021 at 8:54 AM foo bar <statuario...@gmail.com> > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > We are using ArtemisMQ version 2.15. We are attempting to > upgrade to > > > 2.18. > > > > > > On 2.18, we generate some load with a couple of threads. We are > > > getting the > > > > > > exact same error noted here: > > > > > > > > > > > > > > > https://www.mail-archive.com/issues@activemq.apache.org/msg58016.html > > > > > > > > > > > > ``` > > > > > > [org.apache.activemq.artemis.utils.critical.CriticalMeasure] > > > Component > > > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl is > expired on > > > path 0 > > > > > > 2021-05-15 10:00:49,258 > > > > > > INFO [org.apache.activemq.artemis.core.server] > > > > > > AMQ224107: The Critical Analyzer detected slow paths on the > broker. > > > It is > > > > > > recommended that you enable trace logs on > > > > > > org.apache.activemq.artemis.utils.critical while you troubleshoot > > > this > > > > > > issue. > > > > > > You should disable the trace logs when you have finished > > > troubleshooting. > > > > > > 2021-05-15 10:00:49,258 ERROR > > > [org.apache.activemq.artemis.core.server] > > > > > > AMQ224079: The process for the virtual machine will be killed, as > > > component > > > > > > QueueImpl[name=q_7, postOffice=PostOfficeImpl > > > > > > > > > > > > > > > > [server=ActiveMQServerImpl::serverUUID=64e3f254-b585-11eb-8f02-000f532921b8], > > > > > > > > > > > > temp=false]@9a33f70 is not responsive > > > > > > 2021-05-15 10:00:50,429 WARN > > > [org.apache.activemq.artemis.core.server] > > > > > > AMQ222199: Thread dump: > > > > > > ``` > > > > > > > > > > > > I do see a ticket here noting that the issue has been resolved in > > > 2.18, > > > > > > although we just encountered it. > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARTEMIS-3271?focusedWorklogId=597221&page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel#worklog-597221 > > > > > > > > > > > > Once this happens the broker is down and needs to be restarted. > Is > > > there a > > > > > > workaround for this? > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > -- > > > Clebert Suconic > > > > > > > -- > Clebert Suconic >
