You should try 2.20.0 that's being released now

On Mon, Dec 20, 2021 at 10:11 AM David Martin <dav...@qoritek.com> wrote:
>
> FYI found a similar issue as the OPs after upgrading from 2.16 to 2.18 -
> perhaps 2.18 is better at detecting slow paths, can see some related
> tickets in 2.17 and 2.18 release notes.
>
> Turned out to be MiB/s throttling on cloud storage - increasing the
> allocation increased the threshold at which throttling would occur.
>
> Possibly decreasing the page size configured in broker.xml would help
> reduce overall IOPS consumption in this type of deployment?
>
>
> Regards,
>
> Dave
>
>
> On Fri, 29 Oct 2021 at 23:27, Clebert Suconic <clebert.suco...@gmail.com>
> wrote:
>
> > The critical analyzer is used to protected against dead locks.
> >
> > if you have a box that's not acting as fast for any other reason, I
> > suggest you turn it off. (Use Log instead of HALT on the Critical
> > Analyzer options).
> >
> > On Wed, Oct 27, 2021 at 11:09 AM Eric Kaplan <ericdotkap...@gmail.com>
> > wrote:
> > >
> > > I am seeing this as well. If this is a load issue, how could I better
> > understand what that load is? I have a cluster of 7 machines with approx 30
> > producers/consumers running across them. The producers produce small
> > messages at regular intervals (15 seconds). We are using non-durable
> > subscribers non-persistent topics. The hornetq data directory is on nfs. I
> > have gotten it a few times though I can't reproduce regularly.
> > >
> > > We are moving from HornetQ (believe it or not) to Artemis 2.19 and would
> > like to understand this issue.
> > >
> > > My log shows the following:
> > >
> > > 2021-10-24 13:04:03,320 INFO
> > [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting
> > ActiveMQ Artemis Server
> > > 2021-10-24 13:04:03,353 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221000: live Message Broker is starting with configuration Broker
> > Configuration
> > (clustered=false,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
> > > 2021-10-24 13:04:03,415 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221013: Using NIO Journal
> > > 2021-10-24 13:04:03,467 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size
> > (-Xmx). being defined as 1,073,741,824
> > > 2021-10-24 13:04:03,491 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043: Protocol module found: [artemis-server]. Adding protocol support
> > for: CORE
> > > 2021-10-24 13:04:03,491 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol
> > support for: AMQP
> > > 2021-10-24 13:04:03,492 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding
> > protocol support for: HORNETQ
> > > 2021-10-24 13:04:03,492 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol
> > support for: MQTT
> > > 2021-10-24 13:04:03,492 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding
> > protocol support for: OPENWIRE
> > > 2021-10-24 13:04:03,492 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol
> > support for: STOMP
> > > 2021-10-24 13:04:03,536 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221034: Waiting indefinitely to obtain live lock
> > > 2021-10-24 13:04:03,536 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221035: Live Server Obtained live lock
> > > 2021-10-24 13:04:04,037 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at 10-234-60-76.ssnc-corp.cloud:61616 for
> > protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
> > > 2021-10-24 13:04:04,039 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > > 2021-10-24 13:04:04,040 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.19.0 [armanta,
> > nodeID=ecbe3206-32c0-11ec-8a24-0050569966af]
> > > 2021-10-24 13:04:04,348 INFO
> > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > activemq-branding plugin
> > > 2021-10-24 13:04:04,514 INFO
> > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > artemis-plugin plugin
> > > 2021-10-24 13:04:05,196 INFO  [io.hawt.HawtioContextListener]
> > Initialising hawtio services
> > > 2021-10-24 13:04:05,214 INFO  [io.hawt.system.ConfigManager]
> > Configuration will be discovered via system properties
> > > 2021-10-24 13:04:05,217 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > Hawtio 2.14.0
> > > 2021-10-24 13:04:05,223 INFO
> > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > authentication filter, JAAS realm: "activemq" authorized role(s): "guest"
> > role principal classes:
> > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2021-10-24 13:04:05,229 INFO  [io.hawt.web.auth.LoginRedirectFilter]
> > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2021-10-24 13:04:05,242 INFO  [io.hawt.web.proxy.ProxyServlet] Proxy
> > servlet is disabled
> > > 2021-10-24 13:04:05,247 INFO
> > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > property: [key=policyLocation,
> > value=file:/shared_dir/awa_install/awa/awa_3.0.13.4dev.597331/jms/artemis/instance/etc/jolokia-access.xml]
> > > 2021-10-24 13:04:05,340 INFO  [org.apache.activemq.artemis] AMQ241001:
> > HTTP Server started at http://localhost:8161
> > > 2021-10-24 13:04:05,340 INFO  [org.apache.activemq.artemis] AMQ241002:
> > Artemis Jolokia REST API available at
> > http://localhost:8161/console/jolokia
> > > 2021-10-24 13:04:05,340 INFO  [org.apache.activemq.artemis] AMQ241004:
> > Artemis Console available at http://localhost:8161/console
> > > 2021-10-24 13:14:03,338 WARN
> > [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component
> > org.apache.activemq.artemis.core.server.impl.QueueImpl is expired on path 2
> > > 2021-10-24 13:14:03,339 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ224107: The Critical Analyzer detected slow paths on the broker.  It is
> > recommended that you enable trace logs on
> > org.apache.activemq.artemis.utils.critical while you troubleshoot this
> > issue. You should disable the trace logs when you have finished
> > troubleshooting.
> > > 2021-10-24 13:14:03,339 ERROR [org.apache.activemq.artemis.core.server]
> > AMQ224079: The process for the virtual machine will be killed, as component
> > QueueImpl[name=d28a82eb-8a73-4148-bcb3-008abad893b5,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=armanta],
> > temp=true]@71707b9b is not responsive
> > > 2021-10-24 13:14:03,376 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222199: Thread dump:
> > *******************************************************************************
> > > Complete Thread dump
> > > "Thread-40 (activemq-netty-threads)" Id=117 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-39 (activemq-netty-threads)" Id=116 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-38 (activemq-netty-threads)" Id=115 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-37 (activemq-netty-threads)" Id=114 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-36 (activemq-netty-threads)" Id=113 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-35 (activemq-netty-threads)" Id=112 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-34 (activemq-netty-threads)" Id=111 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-33 (activemq-netty-threads)" Id=110 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-32 (activemq-netty-threads)" Id=109 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-31 (activemq-netty-threads)" Id=108 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-30 (activemq-netty-threads)" Id=107 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-29 (activemq-netty-threads)" Id=106 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-28 (activemq-netty-threads)" Id=105 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-27 (activemq-netty-threads)" Id=104 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-26 (activemq-netty-threads)" Id=103 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-25 (activemq-netty-threads)" Id=102 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-24 (activemq-netty-threads)" Id=101 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-23 (activemq-netty-threads)" Id=100 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-22 (activemq-netty-threads)" Id=99 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-21 (activemq-netty-threads)" Id=98 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-20 (activemq-netty-threads)" Id=97 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-19 (activemq-netty-threads)" Id=96 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-18 (activemq-netty-threads)" Id=95 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-17 (activemq-netty-threads)" Id=94 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-16 (activemq-netty-threads)" Id=93 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-15 (activemq-netty-threads)" Id=92 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-25
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=91 RUNNABLE
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-26
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=90 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-28
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=88 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-29
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=89 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-27
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=87 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-24
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=86 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-20
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=85 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-23
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=84 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-22
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=83 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-21
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=82 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-15
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=81 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-16
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=80 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-19
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=79 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-18
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=78 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-17
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=77 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-14
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=76 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-13
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=75 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-12
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=74 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-14 (activemq-netty-threads)" Id=73 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-13 (activemq-netty-threads)" Id=72 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-11
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=71 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-10
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=70 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-9
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=69 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-8
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=68 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-12 (activemq-netty-threads)" Id=67 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-11 (activemq-netty-threads)" Id=66 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-10 (activemq-netty-threads)" Id=65 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-7
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=64 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-9 (activemq-netty-threads)" Id=63 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-6
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=62 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-5
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=61 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-4
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=60 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-8 (activemq-netty-threads)" Id=59 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-7 (activemq-netty-threads)" Id=58 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-6 (activemq-netty-threads)" Id=57 RUNNABLE
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-5 (activemq-netty-threads)" Id=56 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-4 (activemq-netty-threads)" Id=55 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-3
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=53 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-3 (activemq-netty-threads)" Id=52 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-2
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=49 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-1
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=48 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-0
> > (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@223aa2f7)"
> > Id=47 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8a81fe
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-2 (activemq-netty-threads)" Id=46 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-1 (activemq-netty-threads)" Id=45 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "DestroyJavaVM" Id=44 RUNNABLE
> > >
> > >
> > > "qtp553759818-43-acceptor-1@500b3746-ServerConnector@5c153b9e{HTTP/1.1,
> > (http/1.1)}{localhost:8161}" Id=43 RUNNABLE (in native)
> > >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> > >         at sun.nio.ch
> > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
> > >         at sun.nio.ch
> > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
> > >         -  locked java.lang.Object@4cd49917
> > >         at
> > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
> > >         at
> > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-42-acceptor-0@7c31424f-ServerConnector@5c153b9e{HTTP/1.1,
> > (http/1.1)}{localhost:8161}" Id=42 BLOCKED on java.lang.Object@4cd49917
> > owned by 
> > "qtp553759818-43-acceptor-1@500b3746-ServerConnector@5c153b9e{HTTP/1.1,
> > (http/1.1)}{localhost:8161}" Id=43
> > >         at sun.nio.ch
> > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233)
> > >         -  blocked on java.lang.Object@4cd49917
> > >         at
> > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
> > >         at
> > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "Session-HouseKeeper-7e985ce9-1" Id=41 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7edaeb
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7edaeb
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-40" Id=40 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@34ca9ca7
> > >         -  locked java.util.Collections$UnmodifiableSet@5ea57eb1
> > >         -  locked sun.nio.ch.EPollSelectorImpl@36514853
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-39" Id=39 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@4f0175f0
> > >         -  locked java.util.Collections$UnmodifiableSet@2fcef713
> > >         -  locked sun.nio.ch.EPollSelectorImpl@5aeb44ad
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-38" Id=38 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@66c49c73
> > >         -  locked java.util.Collections$UnmodifiableSet@222edbae
> > >         -  locked sun.nio.ch.EPollSelectorImpl@7a80d697
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-37" Id=37 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@7ace3b46
> > >         -  locked java.util.Collections$UnmodifiableSet@382b0768
> > >         -  locked sun.nio.ch.EPollSelectorImpl@6c0c5868
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-36" Id=36 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@10441f10
> > >         -  locked java.util.Collections$UnmodifiableSet@1bdb1255
> > >         -  locked sun.nio.ch.EPollSelectorImpl@7bab4601
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-35" Id=35 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@75ea9e04
> > >         -  locked java.util.Collections$UnmodifiableSet@45e6a336
> > >         -  locked sun.nio.ch.EPollSelectorImpl@66c5865
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-34" Id=34 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@27d69de7
> > >         -  locked java.util.Collections$UnmodifiableSet@66d2da7
> > >         -  locked sun.nio.ch.EPollSelectorImpl@76e2c14f
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "qtp553759818-33" Id=33 RUNNABLE (in native)
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         -  locked sun.nio.ch.Util$3@12a91086
> > >         -  locked java.util.Collections$UnmodifiableSet@1839e91
> > >         -  locked sun.nio.ch.EPollSelectorImpl@6eec9b13
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.nioSelect(ManagedSelector.java:183)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector.select(ManagedSelector.java:190)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
> > >         at org.eclipse.jetty.io
> > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > >         at
> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > >         at 
> > > org.eclipse.jetty.io.ManagedSelector$$Lambda$113/968957548.run(Unknown
> > Source)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> > >         at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "Thread-0 (activemq-netty-threads)" Id=32 RUNNABLE (in native)
> > >         at io.netty.channel.epoll.Native.epollWait(Native Method)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:192)
> > >         at io.netty.channel.epoll.Native.epollWait(Native.java:185)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
> > >         at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
> > >         at
> > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> > >         at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-4 (ActiveMQ-scheduled-threads)" Id=31 WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-3 (ActiveMQ-scheduled-threads)" Id=30 TIMED_WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-2 (ActiveMQ-scheduled-threads)" Id=29 WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "activemq-failure-check-thread" Id=28 TIMED_WAITING on
> > java.util.concurrent.CountDownLatch$Sync@5493cb52
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on java.util.concurrent.CountDownLatch$Sync@5493cb52
> > >         at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> > >         at
> > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> > >         at
> > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:787)
> > >
> > >
> > > "Thread-1 (ActiveMQ-scheduled-threads)" Id=25 WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "Thread-0 (ActiveMQ-scheduled-threads)" Id=24 WAITING on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e64d738
> > >         at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >
> > > "activemq-buffer-timeout" Id=23 WAITING on
> > java.util.concurrent.Semaphore$NonfairSync@4979a047
> > >         at sun.misc.Unsafe.park(Native Method)
> > >         -  waiting on java.util.concurrent.Semaphore$NonfairSync@4979a047
> > >         at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> > >         at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > >         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> > >         at
> > org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:479)
> > >         at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > "Thread-0 (-scheduled-threads)" Id=22 RUNNABLE
> > >         at sun.management.ThreadImpl.dumpThreads0(Native Method)
> > >         at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:496)
> > >         at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:484)
> > >         at
> > org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47)
> > >         at
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:1140)
> > >         at
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$0(ActiveMQServerImpl.java:734)
> > >         at
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$35/1117519786.run(Unknown
> > Source)
> > >         at
> > org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireActions(CriticalAnalyzerImpl.java:155)
> > >         at
> > org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:140)
> > >         at
> > org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53)
> > >         at
> > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:313)
> > >         at
> > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.bookedRunForScheduler(ActiveMQScheduledComponent.java:328)
> > >         at
> > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForScheduler(ActiveMQScheduledComponent.java:339)
> > >         at
> > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$start$0(ActiveMQScheduledComponent.java:166)
> > >         at
> > org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$$Lambda$34/341138954.run(Unknown
> > Source)
> > >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > >         at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > >         at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >         at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > >
> > >         Number of locked synchronizers = 1
> > >         - java.util.concurrent.ThreadPoolExecutor$Worker@3d5c822d
> > >
> > >
> > > "ActiveMQ Artemis Server Shutdown Timer" Id=20 TIMED_WAITING on
> > java.util.TaskQueue@43c78e78
> > >         at java.lang.Object.wait(Native Method)
> > >         -  waiting on java.util.TaskQueue@43c78e78
> > >         at java.util.TimerThread.mainLoop(Timer.java:552)
> > >         at java.util.TimerThread.run(Timer.java:505)
> > >
> > >
> > > "Signal Dispatcher" Id=5 RUNNABLE
> > >
> > >
> > > "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@6bb24c2f
> > >         at java.lang.Object.wait(Native Method)
> > >         -  waiting on java.lang.ref.ReferenceQueue$Lock@6bb24c2f
> > >         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> > >         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> > >         at
> > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
> > >
> > >
> > > "Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@21b1c350
> > >         at java.lang.Object.wait(Native Method)
> > >         -  waiting on java.lang.ref.Reference$Lock@21b1c350
> > >         at java.lang.Object.wait(Object.java:502)
> > >         at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> > >         at
> > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> > >
> > >
> > >
> > ===============================================================================
> > > End Thread dump
> > >
> > *******************************************************************************
> > >
> > >
> > >
> > > On 2021/09/16 14:46:19, Justin Bertram <jbert...@apache.org> wrote:
> > > > I think perhaps you've misunderstood the issue reported via
> > ARTEMIS-3271
> > > > [1]. The issue was that the critical analyzer [2] would sometimes
> > report
> > > > false positives. The fix eliminated the false positives, but it didn't
> > > > eliminate *all* reports. You may be hitting an actual issue. In order
> > to
> > > > investigate further you need to provide the thread dump logged by the
> > > > critical analyzer.
> > > >
> > > > For what it's worth, you can configure the critical analyzer to just
> > log
> > > > the issue and not shutdown the broker. The documentation [2] has the
> > > > configuration details.
> > > >
> > > >
> > > > Justin
> > > >
> > > > [1] https://issues.apache.org/jira/browse/ARTEMIS-3271
> > > > [2]
> > > >
> > https://activemq.apache.org/components/artemis/documentation/latest/critical-analysis.html
> > > >
> > > > On Thu, Sep 16, 2021 at 8:54 AM foo bar <statuario...@gmail.com>
> > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > We are using ArtemisMQ version 2.15. We are attempting to upgrade to
> > 2.18.
> > > > > On 2.18, we generate some load with a couple of threads. We are
> > getting the
> > > > > exact same error noted here:
> > > > >
> > > > >
> > https://www.mail-archive.com/issues@activemq.apache.org/msg58016.html
> > > > >
> > > > > ```
> > > > > [org.apache.activemq.artemis.utils.critical.CriticalMeasure]
> > Component
> > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl is expired on
> > path 0
> > > > >  2021-05-15 10:00:49,258
> > > > > INFO  [org.apache.activemq.artemis.core.server]
> > > > > AMQ224107: The Critical Analyzer detected slow paths on the broker.
> > It is
> > > > > recommended that you enable trace logs on
> > > > > org.apache.activemq.artemis.utils.critical while you troubleshoot
> > this
> > > > > issue.
> > > > > You should disable the trace logs when you have finished
> > troubleshooting.
> > > > >    2021-05-15 10:00:49,258 ERROR
> > [org.apache.activemq.artemis.core.server]
> > > > > AMQ224079: The process for the virtual machine will be killed, as
> > component
> > > > > QueueImpl[name=q_7, postOffice=PostOfficeImpl
> > > > >
> > > > >
> > [server=ActiveMQServerImpl::serverUUID=64e3f254-b585-11eb-8f02-000f532921b8],
> > > > >
> > > > > temp=false]@9a33f70 is not responsive
> > > > >    2021-05-15 10:00:50,429 WARN
> > [org.apache.activemq.artemis.core.server]
> > > > > AMQ222199: Thread dump:
> > > > > ```
> > > > >
> > > > > I do see a ticket here noting that the issue has been resolved in
> > 2.18,
> > > > > although we just encountered it.
> > > > >
> > > > >
> > > > >
> > https://issues.apache.org/jira/browse/ARTEMIS-3271?focusedWorklogId=597221&page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel#worklog-597221
> > > > >
> > > > > Once this happens the broker is down and needs to be restarted. Is
> > there a
> > > > > workaround for this?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> >
> >
> >
> > --
> > Clebert Suconic
> >



-- 
Clebert Suconic

Reply via email to