Hi Matt,
There is only one IP and we are planning to reduce the number of maximum
connections, but I am not sure why there is a thread leak with the number of
"Broker Service" and "Inactivity Worker" threads steadily increasing and as a
result it eventually runs out of heap space and crashes. The connections
initiated by the producers and consumers are being closed and I don't see any
connection leaks in code in the client applications. I don't have the full
thread dump with me, here is another snippet that shows the thread stacks, with
the InactivtyMonitor waiting to get a monitor entry that is held by the
BorkerService thread:
ActiveMQ BrokerService[TestBroker] Task-11" #99 daemon prio=5 os_prio=0
cpu=1892.01ms elapsed=4759.33s tid=0x00007fb92c02f000 nid=0x10f8e6 runnable
[0x00007fb875db5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.wait([email protected]/Native Method)
at
sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:120)
at
sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:124)
- locked <0x0000000518b702c8> (a sun.nio.ch.Util$2)
- locked <0x0000000518b70270> (a sun.nio.ch.EPollSelectorImpl)
at
sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:136)
at
org.apache.activemq.transport.nio.NIOSSLTransport.doHandshake(NIOSSLTransport.java:447)
at
org.apache.activemq.transport.nio.NIOSSLTransport.initializeStreams(NIOSSLTransport.java:165)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:543)
at
org.apache.activemq.transport.nio.NIOTransport.doStart(NIOTransport.java:181)
at
org.apache.activemq.transport.nio.NIOSSLTransport.doStart(NIOSSLTransport.java:479)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:172)
at
org.apache.activemq.transport.InactivityMonitor.start(InactivityMonitor.java:52)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
at
org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1077)
- locked <0x0000000518b704c8> (a
org.apache.activemq.broker.jmx.ManagedTransportConnection)
at
org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:235)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:829)
"ActiveMQ InactivityMonitor Worker 7" #7861 daemon prio=5 os_prio=0
cpu=156.66ms elapsed=4726.28s tid=0x00007fb8cc034800 nid=0x11101a waiting for
monitor entry [0x00007fb2253d1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.broker.jmx.ManagedTransportConnection.stopAsync(ManagedTransportConnection.java:67)
- waiting to lock <0x0000000518b704c8> (a
org.apache.activemq.broker.jmx.ManagedTransportConnection)
at
org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:1140)
at
org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:249)
at
org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:215)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:346)
at
org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:95)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:829)
Thanks again,
Karthik
-----Original Message-----
From: Matt Pavlovich <[email protected]>
Sent: Friday, December 13, 2024 3:17 PM
To: [email protected]
Subject: Re: Thread leak/ memory leak in active mq classic instance(v 5.18.3)
[You don't often get email from [email protected]. Learn why this is important
at https://aka.ms/LearnAboutSenderIdentification ]
Attention! - This email has originated from an External Source outside of
eClinicalWorks. Always use caution when opening attachments, clicking links, or
when responding to this email. If you feel this is a phishing scam, please use
the Phish Alert Report button in Outlook.
Hi Kevin-
How many IPs are on that host? 65,000 connections is a risky limit, as there
are only 65,535-per-IP allowed. This number also must account for OS TCP/IP
tuning around how long to leave perviously closed and completed TCP/IP sockets
in a CLOSE_WAIT and TIME_WAIT states before allowing re-use of the port. You
may just be topping out the OS configured limit and need to scale horizontally
or add more IPs for clients to access.
Please share the full thread dump. The partial provided thread dump is missing
the other half of the blocked lock attempt.
Thanks,
Matt Pavlovich
> On Dec 13, 2024, at 2:06 PM, Karthik Venkataraman
> <[email protected]> wrote:
>
> Hello All,
>
> I am using an active-mq classic v 5.18.3 instance and we are running into
> an issue where the active mq process shows steady increase in threads and
> memory utilization and it eventually crashes.
> We have many active connections(thousands) from different producers and
> consumers going into this active-mq instance and we know that the
> system(hardware) where this is run has enough CPU and memory to handle it.
> The maximumConnections value for the NIO transport connector is set to 65000.
> The heap memory for active mq is set to grow up to 12 GB. I got 2 thread
> dumps from the active mq process and it shows the following 2 threads
> steadily increasing:
>
> "ActiveMQ BrokerService[TestBroker] Task-3906" #44297 daemon prio=5 os_prio=0
> cpu=335.99ms elapsed=3286.97s tid=0x00007fb408017000 nid=0x120b9a runnable
> [0x00007fb161a1e000]
> java.lang.Thread.State: RUNNABLE
> at
> sun.nio.ch.EPoll.wait([email protected]/Native<mailto:[email protected]/Native>
> Method)
> at
> sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:120<mailto:[email protected]/EPollSelectorImpl.java:120>)
> at
> sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:124<mailto:[email protected]/SelectorImpl.java:124>)
> - locked <0x0000000530d66360> (a sun.nio.ch.Util$2)
> - locked <0x0000000530d66308> (a sun.nio.ch.EPollSelectorImpl)
> at
> sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:136<mailto:[email protected]/SelectorImpl.java:136>)
> at
> org.apache.activemq.transport.nio.NIOSSLTransport.doHandshake(NIOSSLTransport.java:447)
> at
> org.apache.activemq.transport.nio.NIOSSLTransport.initializeStreams(NIOSSLTransport.java:165)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:543)
> at
> org.apache.activemq.transport.nio.NIOTransport.doStart(NIOTransport.java:181)
> at
> org.apache.activemq.transport.nio.NIOSSLTransport.doStart(NIOSSLTransport.java:479)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:172)
> at
> org.apache.activemq.transport.InactivityMonitor.start(InactivityMonitor.java:52)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
> at
> org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1077)
> - locked <0x0000000530d5a3a0> (a
> org.apache.activemq.broker.jmx.ManagedTransportConnection)
> at
> org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:235)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128<mailto:[email protected]/ThreadPoolExecutor.java:1128>)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628<mailto:[email protected]/ThreadPoolExecutor.java:628>)
> at
> java.lang.Thread.run([email protected]/Thread.java:829<mailto:[email protected]/Thread.java:829>)
>
> "ActiveMQ InactivityMonitor Worker 159" #44315 daemon prio=5 os_prio=0
> cpu=125.60ms elapsed=3284.27s tid=0x00007fb8cc0d0800 nid=0x120bc0 waiting for
> monitor entry [0x00007fb1ef572000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.activemq.broker.jmx.ManagedTransportConnection.stopAsync(ManagedTransportConnection.java:67)
> - waiting to lock <0x0000000530d474b0> (a
> org.apache.activemq.broker.jmx.ManagedTransportConnection)
> at
> org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:1140)
> at
> org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:249)
> at
> org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:215)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:346)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:95)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128<mailto:[email protected]/ThreadPoolExecutor.java:1128>)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628<mailto:[email protected]/ThreadPoolExecutor.java:628>)
> at
> java.lang.Thread.run([email protected]/Thread.java:829<mailto:[email protected]/Thread.java:829>)
>
> Is there any know issue related to thread leaks/memory leak on version 5.18.3?
>
> We have verified that the code used by the producers and consumers are
> closing connections properly. We also verified that we are not seeing any
> major errors in the network between the active mq broker and the clients.
> Could anyone provide insight into why so many of these threads would get
> created and get blocked(InactivityMonitor) and suggest possible steps to
> debug and find the root cause?
>
> Thanks in advance for the help!!
>
> Karthik
>
> CONFIDENTIALITY NOTICE TO RECIPIENT: This transmission contains confidential
> information belonging to the sender that is legally privileged and
> proprietary and may be subject to protection under the law, including the
> Health Insurance Portability and Accountability Act (HIPAA). If you are not
> the intended recipient of this e-mail, you are prohibited from sharing,
> copying, or otherwise using or disclosing its contents. If you have received
> this e-mail in error, please notify the sender immediately by reply e-mail
> and permanently delete this e-mail and any attachments without reading,
> forwarding or saving them. Thank you.
CONFIDENTIALITY NOTICE TO RECIPIENT: This transmission contains confidential
information belonging to the sender that is legally privileged and proprietary
and may be subject to protection under the law, including the Health Insurance
Portability and Accountability Act (HIPAA). If you are not the intended
recipient of this e-mail, you are prohibited from sharing, copying, or
otherwise using or disclosing its contents. If you have received this e-mail in
error, please notify the sender immediately by reply e-mail and permanently
delete this e-mail and any attachments without reading, forwarding or saving
them. Thank you.
CONFIDENTIALITY NOTICE TO RECIPIENT: This transmission contains confidential
information belonging to the sender that is legally privileged and proprietary
and may be subject to protection under the law, including the Health Insurance
Portability and Accountability Act (HIPAA). If you are not the intended
recipient of this e-mail, you are prohibited from sharing, copying, or
otherwise using or disclosing its contents. If you have received this e-mail in
error, please notify the sender immediately by reply e-mail and permanently
delete this e-mail and any attachments without reading, forwarding or saving
them. Thank you.
