zhiwei wang created AMQ-7366:
--------------------------------
Summary: ActiveMQ broker always hangs up at intervals in the case
of high concurrent connections
Key: AMQ-7366
URL: https://issues.apache.org/jira/browse/AMQ-7366
Project: ActiveMQ
Issue Type: Bug
Components: Broker, Connector, MQTT
Affects Versions: 5.15.10
Environment: activeMQ version:5.15.0
linux OS version：Red Hat Enterprise Linux Server release 6.9 (Santiago)
linux server configuration: Memery:32G、CPU:16、disk:200G
Reporter: zhiwei wang
Fix For: 5.15.10
My broker JVM configuration is "{color:#FF0000}- xms16g - xmx16g - xss256k -
XX: metaspacesize = 128M{color}".
The transport connector in activemq.xml is configured as follows:
{color:#FF0000}<transportConnector name="sslmqtt"
uri="mqtt+ssl://0.0.0.0:1883?maximumConnections=200000&wireFormat.maxFrameSize=104857600&jms.watchTopicAdvisories=false&transport.defaultKeepAlive=30000"/>{color}
*At present, it can support 15000 client connections. But every two days or
less, ActiveMQ broker will hang up*. When I check the logs of ActiveMQ broker,
I find the following errors:
1.
{color:#FF0000}org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>60000){color} long:
tcp://180.162.0.129:50014org.apache.activemq.transport.InactivityIOException:
Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014 at
org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,123 |
DEBUG | No CONNECT frame received in time for tcp:///101.80.38.31:4751@1883!
Throwing InactivityIOException. |
org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor
ReadCheck2019-12-17 13:48:28,124 | DEBUG | Transport Connection to:
tcp://101.88.157.200:6892 failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>60000) long: tcp://101.88.157.200:6892 |
org.apache.activemq.broker.TransportConnection.Transport |
MQTTInactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@ddca0bb[State = -1, empty
queue]org.apache.activemq.transport.InactivityIOException: Channel was inactive
for too (>60000) long: tcp://101.88.157.200:6892 at
org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,136 |
DEBUG | No CONNECT frame received in time for tcp:///114.83.88.242:36665@1883!
Throwing InactivityIOException. |
org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor
ReadCheck
2.
{color:#FF0000}javax.net.ssl.SSLHandshakeException: Remote host closed
connection during handshake{color}
javax.net.ssl.SSLHandshakeException: Remote host closed connection during
handshake at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)[:1.8.0_151]
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)[:1.8.0_151]
at
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)[:1.8.0_151]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)[:1.8.0_151]
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)[activemq-client-5.15.0.jar:5.15.0]
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)[activemq-client-5.15.0.jar:5.15.0]
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)[activemq-client-5.15.0.jar:5.15.0]
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)[activemq-client-5.15.0.jar:5.15.0]
at java.io.DataInputStream.readByte(DataInputStream.java:265)[:1.8.0_151] at
org.apache.activemq.transport.mqtt.MQTTWireFormat.unmarshal(MQTTWireFormat.java:86)[activemq-mqtt-5.15.0.jar:5.15.0]
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)[activemq-client-5.15.0.jar:5.15.0]
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)[activemq-client-5.15.0.jar:5.15.0]
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.0.jar:5.15.0]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]
3.
{color:#FF0000} java.lang.OutOfMemoryError：unable to create new native
thread{color}
*At the same time, the number of threads in ActiveMQ service keeps
skyrocketing, reaching 60000, resulting in memory overflow。{color:#FF0000}I
found that when the channel was inactive for too long is kicked out by the
broker, the old thread is not destroyed. At this time, new requests come in,
and the thread will still be created continuously, resulting in memory
overflow{color}.*
*My requirement is that ActiveMQ broker can support 30000 client
connections.How can I solve this problem? Can someone help me*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
