We recently made few changes to our AMQ 5.15. After that consumers start dropping in few hours, and not able to connect back. In case of restart of broker, all are gone.
Changes we did:- old config:- <kahaDB directory="/opt/apachemq/data/kahadb" /> New config:- <kahaDB directory="/opt/apachemq/data/kahadb" ignoreMissingJournalfiles="true" indexWriteBatchSize="10000" enableIndexWriteAsync="true" journalMaxFileLength="50mb" checksumJournalFiles="false" checkForCorruptJournalFiles="true"/> old config:- transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600&wireFormat.maxInactivityDuration=0"/> New config:- <transportConnector name="nio" uri="nio://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600&wireFormat.maxInactivityDuration=0"/> Old config:- -Dorg.apache.activemq.UseDedicatedTaskRunner=true New config:- -Dorg.apache.activemq.UseDedicatedTaskRunner=false Log snip:- 2018-02-16 07:51:59,376 | WARN | Failed to add Connection ID:ABC-52048-1516310038124-7:1 due to javax.jms.InvalidClientIDException: Broker: localhost - Client: 130 already connected from tcp://10.115.178.41:61040 | org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker 2 2018-02-16 07:51:59,376 | WARN | Failed to add Connection ID:PQR-52049-1516310038273-7:1 due to javax.jms.InvalidClientIDException: Broker: localhost - Client: 130 already connected from tcp://10.115.178.41:61040 | org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker 9 2018-02-16 07:51:59,441 | WARN | Transport Connection to: tcp://10.x.x.x.x:62651 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 6 2018-02-16 07:51:59,443 | WARN | Transport Connection to: tcp://10.x.y.x:62652 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 8 2018-02-16 07:51:59,494 | WARN | Failed to add Connection ID:xyz-55854-1516309827058-7:1 due to javax.jms.InvalidClientIDException: Broker: localhost - Client: 130 already connected from tcp://10.115.178.41:61040 | org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker 5 2018-02-16 07:51:59,570 | WARN | Transport Connection to: tcp://10.x.y.q:61041 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 9 2018-02-16 09:00:48,437 | WARN | Transport Connection to: tcp://192.g.y.z:43020 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 7 2018-02-16 10:13:08,884 | WARN | Transport Connection to: tcp://192.P.r.z:37522 failed: java.io.IOException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 9 2018-02-16 07:51:59,376 | WARN | Failed to add Connection ID:ghy-52048-1516310038124-7:1 due to javax.jms.InvalidClientIDException: Broker: localhost - Client: 130 already connected from tcp://10.115.178.41:61040 | org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker 2 2018-02-16 07:51:59,376 | WARN | Failed to add Connection ID:yjlk-52049-1516310038273-7:1 due to javax.jms.InvalidClientIDException: Broker: localhost - Client: 130 already connected from tcp://10.115.178.41:61040 | org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker 9 2018-02-16 07:51:59,441 | WARN | Transport Connection to: tcp://10.x.y.42:62651 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 6 2018-02-16 07:51:59,443 | WARN | Transport Connection to: tcp://10.115.178.42:62652 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 8 2018-02-16 07:51:59,494 | WARN | Failed to add Connection ID:GMerfIRG-55854-1516309827058-7:1 due to javax.jms.InvalidClientIDException: Broker: localhost - Client: 130 already connected from tcp://10.115.178.41:61040 | org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker 5 2018-02-16 07:51:59,570 | WARN | Transport Connection to: tcp://10.x.yz.z:61041 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 9 2018-02-16 09:00:48,437 | WARN | Transport Connection to: tcp://192.34.195.67:43020 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 7 2018-02-16 10:13:08,884 | WARN | Transport Connection to: tcp://192.98.175.61:37522 failed: java.io.IOException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 9 2018-02-16 11:42:31,906 | WARN | Transport Connection to: tcp://172.09.5.21:35064 failed: java.io.IOException: Connection reset by peer | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 9 2018-02-16 16:14:37,807 | WARN | Transport Connection to: tcp://172.65.5.233:50256 failed: java.io.IOException: Connection reset by peer | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 9 2018-02-16 18:08:17,249 | WARN | Transport Connection to: tcp://172.87.5.47:41618 failed: java.io.IOException: Connection timed out | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 6 2018-02-16 18:17:27,726 | WARN | Transport Connection to: tcp://10.8.178.41:61040 failed: java.io.IOException: Connection timed out | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 8 2018-02-16 18:17:27,726 | WARN | Transport Connection to: tcp://10.8.178.41:61040 failed: java.io.IOException: Connection timed out | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 8 2018-02-17 01:51:24,616 | WARN | Commit failed | org.eclipse.jetty.server.HttpChannel | Scheduler-1603696865 java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_71] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_71] at java.lang.Thread.run(Thread.java:745)[:1.8.0_71] 2018-02-17 10:54:35,534 | WARN | complete failed | org.eclipse.jetty.server.HttpChannel | qtp107994825-1004 java.nio.channels.WritePendingException at org.eclipse.jetty.server.HttpConnection$SendCallback.reset(HttpConnection.java:624)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpConnection$SendCallback.access$100(HttpConnection.java:594)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:479)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:768)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:171)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpWriter.close(HttpWriter.java:49)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at java.io.PrintWriter.close(PrintWriter.java:339)[:1.8.0_71] at org.eclipse.jetty.server.Response.closeOutput(Response.java:1002)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:421)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at java.lang.Thread.run(Thread.java:745)[:1.8.0_71] 2018-02-17 11:13:49,171 | WARN | Commit failed | org.eclipse.jetty.server.HttpChannel | Scheduler-12905860 java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_71] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Sc -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html