[ https://issues.apache.org/jira/browse/STRATOS-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Imesh Gunaratne resolved STRATOS-1308. -------------------------------------- Resolution: Fixed AFAIK this issue should be now fixed with the reconnection logic implemented in messaging component (MQTT & AMQP). Thanks > Reconnection logic is missing the connection cleaning up part > ------------------------------------------------------------- > > Key: STRATOS-1308 > URL: https://issues.apache.org/jira/browse/STRATOS-1308 > Project: Stratos > Issue Type: Bug > Affects Versions: 4.1.0 Alpha > Reporter: Mariangela Hills > Assignee: Imesh Gunaratne > Priority: Critical > > I got a message saying "could not connect to the message broker" even when > the message broker connection was already in place (started). There seems to > be some reconnection logic, which finally leads to a too many files opened > error (part of the log can be seen in [1]). When this happens the web browser > on my machine (Chrome) gets stuck and I have to restart Stratos to solve the > issue. In addition, I would like to add that my max file limit was 8000 when > this issue occurred (ulimit -Sn 8000). Therefore, it looks like the > reconnection logic is missing the connection cleaning up part. > --------------------- > [1] > [2015-04-02 13:40:16,319] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 5 seconds: [topic-name] > tenant.CompleteTenantEvent > [2015-04-02 13:40:16,319] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Connection to the message broker failed > [2015-04-02 13:40:16,319] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 2 seconds: [topic-name] > tenant.CompleteTenantEvent > Exception in thread "ActiveMQ Transport: > tcp://localhost/127.0.0.1:61616@50628" java.lang.OutOfMemoryError: unable to > create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:673) > at > java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) > at > java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650) > at > org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002) > at > org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 > 13:40:16,423] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not connect to message broker > javax.jms.JMSException: Cannot send, channel has already failed: > tcp://127.0.0.1:61616 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) > at > org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) > at > org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, > channel has already failed: tcp://127.0.0.1:61616 > at > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) > at > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) > at > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > at > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > at > org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) > at org.apache.activemq.transport.ResponseCorrelator. > request(ResponseCorrelator.java:86) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) > ... 9 more > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > at > org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > at > org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160) > at > org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314) > at > org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96) > at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,424] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not reconnect to message broker > org.apache.stratos.messaging.domain.exception.MessagingException: Could not > connect to message broker > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > Caused by: javax.jms.JMSException: Cannot send, channel has already failed: > tcp://127.0.0.1:61616 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) > at > org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) > at > org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) > ... 6 more > Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, > channel has already failed: tcp://127.0.0.1:61616 > at > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) > at > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) > at > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > at > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > at > org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) > at > org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) > ... 9 more > [2015-04-02 13:40:16,424] ERROR > {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load > properties from file: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > java.io.FileNotFoundException: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > (Too many open files in system) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:120) > at > org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,425] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 20 seconds: [topic-name] > application.signup.CompleteApplicationSignUpsEvent > [2015-04-02 13:40:16,427] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not create topic connector > java.lang.NullPointerException > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,427] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not reconnect to message broker > org.apache.stratos.messaging.domain.exception.MessagingException: Could not > create topic connector > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > Caused by: java.lang.NullPointerException > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) > ... 6 more > [2015-04-02 13:40:16,428] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 20 seconds: [topic-name] > application.signup.CompleteApplicationSignUpsEvent > [2015-04-02 13:40:16,445] ERROR > {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load > properties from file: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > java.io.FileNotFoundException: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > (Too many open files in system) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:120) > at > org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,448] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not create topic connector -- This message was sent by Atlassian JIRA (v6.3.4#6332)