I think you guys are right, It seems I was able to isolate the issue and it appears to be related to the crm which we use to start, stop and monitor processes. Removing the activemq and stratos from the crm config and instead invoke it simply through a shell script as part of the octl startup process fixed (at least temporarily as a workaround) the issue.
Actually, shutdown during tenant creation was reported by Jeffrey, I typically don’t see this in my environment Thanks Martin From: Udara Liyanage [mailto:[email protected]] Sent: Friday, October 31, 2014 9:38 PM To: dev Subject: Re: [grouping] [question] stratos error during startup Hi Martin, You have reported shutdown thing when tenant creation also. It seems another process kill/send SIGINT to Stratos. May be it tries to kill when Stratos produces error messages. Touched, not typed. Erroneous words are a feature, not a typo. On Oct 31, 2014 11:42 PM, "Nirmal Fernando" <[email protected]<mailto:[email protected]>> wrote: Hi Martin, Is this a puppet based deployment? If so please check whether it's puppet causing this. On Fri, Oct 31, 2014 at 6:31 PM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Yes, I tried telnet (telnet octl.qmog.cisco.com<http://octl.qmog.cisco.com> 61616) after the exception happened and it is accessible. According to the stratos log it will retry to connect in 30 seconds but then the shutdown hook is triggered. My question why / what will trigger the shutdown (see logs below) ? TID: [0] [STRATOS] [2014-10-31 17:22:27,196] ERROR {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error while subscribing to the topic: application-status {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} javax.jms.JMSException: Could not connect to broker URL: tcp://octl.qmog.cisco.com:61616<http://octl.qmog.cisco.com:61616>. Reason: java.net.ConnectException: Connection refused at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36) at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358) at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303) at org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277) at org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68) at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64) at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501) at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464) at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) at org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338) ... 6 moreTID: [0] [STRATOS] [2014-10-31 17:22:27,199] INFO {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try to subscribe again in 30 sec {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}TID: [0] [STRATOS] [2014-10-31 17:22:27,520] INFO {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook triggered.... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0] [STRATOS] [2014-10-31 17:22:27,524] INFO {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0] [STRATOS] [2014-10-31 17:22:27,531] INFO {org.wso2.carbon.core.ServerManagement} - Starting to switch to maintenance mode... {org.wso2.carbon.core.ServerManagement}TID: [0] [STRATOS] [2014-10-31 17:22:27,532] INFO {org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners {org.wso2.carbon.core.ServerManagement}TID: [0] [STRATOS] [2014-10-31 17:22:27,532] INFO {org.wso2.carbon.core.ServerManagement} - Waiting for request service completion... {org.wso2.carbon.core.ServerManagement} From: Martin Eppel (meppel) Sent: Friday, October 31, 2014 10:02 AM To: [email protected]<mailto:[email protected]> Subject: RE: [grouping] [question] stratos error during startup Ok, I try it. One questions, does stratos limits the numbers of retries or does it eventually shut down (this is what it seems to happen that stratos shuts down after it fails to connect to the activemq) Thanks Martin From: Imesh Gunaratne [mailto:[email protected]] Sent: Friday, October 31, 2014 9:57 AM To: dev Subject: Re: [grouping] [question] stratos error during startup Hi Martin, According to the error log it seems like Stratos could not reach the message broker at the given tcp URL. However it should try to reconnect afterwards. Will us be able to check whether it is possible to telnet to this tcp socket fromt he Stratos host? Thanks On Fri, Oct 31, 2014 at 7:49 PM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Yes it is running when stratos starts up From: Udara Liyanage [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, October 30, 2014 8:09 PM To: dev Subject: Re: [grouping] [question] stratos error during startup Hi Martin, Is ActiveMQ available at the time of Stratos startup time? Touched, not typed. Erroneous words are a feature, not a typo. On Oct 31, 2014 2:10 AM, "Martin Eppel (meppel)" <[email protected]<mailto:[email protected]>> wrote: I am trying to migrate the grouping stratos binaries to our orchestration environment but consistently run into the issue that when stratos starts up the first time an (activemq) eror is thrown when trying to subscribe to application-status (Error while subscribing to the topic: application-status). ActiveMq seems to be up and running and after restarting stratos, stratos comes up without the exception (without having activemq to restart). Any hints / ideas what could cause this or what I should look for ? Thanks Martin (I attached the complete log) TID: [0] [STRATOS] [2014-10-30 19:34:52,071] INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} - Mgt Console URL : https://172.16.2.207:9443/carbon/ {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} TID: [0] [STRATOS] [2014-10-30 19:34:58,071] INFO {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - CEP topology receiver thread started {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} TID: [0] [STRATOS] [2014-10-30 19:34:58,072] INFO {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator} - Topology event message delegator started {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator} TID: [0] [STRATOS] [2014-10-30 19:35:01,024] DEBUG {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - Executing topology synchronization task {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} TID: [0] [STRATOS] [2014-10-30 19:35:01,025] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyManager} - Trying to retrieve topology from registry {org.apache.stratos.cloud.controller.topology.TopologyManager} TID: [0] [STRATOS] [2014-10-30 19:35:01,027] DEBUG {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing complete tenant event {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} TID: [0] [STRATOS] [2014-10-30 19:35:01,032] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology not found in registry, creating new {org.apache.stratos.cloud.controller.topology.TopologyManager} TID: [0] [STRATOS] [2014-10-30 19:35:01,033] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology initialized {org.apache.stratos.cloud.controller.topology.TopologyManager} TID: [0] [STRATOS] [2014-10-30 19:35:01,035] DEBUG {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing complete topology event {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} TID: [0] [STRATOS] [2014-10-30 19:35:01,155] INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} - Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} - Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} - Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - [CompleteTopologyEvent] Received: class org.apache.stratos.messaging.event.topology.CompleteTopologyEvent {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} - Topology initialized {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} TID: [0] [STRATOS] [2014-10-30 19:35:01,159] INFO {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} - [CompleteTopologyEventListener] Received: class org.apache.stratos.messaging.event.topology.CompleteTopologyEvent {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Complete topology event received to fault handling window processor. {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Member time stamp map was successfully loaded from the topology. {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} TID: [0] [STRATOS] [2014-10-30 19:35:04,168] DEBUG {org.apache.stratos.manager.publisher.TenantEventPublisher} - Publishing tenant created event: [tenant-id] 1 [tenant-domain] cloud1.qmog.cisco.com<http://cloud1.qmog.cisco.com> {org.apache.stratos.manager.publisher.TenantEventPublisher} TID: [0] [STRATOS] [2014-10-30 19:35:06,045] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - instance-status topic health checker is failed and will try to subscribe again in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} TID: [0] [STRATOS] [2014-10-30 19:35:06,104] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - instance-status topic health checker is failed and will try to subscribe again in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} TID: [0] [STRATOS] [2014-10-30 19:35:06,106] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology topic health checker is failed and will try to subscribe again in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} TID: [0] [STRATOS] [2014-10-30 19:35:06,225] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology topic health checker is failed and will try to subscribe again in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} TID: [0] [STRATOS] [2014-10-30 19:35:06,229] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology topic health checker is failed and will try to subscribe again in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} TID: [0] [STRATOS] [2014-10-30 19:35:06,319] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - summarized-health-stats topic health checker is failed and will try to subscribe again in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} TID: [0] [STRATOS] [2014-10-30 19:35:06,397] ERROR {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology topic health checker is failed and will try to subscribe again in 30 seconds. {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} TID: [0] [STRATOS] [2014-10-30 19:35:06,656] INFO {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver} - Cloud controller application status thread started {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver} TID: [0] [STRATOS] [2014-10-30 19:35:06,658] INFO {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator} - Application status event message delegator started {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator} TID: [0] [STRATOS] [2014-10-30 19:35:06,663] ERROR {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error while subscribing to the topic: application-status {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} javax.jms.JMSException: Could not connect to broker URL: tcp://octl.qmog.cisco.com:61616<http://octl.qmog.cisco.com:61616>. Reason: java.net.ConnectException: Connection refused at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36) at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358) at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303) at org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277) at org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68) at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64) at org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501) at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464) at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) at org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338) ... 6 more TID: [0] [STRATOS] [2014-10-30 19:35:06,666] INFO {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try to subscribe again in 30 sec {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} TID: [0] [STRATOS] [2014-10-30 19:35:07,146] INFO {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook triggered.... {org.wso2.carbon.core.init.CarbonServerManager} TID: [0] [STRATOS] [2014-10-30 19:35:07,152] INFO {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager} TID: [0] [STRATOS] [2014-10-30 19:35:07,156] INFO {org.wso2.carbon.core.ServerManagement} - Starting to switch to maintenance mode... {org.wso2.carbon.core.ServerManagement} TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO {org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners {org.wso2.carbon.core.ServerManagement} TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO {org.wso2.carbon.core.ServerManagement} - Waiting for request service completion... {org.wso2.carbon.core.ServerManagement} TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO {org.wso2.carbon.core.ServerManagement} - All requests have been served. {org.wso2.carbon.core.ServerManagement} TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO {org.wso2.carbon.core.ServerManagement} - Waiting for deployment completion... {org.wso2.carbon.core.ServerManagement} TID: [0] [STRATOS] [2014-10-30 19:35:07,180] INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/STRATOS_ROOT] {org.wso2.carbon.webapp.mgt.WebApplication} TID: [0] [STRATOS] [2014-10-30 19:35:07,181] INFO {org.springframework.web.context.support.XmlWebApplicationContext} -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos -- Best Regards, Nirmal Nirmal Fernando. PPMC Member & Committer of Apache Stratos, Senior Software Engineer, WSO2 Inc. Blog: http://nirmalfdo.blogspot.com/
