Yes, I tried telnet (telnet octl.qmog.cisco.com 61616) after the exception
happened and it is accessible.
According to the stratos log it will retry to connect in 30 seconds but then
the shutdown hook is triggered. My question why / what will trigger the
shutdown (see logs below) ?
TID: [0] [STRATOS] [2014-10-31 17:22:27,196] ERROR
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error while
subscribing to the topic: application-status
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
javax.jms.JMSException: Could not connect to broker URL:
tcp://octl.qmog.cisco.com:61616. Reason: java.net.ConnectException: Connection
refused
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303)
at
org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277)
at
org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68)
at
org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64)
at
org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:579)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338)
... 6 moreTID: [0] [STRATOS] [2014-10-31 17:22:27,199] INFO
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try to
subscribe again in 30 sec
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}TID: [0]
[STRATOS] [2014-10-31 17:22:27,520] INFO
{org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook triggered....
{org.wso2.carbon.core.init.CarbonServerManager}TID: [0] [STRATOS] [2014-10-31
17:22:27,524] INFO {org.wso2.carbon.core.init.CarbonServerManager} -
Gracefully shutting down Apache Stratos...
{org.wso2.carbon.core.init.CarbonServerManager}TID: [0] [STRATOS] [2014-10-31
17:22:27,531] INFO {org.wso2.carbon.core.ServerManagement} - Starting to
switch to maintenance mode... {org.wso2.carbon.core.ServerManagement}TID: [0]
[STRATOS] [2014-10-31 17:22:27,532] INFO
{org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners
{org.wso2.carbon.core.ServerManagement}TID: [0] [STRATOS] [2014-10-31
17:22:27,532] INFO {org.wso2.carbon.core.ServerManagement} - Waiting for
request service completion... {org.wso2.carbon.core.ServerManagement}
From: Martin Eppel (meppel)
Sent: Friday, October 31, 2014 10:02 AM
To: [email protected]
Subject: RE: [grouping] [question] stratos error during startup
Ok, I try it.
One questions, does stratos limits the numbers of retries or does it eventually
shut down (this is what it seems to happen that stratos shuts down after it
fails to connect to the activemq)
Thanks
Martin
From: Imesh Gunaratne [mailto:[email protected]]
Sent: Friday, October 31, 2014 9:57 AM
To: dev
Subject: Re: [grouping] [question] stratos error during startup
Hi Martin,
According to the error log it seems like Stratos could not reach the message
broker at the given tcp URL. However it should try to reconnect afterwards.
Will us be able to check whether it is possible to telnet to this tcp socket
fromt he Stratos host?
Thanks
On Fri, Oct 31, 2014 at 7:49 PM, Martin Eppel (meppel)
<[email protected]<mailto:[email protected]>> wrote:
Yes it is running when stratos starts up
From: Udara Liyanage [mailto:[email protected]<mailto:[email protected]>]
Sent: Thursday, October 30, 2014 8:09 PM
To: dev
Subject: Re: [grouping] [question] stratos error during startup
Hi Martin,
Is ActiveMQ available at the time of Stratos startup time?
Touched, not typed. Erroneous words are a feature, not a typo.
On Oct 31, 2014 2:10 AM, "Martin Eppel (meppel)"
<[email protected]<mailto:[email protected]>> wrote:
I am trying to migrate the grouping stratos binaries to our orchestration
environment but consistently run into the issue that when stratos starts up the
first time an (activemq) eror is thrown when trying to subscribe to
application-status (Error while subscribing to the topic: application-status).
ActiveMq seems to be up and running and after restarting stratos, stratos
comes up without the exception (without having activemq to restart).
Any hints / ideas what could cause this or what I should look for ?
Thanks
Martin
(I attached the complete log)
TID: [0] [STRATOS] [2014-10-30 19:34:52,071] INFO
{org.wso2.carbon.ui.internal.CarbonUIServiceComponent} - Mgt Console URL :
https://172.16.2.207:9443/carbon/
{org.wso2.carbon.ui.internal.CarbonUIServiceComponent}
TID: [0] [STRATOS] [2014-10-30 19:34:58,071] INFO
{org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - CEP topology
receiver thread started
{org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:34:58,072] INFO
{org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
- Topology event message delegator started
{org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
TID: [0] [STRATOS] [2014-10-30 19:35:01,024] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
Executing topology synchronization task
{org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask}
TID: [0] [STRATOS] [2014-10-30 19:35:01,025] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Trying to
retrieve topology from registry
{org.apache.stratos.cloud.controller.topology.TopologyManager}
TID: [0] [STRATOS] [2014-10-30 19:35:01,027] DEBUG
{org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
complete tenant event
{org.apache.stratos.manager.publisher.TenantSynzhronizerTask}
TID: [0] [STRATOS] [2014-10-30 19:35:01,032] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology not
found in registry, creating new
{org.apache.stratos.cloud.controller.topology.TopologyManager}
TID: [0] [STRATOS] [2014-10-30 19:35:01,033] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
initialized {org.apache.stratos.cloud.controller.topology.TopologyManager}
TID: [0] [STRATOS] [2014-10-30 19:35:01,035] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
Publishing complete topology event
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher}
TID: [0] [STRATOS] [2014-10-30 19:35:01,155] INFO
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
- Topology initialized
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
- Topology initialized
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
- Topology initialized
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO
{org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
- [CompleteTopologyEvent] Received: class
org.apache.stratos.messaging.event.topology.CompleteTopologyEvent
{org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
- Topology initialized
{org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:01,159] INFO
{org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
- [CompleteTopologyEventListener] Received: class
org.apache.stratos.messaging.event.topology.CompleteTopologyEvent
{org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO
{org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Complete
topology event received to fault handling window processor.
{org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Member time
stamp map was successfully loaded from the topology.
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-10-30 19:35:04,168] DEBUG
{org.apache.stratos.manager.publisher.TenantEventPublisher} - Publishing
tenant created event: [tenant-id] 1 [tenant-domain]
cloud1.qmog.cisco.com<http://cloud1.qmog.cisco.com>
{org.apache.stratos.manager.publisher.TenantEventPublisher}
TID: [0] [STRATOS] [2014-10-30 19:35:06,045] ERROR
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
instance-status topic health checker is failed and will try to subscribe again
in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,104] ERROR
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
instance-status topic health checker is failed and will try to subscribe again
in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,106] ERROR
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology
topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,225] ERROR
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology
topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,229] ERROR
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology
topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,319] ERROR
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
summarized-health-stats topic health checker is failed and will try to
subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,397] ERROR
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - topology
topic health checker is failed and will try to subscribe again in 30 seconds.
{org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
TID: [0] [STRATOS] [2014-10-30 19:35:06,656] INFO
{org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver}
- Cloud controller application status thread started
{org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver}
TID: [0] [STRATOS] [2014-10-30 19:35:06,658] INFO
{org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator}
- Application status event message delegator started
{org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator}
TID: [0] [STRATOS] [2014-10-30 19:35:06,663] ERROR
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error while
subscribing to the topic: application-status
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
javax.jms.JMSException: Could not connect to broker URL:
tcp://octl.qmog.cisco.com:61616<http://octl.qmog.cisco.com:61616>. Reason:
java.net.ConnectException: Connection refused
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303)
at
org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277)
at
org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68)
at
org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64)
at
org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338)
... 6 more
TID: [0] [STRATOS] [2014-10-30 19:35:06,666] INFO
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try to
subscribe again in 30 sec
{org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
TID: [0] [STRATOS] [2014-10-30 19:35:07,146] INFO
{org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook triggered....
{org.wso2.carbon.core.init.CarbonServerManager}
TID: [0] [STRATOS] [2014-10-30 19:35:07,152] INFO
{org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down
Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager}
TID: [0] [STRATOS] [2014-10-30 19:35:07,156] INFO
{org.wso2.carbon.core.ServerManagement} - Starting to switch to maintenance
mode... {org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO
{org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners
{org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO
{org.wso2.carbon.core.ServerManagement} - Waiting for request service
completion... {org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO
{org.wso2.carbon.core.ServerManagement} - All requests have been served.
{org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO
{org.wso2.carbon.core.ServerManagement} - Waiting for deployment completion...
{org.wso2.carbon.core.ServerManagement}
TID: [0] [STRATOS] [2014-10-30 19:35:07,180] INFO
{org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp:
StandardEngine[Catalina].StandardHost[localhost].StandardContext[/STRATOS_ROOT]
{org.wso2.carbon.webapp.mgt.WebApplication}
TID: [0] [STRATOS] [2014-10-30 19:35:07,181] INFO
{org.springframework.web.context.support.XmlWebApplicationContext}
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
