Hi Martin,
Is this a puppet based deployment? If so please check whether it's puppet
causing this.
On Fri, Oct 31, 2014 at 6:31 PM, Martin Eppel (meppel) <mep...@cisco.com>
wrote:
> Yes, I tried telnet (telnet octl.qmog.cisco.com 61616) after the
> exception happened and it is accessible.
>
>
>
> According to the stratos log it will retry to connect in 30 seconds but
> then the shutdown hook is triggered. My question why / what will trigger
> the shutdown (see logs below) ?
>
>
>
> TID: [0] [STRATOS] [2014-10-31 17:22:27,196] ERROR
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error
> while subscribing to the topic: application-status
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
>
> javax.jms.JMSException: Could not connect to broker URL: tcp://
> octl.qmog.cisco.com:61616. Reason: java.net.ConnectException: Connection
> refused
>
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358)
>
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303)
>
> at
> org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277)
>
> at
> org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68)
>
> at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64)
>
> at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117)
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>
> at
> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
> java.net.Socket.connect(Socket.java:579)
>
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
>
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
>
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
>
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
>
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
>
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338)
>
> ... 6 moreTID: [0] [STRATOS] [2014-10-31 17:22:27,199] INFO
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try
> to subscribe again in 30 sec
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}TID: [0]
> [STRATOS] [2014-10-31 17:22:27,520] INFO
> {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook
> triggered.... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0]
> [STRATOS] [2014-10-31 17:22:27,524] INFO
> {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down
> Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0]
> [STRATOS] [2014-10-31 17:22:27,531] INFO
> {org.wso2.carbon.core.ServerManagement} - Starting to switch to
> maintenance mode... {org.wso2.carbon.core.ServerManagement}TID: [0]
> [STRATOS] [2014-10-31 17:22:27,532] INFO
> {org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners
> {org.wso2.carbon.core.ServerManagement}TID: [0] [STRATOS] [2014-10-31
> 17:22:27,532] INFO {org.wso2.carbon.core.ServerManagement} - Waiting for
> request service completion... {org.wso2.carbon.core.ServerManagement}
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, October 31, 2014 10:02 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: [grouping] [question] stratos error during startup
>
>
>
> Ok, I try it.
>
>
>
> One questions, does stratos limits the numbers of retries or does it
> eventually shut down (this is what it seems to happen that stratos shuts
> down after it fails to connect to the activemq)
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Imesh Gunaratne [mailto:im...@apache.org <im...@apache.org>]
> *Sent:* Friday, October 31, 2014 9:57 AM
> *To:* dev
> *Subject:* Re: [grouping] [question] stratos error during startup
>
>
>
> Hi Martin,
>
>
>
> According to the error log it seems like Stratos could not reach the
> message broker at the given tcp URL. However it should try to reconnect
> afterwards.
>
>
>
> Will us be able to check whether it is possible to telnet to this tcp
> socket fromt he Stratos host?
>
>
>
> Thanks
>
>
>
> On Fri, Oct 31, 2014 at 7:49 PM, Martin Eppel (meppel) <mep...@cisco.com>
> wrote:
>
> Yes it is running when stratos starts up
>
>
>
> *From:* Udara Liyanage [mailto:ud...@wso2.com]
> *Sent:* Thursday, October 30, 2014 8:09 PM
> *To:* dev
> *Subject:* Re: [grouping] [question] stratos error during startup
>
>
>
> Hi Martin,
>
> Is ActiveMQ available at the time of Stratos startup time?
>
> Touched, not typed. Erroneous words are a feature, not a typo.
>
> On Oct 31, 2014 2:10 AM, "Martin Eppel (meppel)" <mep...@cisco.com> wrote:
>
> I am trying to migrate the grouping stratos binaries to our orchestration
> environment but consistently run into the issue that when stratos starts up
> the first time an (activemq) eror is thrown when trying to subscribe to
> application-status (Error while subscribing to the topic:
> application-status). ActiveMq seems to be up and running and after
> restarting stratos, stratos comes up without the exception (without having
> activemq to restart).
>
>
>
> Any hints / ideas what could cause this or what I should look for ?
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> (I attached the complete log)
>
>
>
>
>
> TID: [0] [STRATOS] [2014-10-30 19:34:52,071] INFO
> {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} - Mgt Console URL
> : https://172.16.2.207:9443/carbon/
> {org.wso2.carbon.ui.internal.CarbonUIServiceComponent}
>
> TID: [0] [STRATOS] [2014-10-30 19:34:58,071] INFO
> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - CEP topology
> receiver thread started
> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
>
> TID: [0] [STRATOS] [2014-10-30 19:34:58,072] INFO
> {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
> - Topology event message delegator started
> {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,024] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
> Executing topology synchronization task
> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,025] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Trying to
> retrieve topology from registry
> {org.apache.stratos.cloud.controller.topology.TopologyManager}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,027] DEBUG
> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
> complete tenant event
> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,032] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
> not found in registry, creating new
> {org.apache.stratos.cloud.controller.topology.TopologyManager}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,033] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
> initialized {org.apache.stratos.cloud.controller.topology.TopologyManager}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,035] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
> Publishing complete topology event
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,155] INFO
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
> - Topology initialized
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
> - Topology initialized
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
> - Topology initialized
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO
> {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
> - [CompleteTopologyEvent] Received: class
> org.apache.stratos.messaging.event.topology.CompleteTopologyEvent
> {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
> - Topology initialized
> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,159] INFO
> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
> - [CompleteTopologyEventListener] Received: class
> org.apache.stratos.messaging.event.topology.CompleteTopologyEvent
> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO
> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Complete
> topology event received to fault handling window processor.
> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO
> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Member
> time stamp map was successfully loaded from the topology.
> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:04,168] DEBUG
> {org.apache.stratos.manager.publisher.TenantEventPublisher} - Publishing
> tenant created event: [tenant-id] 1 [tenant-domain] cloud1.qmog.cisco.com
> {org.apache.stratos.manager.publisher.TenantEventPublisher}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,045] ERROR
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
> instance-status topic health checker is failed and will try to subscribe
> again in 30 seconds.
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,104] ERROR
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
> instance-status topic health checker is failed and will try to subscribe
> again in 30 seconds.
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,106] ERROR
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
> topology topic health checker is failed and will try to subscribe again in
> 30 seconds.
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,225] ERROR
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
> topology topic health checker is failed and will try to subscribe again in
> 30 seconds.
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,229] ERROR
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
> topology topic health checker is failed and will try to subscribe again in
> 30 seconds.
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,319] ERROR
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
> summarized-health-stats topic health checker is failed and will try to
> subscribe again in 30 seconds.
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,397] ERROR
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} -
> topology topic health checker is failed and will try to subscribe again in
> 30 seconds.
> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,656] INFO
> {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver}
> - Cloud controller application status thread started
> {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,658] INFO
> {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator}
> - Application status event message delegator started
> {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,663] ERROR
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error
> while subscribing to the topic: application-status
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
>
> javax.jms.JMSException: Could not connect to broker URL: tcp://
> octl.qmog.cisco.com:61616. Reason: java.net.ConnectException: Connection
> refused
>
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)
>
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358)
>
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303)
>
> at
> org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277)
>
> at
> org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68)
>
> at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64)
>
> at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:579)
>
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
>
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
>
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
>
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
>
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
>
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
>
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
>
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
>
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338)
>
> ... 6 more
>
> TID: [0] [STRATOS] [2014-10-30 19:35:06,666] INFO
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try
> to subscribe again in 30 sec
> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,146] INFO
> {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook
> triggered.... {org.wso2.carbon.core.init.CarbonServerManager}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,152] INFO
> {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down
> Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,156] INFO
> {org.wso2.carbon.core.ServerManagement} - Starting to switch to
> maintenance mode... {org.wso2.carbon.core.ServerManagement}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO
> {org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners
> {org.wso2.carbon.core.ServerManagement}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO
> {org.wso2.carbon.core.ServerManagement} - Waiting for request service
> completion... {org.wso2.carbon.core.ServerManagement}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO
> {org.wso2.carbon.core.ServerManagement} - All requests have been served.
> {org.wso2.carbon.core.ServerManagement}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO
> {org.wso2.carbon.core.ServerManagement} - Waiting for deployment
> completion... {org.wso2.carbon.core.ServerManagement}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,180] INFO
> {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp:
> StandardEngine[Catalina].StandardHost[localhost].StandardContext[/STRATOS_ROOT]
> {org.wso2.carbon.webapp.mgt.WebApplication}
>
> TID: [0] [STRATOS] [2014-10-30 19:35:07,181] INFO
> {org.springframework.web.context.support.XmlWebApplicationContext}
>
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
