Thanks
Imesh
On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage
<ud...@wso2.com> wrote:
Hi,
This arises when the application is deleted
immediately after the application underemployment. I guess CC
does not have the member context when the
application is removed. I will have a look.
A simple workaround it to wait sometime to remove
application after undeployment.
On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne
<im...@apache.org> wrote:
Thanks Martin! We will have a look at this.
On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel
(meppel) <mep...@cisco.com> wrote:
I opened a jira
https://issues.apache.org/jira/browse/STRATOS-1281 to track the issue,
Thanks
Martin
*From:*Imesh Gunaratne
[mailto:im...@apache.org]
*Sent:* Friday, March 20, 2015 3:27 AM
*To:* dev
*Subject:* Re: Stratos 4.1.0 - tracebacks
seen when issuing application undeploy/remove
Hi Vanson,
Thanks for reporting this problem.
According to the logs stratos is trying to remove the same
member twice in this flow:
TID: [0] [STRATOS] [2015-03-19
19:45:43,952] INFO
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
-
[Application] cisco-sample-vm has been
successfully removed
TID: [0] [STRATOS] [2015-03-19
19:45:44,786] INFO
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member terminated:
[member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:50,012] INFO
{org.apache.stratos.common.client.CloudControllerServiceClient} - Terminating
instance via cloud
controller: [member]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:50,017] ERROR
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
- Could not
terminate instance, member context not
found: [member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
We might need to investigate and see what's
causing this.
Thanks
On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim
<v...@cisco.com> wrote:
Hi,
We are testing the behavior of stratos
4.1.0 rest api's and found that if we issue an
undeploy-application followed immediately
with application-remove, that we see the following
traceback.
The VM successfully get's deleted but not
sure what kind of side effects this has on the system.
Here's the snippet from the wso2carbon.log
file. I've also attached the entire log.
Steps to reproduce this:
1) deploy an application which leads to
startup a single instance of a cartridge.
2) wait for it to become active
3) issue the undeploy application and
application remove rest api calls.
-Vanson
TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO {org.wso2.carbon.databridge.core.DataBridge} -
admin connected
TID: [0] [STRATOS] [2015-03-19
19:45:43,023] INFO
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Starting
to undeploy
application: [application-id]
cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,024] INFO
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Removing
application
signup: [application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,051] INFO
{org.apache.stratos.manager.components.ApplicationSignUpHandler} - Removing
application signup:
[application-id] cisco-sample-vm
[tenant-id] -1234
TID: [0] [STRATOS] [2015-03-19
19:45:43,084] INFO
{org.apache.stratos.manager.components.ApplicationSignUpHandler} - Application
signup removed
successfully: [application-id]
cisco-sample-vm [tenant-id] -1234
TID: [0] [STRATOS] [2015-03-19
19:45:43,091] INFO
{org.apache.stratos.autoscaler.context.AutoscalerContext} - Network partition
algorithm context
is removed successfully: [id]
cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,092] INFO
{org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Publishing
Cluster terminating
event for [application] cisco-sample-vm
[cluster]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19
19:45:43,109] INFO
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
Cluster Terminating
adding status started
forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
TID: [0] [STRATOS] [2015-03-19
19:45:43,117] INFO
{org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher} -
Publishing
application inactivated event:
[application] cisco-sample-vm [instance] cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19
19:45:43,120] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing
Cluster terminating event: [application-id]
cisco-sample-vm [cluster id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance-id]
cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19
19:45:43,144] INFO
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
Application undeployed
successfully: [application-id]
cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,154] INFO
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
-
[ClusterTerminatingEvent] Received: class
org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent
TID: [0] [STRATOS] [2015-03-19
19:45:43,155] INFO
{org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher} -
Publishing
Instance Cleanup Event: [cluster]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
TID: [0] [STRATOS] [2015-03-19
19:45:43,171] WARN
{org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor}
- No
possible state change found for [type]
[cluster]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
TID: [0] [STRATOS] [2015-03-19
19:45:43,192] INFO
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
member maintenance
mode event adding status started
TID: [0] [STRATOS] [2015-03-19
19:45:43,204] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing
member in maintenance mode event:
[service-name] cisco-sample-vm [cluster-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [cluster-instance-id]
cisco-sample-vm-1
[member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
[network-partition-id] RegionOne
[partition-id] whole-region
TID: [0] [STRATOS] [2015-03-19
19:45:43,230] INFO
{org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor}
- Member
updated as In_Maintenance: [service]
cisco-sample-vm [cluster]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:43,231] INFO
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
Member Ready to shut
down event adding status started
TID: [0] [STRATOS] [2015-03-19
19:45:43,242] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing
member ready to shut down event:
[service-name] cisco-sample-vm [cluster-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [cluster-instance-id]
cisco-sample-vm-1
[member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
[network-partition-id] RegionOne
[partition-id] whole-region
TID: [0] [STRATOS] [2015-03-19
19:45:43,262] INFO
{org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} -
Member updated as Ready to shutdown: [service] cisco-sample-vm [cluster]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:43,264] INFO
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
-
[MemberReadyToShutdownEvent] Received: class
org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent
TID: [0] [STRATOS] [2015-03-19
19:45:43,264] INFO
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
- Starting the
moving of termination pending to obsolete
for [member]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:43,264] INFO
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
- Found
termination pending member and trying to
move [member]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
to
obsolete list
TID: [0] [STRATOS] [2015-03-19
19:45:43,266] INFO
{org.apache.stratos.common.client.CloudControllerServiceClient} - Terminating
instance via cloud
controller: [member]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:43,275] INFO
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to
terminate member:
[cartridge-type] cisco-sample-vm [member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:43,883] INFO
{org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy}
- Removed lock
for application: [application-id]
cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,909] INFO
{org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver} -
Application deleted event received: [application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,909] INFO
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
Removed application [
cisco-sample-vm ]'s Cluster [
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the
topology
TID: [0] [STRATOS] [2015-03-19
19:45:43,910] INFO
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
Application deleted
successfully: [application-id]
TID: [0] [STRATOS] [2015-03-19
19:45:43,930] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing
application clusters removed event:
[application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,950] INFO
{org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor}
- Cluster
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed from topology for
application cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19
19:45:43,952] INFO
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
-
[Application] cisco-sample-vm has been
successfully removed
TID: [0] [STRATOS] [2015-03-19
19:45:44,786] INFO
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member terminated:
[member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:44,786] WARN
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
Cluster
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not exist
TID: [0] [STRATOS] [2015-03-19
19:45:50,012] INFO
{org.apache.stratos.common.client.CloudControllerServiceClient} - Terminating
instance via cloud
controller: [member]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:50,017] ERROR
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
- Could not
terminate instance, member context not
found: [member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19
19:45:50,018] ERROR
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
- Could not
terminate instance: [member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not
terminate
instance, member context not found:
[member-id]
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
at
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
at
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
at
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
