Hi, I will have a look.
On Mon, Mar 23, 2015 at 3:38 AM, Vanson Lim <[email protected]> wrote: > Devs, > > We are continuing to work on testing the latest stratos 4.1.0 codebase. > > This problem is seen only for VM that have floating ip. I've tested > with the non floating ip case and don't see issues. > > The error return code from jcloud api call is preventing stratos from > cleaning up its state. > > Stratos seems to forever throw tracebacks as it repeatedly tries to > terminate the faulty instance. > > Meanwhile, the "down" VM is still being reported as active in the topology > events, which seems wrong. If stratos detects that the VM is faulty, > shouldn't it report it immediately in the topology events? Stratos > currently has the following states define and none of them seem to be > appropriate. > > Created > Initialized > Starting > Active > In_Maintenance > ReadyToShutdown > Suspended > Terminated > > > Do we need new state TIMED-OUT state that stratos reports for VM as > stratos works to terminate it? > > How to reproduce this issue: > > 1) Start a sample cartridge instance that has a floating ip. > > 2) wait for sample cartridge to become active > 3) terminate sample vm via openstack horizon interface, and wait for > stratos to detect VM the error. > > > Testing using a version of stratos built off the following commit id: > > commit 01dd9e491ad3acf7cc4e0f2895aaba336b82539d > Author: R-Rajkumar <[email protected]> <[email protected]> > Date: Fri Mar 20 19:51:06 2015 +0530 > > fixing an NPE in AS > > > I've attached the full wso2carbon.log Included below is the observed > traceback: > > -Vanson > > > TID: [0] [STRATOS] [2015-03-22 20:53:21,554] INFO > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - > Publishing member fault event for [member-id] > cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd > TID: [0] [STRATOS] [2015-03-22 20:54:06,386] INFO > {org.apache.stratos.common.client.CloudControllerServiceClient} - > Terminating instance via cloud controller: [member] > cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd > TID: [0] [STRATOS] [2015-03-22 20:54:06,399] INFO > {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to > terminate member: [cartridge-type] cisco-sample-vm [member-id] > cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd > TID: [0] [STRATOS] [2015-03-22 20:54:06,450] ERROR > {org.apache.stratos.cloud.controller.services.impl.InstanceTerminator} - > Instance termination failed! MemberContext [applicationId=cisco-sample-vm, > cartridgeType=cisco-sample-vm, > clusterId=cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain, > memberId=cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd, > instanceId=RegionOne/83751110-4e5b-4aef-b6a3-c291c9eaad3d, > partition=Partition [id=whole-region, description=null, isPublic=false, > provider=Core, properties=Properties [properties=[Property [name=region, > value=RegionOne]]]], defaultPrivateIP=172.16.2.17, > defaultPublicIP=10.0.0.102, allocatedIPs=[10.0.0.102], > publicIPs=[10.0.0.102], privateIPs=[172.16.2.17], initTime=1427057106433, > lbClusterId=null, networkPartitionId=RegionOne, kubernetesPodId=null, > kubernetesPodLabel=null, loadBalancingIPType=Private, > instanceMetadata=org.apache.stratos.cloud.controller.domain.InstanceMetadata@5b176e44, > properties=Properties [properties=[Property [name=PRIMARY, value=false], > Property [name=MIN_COUNT, value=1]]]] > java.lang.NullPointerException: arg[0] in > {invocation=org.jclouds.openstack.nova.v2_0.NovaApi.public abstract > com.google.common.base.Optional > org.jclouds.openstack.nova.v2_0.NovaApi.getFloatingIPExtensionForZone(java.lang.String)[null], > result={annotationParser={caller=NovaApi.getFloatingIPExtensionForZone[null]}}} > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:253) > at > org.jclouds.openstack.v2_0.functions.PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.apply(PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.java:67) > at > org.jclouds.openstack.v2_0.functions.PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.apply(PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.java:43) > at > org.jclouds.rest.internal.DelegatesToInvocationFunction.propagateContextToDelegate(DelegatesToInvocationFunction.java:205) > at > org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:154) > at > org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) > at com.sun.proxy.$Proxy119.getFloatingIPExtensionForZone(Unknown > Source) > at > org.apache.stratos.cloud.controller.iaases.openstack.networking.NovaNetworkingApi.releaseAddress(NovaNetworkingApi.java:239) > at > org.apache.stratos.cloud.controller.iaases.openstack.OpenstackIaas.releaseAddress(OpenstackIaas.java:239) > at > org.apache.stratos.cloud.controller.iaases.JcloudsIaas.destroyNode(JcloudsIaas.java:334) > at > org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:314) > at > org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > TID: [0] [STRATOS] [2015-03-22 20:54:21,563] INFO > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty > member detected [member-id] > cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd > with [last time-stamp] 1427057336960 [time-out] 60000 milliseconds > TID: [0] [STRATOS] [2015-03-22 20:54:21,563] INFO > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - > Publishing member fault event for [member-id] > cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd > > -- Udara Liyanage Software Engineer WSO2, Inc.: http://wso2.com lean. enterprise. middleware web: http://udaraliyanage.wordpress.com phone: +94 71 443 6897
