Hi Pubudu, In your case, the monitors are not started/scheduled after server restart because of the following error. There seems to be a communication issue between AS and CC, may be CC web service was not ready at that time.
TID: [0] [STRATOS] [2015-11-18 12:08:41,675] INFO {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - Starting dependent monitor: [application] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50 [component] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain TID: [0] [STRATOS] [2015-11-18 12:08:41,675] INFO {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - Monitor scheduled: [type] cluster [component] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain TID: [0] [STRATOS] [2015-11-18 12:08:41,676] INFO {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - Starting monitor: [type] cluster [component] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain TID: [0] [STRATOS] [2015-11-18 12:08:41,680] INFO {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Monitor started successfully: [application] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50 [dependents] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain [startup-time] 0 seconds TID: [0] [STRATOS] [2015-11-18 12:08:41,830] INFO {org.apache.axis2.transport.http.HTTPSender} - Unable to sendViaPost to url[https://localhost:9443/services/CloudControllerService/] org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found at org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329) at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196) at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430) at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070) at org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201) at org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270) at org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83) at org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) TID: [0] [STRATOS] [2015-11-18 12:08:41,831] ERROR {org.apache.stratos.autoscaler.monitor.MonitorFactory} - Error while validating deployment policy from cloud controller [network-partition-id] network-partition-openstack org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found at org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329) at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196) at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430) at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070) at org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201) at org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270) at org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83) at org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) TID: [0] [STRATOS] [2015-11-18 12:08:41,831] ERROR {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - An error occurred while starting monitor: [type] cluster [component] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain java.lang.RuntimeException: Error while validating deployment policy from cloud controller [network-partition-id] network-partition-openstack at org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:278) at org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83) at org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found at org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329) at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196) at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430) at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070) at org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201) at org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270) Thanks, Raj. On Wed, Nov 18, 2015 at 12:27 PM, Pubudu Gunatilaka <pubu...@wso2.com> wrote: > Hi, > > I did not get the mentioned exception and application came to created > state when I undeploy the application. But instance did not get terminated > and it is running on openstack. I enabled debug logs in cloud controller > and the log is attached. > > Thank you! > > On Wed, Nov 18, 2015 at 11:09 AM, Rajkumar Rajaratnam <rajkum...@wso2.com> > wrote: > >> Hi Akila, >> >> I haven't tested it with ec2/openstack/gce, but the issue you got is not >> bound to any specific iaases, I guess. >> >> According to your stack trace, the cartridge "php-gce" is not found in >> CloudControllerContext#cartridgeTypeToCartridgeMap when Stratos tried to >> terminate the instance. That means, something gone wrong in server restart, >> and the map didn't get populated correctly. But, I have restarted Stratos >> and verified that this map is updated correctly. >> >> If you have complete debug logs after server restart, please attach it >> here. Otherwise, lets wait for Pubudu's test results. >> >> Thanks, >> Raj. >> >> On Wed, Nov 18, 2015 at 10:38 AM, Gayan Gunarathne <gay...@wso2.com> >> wrote: >> >>> Hi Akila, >>> >>> Seems like cartridge type "php-gce" not available in the cloud >>> controller context. To simplify this did you check whether cartridge " >>> php-gce" is available in that case? >>> >>> Thanks, >>> Gayan >>> >>> On Wed, Nov 18, 2015 at 12:35 AM, Akila Ravihansa Perera < >>> raviha...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> I faced an issue where if I restart the Stratos server after >>>> successfully deploying an app then try to undeploy the app, it will throw >>>> the following error; >>>> >>>> [2015-11-18 00:31:55,537] ERROR >>>> {org.apache.stratos.cloud.controller.services.impl.InstanceTerminator} - >>>> Instance termination failed! MemberContext >>>> [applicationId=single-cartridge-app-gce, cartridgeType=php-gce, >>>> clusterId=single-cartridge-app-gce.my-php-app-gce.php-gce.domain, >>>> memberId=single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca, >>>> instanceId=null, partition=Partition [id=partition-1, description=null, >>>> isPublic=false, provider=gce, properties=Properties [properties=[Property >>>> [name=region, value=asia-east1], Property [name=zone, >>>> value=asia-east1-a]]]], defaultPrivateIP=null, defaultPublicIP=null, >>>> allocatedIPs=null, publicIPs=null, privateIPs=null, initTime=1447786797325, >>>> lbClusterId=null, networkPartitionId=network-partition-gce, >>>> kubernetesPodId=null, kubernetesPodName=null, loadBalancingIPType=Private, >>>> instanceMetadata=null, properties=Properties [properties=[Property >>>> [name=MIN_COUNT, value=1], Property [name=SCALING_DECISION_ID, >>>> value=single-cartridge-app-gce.my-php-app-gce.php-gce.domain-52271c86-c13e-4db1-8499-8c28c149b7b2], >>>> Property [name=EC2_REGION, value=asia-east1], Property >>>> [name=EC2_AVAILABILITY_ZONE, value=asia-east1-a]]]] >>>> org.apache.stratos.cloud.controller.exception.InvalidCartridgeTypeException: >>>> Member termination failed, could not find cartridge in cloud controller >>>> context: [cartridge-type] php-gce [member-id] >>>> single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca >>>> at >>>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:307) >>>> at >>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Exception in thread "pool-32-thread-3" >>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>> Instance termination failed! MemberContext >>>> [applicationId=single-cartridge-app-gce, cartridgeType=php-gce, >>>> clusterId=single-cartridge-app-gce.my-php-app-gce.php-gce.domain, >>>> memberId=single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca, >>>> instanceId=null, partition=Partition [id=partition-1, description=null, >>>> isPublic=false, provider=gce, properties=Properties [properties=[Property >>>> [name=region, value=asia-east1], Property [name=zone, >>>> value=asia-east1-a]]]], defaultPrivateIP=null, defaultPublicIP=null, >>>> allocatedIPs=null, publicIPs=null, privateIPs=null, initTime=1447786797325, >>>> lbClusterId=null, networkPartitionId=network-partition-gce, >>>> kubernetesPodId=null, kubernetesPodName=null, loadBalancingIPType=Private, >>>> instanceMetadata=null, properties=Properties [properties=[Property >>>> [name=MIN_COUNT, value=1], Property [name=SCALING_DECISION_ID, >>>> value=single-cartridge-app-gce.my-php-app-gce.php-gce.domain-52271c86-c13e-4db1-8499-8c28c149b7b2], >>>> Property [name=EC2_REGION, value=asia-east1], Property >>>> [name=EC2_AVAILABILITY_ZONE, value=asia-east1-a]]]] >>>> at >>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:63) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: >>>> org.apache.stratos.cloud.controller.exception.InvalidCartridgeTypeException: >>>> Member termination failed, could not find cartridge in cloud controller >>>> context: [cartridge-type] php-gce [member-id] >>>> single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca >>>> at >>>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:307) >>>> at >>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56) >>>> ... 3 more >>>> >>>> >>>> Has anyone faced this before? >>>> >>>> -- >>>> Akila Ravihansa Perera >>>> WSO2 Inc.; http://wso2.com/ >>>> >>>> Blog: http://ravihansa3000.blogspot.com >>>> >>> >>> >>> >>> -- >>> >>> Gayan Gunarathne >>> Technical Lead, WSO2 Inc. (http://wso2.com) >>> Committer & PMC Member, Apache Stratos >>> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985> >>> >>> >>> >> >> >> >> -- >> Rajkumar Rajaratnam >> Committer & PMC Member, Apache Stratos >> Software Engineer, WSO2 >> >> Mobile : +94777568639 >> Blog : rajkumarr.com >> > > > > -- > *Pubudu Gunatilaka* > Committer and PMC Member - Apache Stratos > Software Engineer > WSO2, Inc.: http://wso2.com > mobile : +94774079049 <%2B94772207163> > > -- Rajkumar Rajaratnam Committer & PMC Member, Apache Stratos Software Engineer, WSO2 Mobile : +94777568639 Blog : rajkumarr.com