Hi Pubudu,
In your case, the monitors are not started/scheduled after server restart
because of the following error. There seems to be a communication issue
between AS and CC, may be CC web service was not ready at that time.
TID: [0] [STRATOS] [2015-11-18 12:08:41,675] INFO
{org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
Starting dependent monitor: [application]
c3c955b2-b459-ebd1-ab6b-2ada34d5fd50 [component]
c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
TID: [0] [STRATOS] [2015-11-18 12:08:41,675] INFO
{org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
Monitor scheduled: [type] cluster [component]
c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
TID: [0] [STRATOS] [2015-11-18 12:08:41,676] INFO
{org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
Starting monitor: [type] cluster [component]
c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
TID: [0] [STRATOS] [2015-11-18 12:08:41,680] INFO
{org.apache.stratos.autoscaler.util.AutoscalerUtil} - Monitor started
successfully: [application] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50
[dependents] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
[startup-time] 0 seconds
TID: [0] [STRATOS] [2015-11-18 12:08:41,830] INFO
{org.apache.axis2.transport.http.HTTPSender} - Unable to sendViaPost to
url[https://localhost:9443/services/CloudControllerService/]
org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found
at
org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329)
at
org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
at
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070)
at
org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201)
at
org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270)
at
org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
at
org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
TID: [0] [STRATOS] [2015-11-18 12:08:41,831] ERROR
{org.apache.stratos.autoscaler.monitor.MonitorFactory} - Error while
validating deployment policy from cloud controller [network-partition-id]
network-partition-openstack
org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found
at
org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329)
at
org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
at
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070)
at
org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201)
at
org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270)
at
org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
at
org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
TID: [0] [STRATOS] [2015-11-18 12:08:41,831] ERROR
{org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
An error occurred while starting monitor: [type] cluster [component]
c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
java.lang.RuntimeException: Error while validating deployment policy from
cloud controller [network-partition-id] network-partition-openstack
at
org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:278)
at
org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
at
org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found
at
org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329)
at
org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
at
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070)
at
org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201)
at
org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270)
Thanks,
Raj.
On Wed, Nov 18, 2015 at 12:27 PM, Pubudu Gunatilaka <pubu...@wso2.com>
wrote:
> Hi,
>
> I did not get the mentioned exception and application came to created
> state when I undeploy the application. But instance did not get terminated
> and it is running on openstack. I enabled debug logs in cloud controller
> and the log is attached.
>
> Thank you!
>
> On Wed, Nov 18, 2015 at 11:09 AM, Rajkumar Rajaratnam <rajkum...@wso2.com>
> wrote:
>
>> Hi Akila,
>>
>> I haven't tested it with ec2/openstack/gce, but the issue you got is not
>> bound to any specific iaases, I guess.
>>
>> According to your stack trace, the cartridge "php-gce" is not found in
>> CloudControllerContext#cartridgeTypeToCartridgeMap when Stratos tried to
>> terminate the instance. That means, something gone wrong in server restart,
>> and the map didn't get populated correctly. But, I have restarted Stratos
>> and verified that this map is updated correctly.
>>
>> If you have complete debug logs after server restart, please attach it
>> here. Otherwise, lets wait for Pubudu's test results.
>>
>> Thanks,
>> Raj.
>>
>> On Wed, Nov 18, 2015 at 10:38 AM, Gayan Gunarathne <gay...@wso2.com>
>> wrote:
>>
>>> Hi Akila,
>>>
>>> Seems like cartridge type "php-gce" not available in the cloud
>>> controller context. To simplify this did you check whether cartridge "
>>> php-gce" is available in that case?
>>>
>>> Thanks,
>>> Gayan
>>>
>>> On Wed, Nov 18, 2015 at 12:35 AM, Akila Ravihansa Perera <
>>> raviha...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I faced an issue where if I restart the Stratos server after
>>>> successfully deploying an app then try to undeploy the app, it will throw
>>>> the following error;
>>>>
>>>> [2015-11-18 00:31:55,537] ERROR
>>>> {org.apache.stratos.cloud.controller.services.impl.InstanceTerminator} -
>>>> Instance termination failed! MemberContext
>>>> [applicationId=single-cartridge-app-gce, cartridgeType=php-gce,
>>>> clusterId=single-cartridge-app-gce.my-php-app-gce.php-gce.domain,
>>>> memberId=single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca,
>>>> instanceId=null, partition=Partition [id=partition-1, description=null,
>>>> isPublic=false, provider=gce, properties=Properties [properties=[Property
>>>> [name=region, value=asia-east1], Property [name=zone,
>>>> value=asia-east1-a]]]], defaultPrivateIP=null, defaultPublicIP=null,
>>>> allocatedIPs=null, publicIPs=null, privateIPs=null, initTime=1447786797325,
>>>> lbClusterId=null, networkPartitionId=network-partition-gce,
>>>> kubernetesPodId=null, kubernetesPodName=null, loadBalancingIPType=Private,
>>>> instanceMetadata=null, properties=Properties [properties=[Property
>>>> [name=MIN_COUNT, value=1], Property [name=SCALING_DECISION_ID,
>>>> value=single-cartridge-app-gce.my-php-app-gce.php-gce.domain-52271c86-c13e-4db1-8499-8c28c149b7b2],
>>>> Property [name=EC2_REGION, value=asia-east1], Property
>>>> [name=EC2_AVAILABILITY_ZONE, value=asia-east1-a]]]]
>>>> org.apache.stratos.cloud.controller.exception.InvalidCartridgeTypeException:
>>>> Member termination failed, could not find cartridge in cloud controller
>>>> context: [cartridge-type] php-gce [member-id]
>>>> single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca
>>>> at
>>>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:307)
>>>> at
>>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Exception in thread "pool-32-thread-3"
>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException:
>>>> Instance termination failed! MemberContext
>>>> [applicationId=single-cartridge-app-gce, cartridgeType=php-gce,
>>>> clusterId=single-cartridge-app-gce.my-php-app-gce.php-gce.domain,
>>>> memberId=single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca,
>>>> instanceId=null, partition=Partition [id=partition-1, description=null,
>>>> isPublic=false, provider=gce, properties=Properties [properties=[Property
>>>> [name=region, value=asia-east1], Property [name=zone,
>>>> value=asia-east1-a]]]], defaultPrivateIP=null, defaultPublicIP=null,
>>>> allocatedIPs=null, publicIPs=null, privateIPs=null, initTime=1447786797325,
>>>> lbClusterId=null, networkPartitionId=network-partition-gce,
>>>> kubernetesPodId=null, kubernetesPodName=null, loadBalancingIPType=Private,
>>>> instanceMetadata=null, properties=Properties [properties=[Property
>>>> [name=MIN_COUNT, value=1], Property [name=SCALING_DECISION_ID,
>>>> value=single-cartridge-app-gce.my-php-app-gce.php-gce.domain-52271c86-c13e-4db1-8499-8c28c149b7b2],
>>>> Property [name=EC2_REGION, value=asia-east1], Property
>>>> [name=EC2_AVAILABILITY_ZONE, value=asia-east1-a]]]]
>>>> at
>>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:63)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by:
>>>> org.apache.stratos.cloud.controller.exception.InvalidCartridgeTypeException:
>>>> Member termination failed, could not find cartridge in cloud controller
>>>> context: [cartridge-type] php-gce [member-id]
>>>> single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca
>>>> at
>>>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:307)
>>>> at
>>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56)
>>>> ... 3 more
>>>>
>>>>
>>>> Has anyone faced this before?
>>>>
>>>> --
>>>> Akila Ravihansa Perera
>>>> WSO2 Inc.; http://wso2.com/
>>>>
>>>> Blog: http://ravihansa3000.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>> Committer & PMC Member, Apache Stratos
>>> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
>
> --
> *Pubudu Gunatilaka*
> Committer and PMC Member - Apache Stratos
> Software Engineer
> WSO2, Inc.: http://wso2.com
> mobile : +94774079049 <%2B94772207163>
>
>
--
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2
Mobile : +94777568639
Blog : rajkumarr.com
