On Wed, Nov 18, 2015 at 1:02 PM, Pubudu Gunatilaka <pubu...@wso2.com> wrote:
> Hi Raj,
>
> I guess this error occurred when I tried to refresh the browser before the
> startup. If you check the log, in line number 1702 there is a graceful
> application undeployment.
>
AS will validate deployment policy with CC when it creates monitors. The
above exception was thrown from monitor factory while creating cluster
monitor. I guess it is not related to browser refresh, isn't it? Can you
enable debug logs for AS as well? It will reveal that the cluster monitors
are not created.
Thanks,
Raj.
>
> Thank you!
>
> On Wed, Nov 18, 2015 at 12:52 PM, Rajkumar Rajaratnam <rajkum...@wso2.com>
> wrote:
>
>> Hi Pubudu,
>>
>> In your case, the monitors are not started/scheduled after server restart
>> because of the following error. There seems to be a communication issue
>> between AS and CC, may be CC web service was not ready at that time.
>>
>>
>> TID: [0] [STRATOS] [2015-11-18 12:08:41,675] INFO
>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
>> Starting dependent monitor: [application]
>> c3c955b2-b459-ebd1-ab6b-2ada34d5fd50 [component]
>> c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
>> TID: [0] [STRATOS] [2015-11-18 12:08:41,675] INFO
>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
>> Monitor scheduled: [type] cluster [component]
>> c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
>> TID: [0] [STRATOS] [2015-11-18 12:08:41,676] INFO
>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
>> Starting monitor: [type] cluster [component]
>> c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
>> TID: [0] [STRATOS] [2015-11-18 12:08:41,680] INFO
>> {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Monitor started
>> successfully: [application] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50
>> [dependents] c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
>> [startup-time] 0 seconds
>> TID: [0] [STRATOS] [2015-11-18 12:08:41,830] INFO
>> {org.apache.axis2.transport.http.HTTPSender} - Unable to sendViaPost to
>> url[https://localhost:9443/services/CloudControllerService/]
>> org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found
>> at
>> org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329)
>> at
>> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196)
>> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
>> at
>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
>> at
>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
>> at
>> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
>> at
>> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
>> at
>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
>> at
>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070)
>> at
>> org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201)
>> at
>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270)
>> at
>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
>> at
>> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> TID: [0] [STRATOS] [2015-11-18 12:08:41,831] ERROR
>> {org.apache.stratos.autoscaler.monitor.MonitorFactory} - Error while
>> validating deployment policy from cloud controller [network-partition-id]
>> network-partition-openstack
>> org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found
>> at
>> org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329)
>> at
>> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196)
>> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
>> at
>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
>> at
>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
>> at
>> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
>> at
>> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
>> at
>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
>> at
>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070)
>> at
>> org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201)
>> at
>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270)
>> at
>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
>> at
>> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> TID: [0] [STRATOS] [2015-11-18 12:08:41,831] ERROR
>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
>> An error occurred while starting monitor: [type] cluster [component]
>> c3c955b2-b459-ebd1-ab6b-2ada34d5fd50.alias2.wso2brs-210.domain
>> java.lang.RuntimeException: Error while validating deployment policy from
>> cloud controller [network-partition-id] network-partition-openstack
>> at
>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:278)
>> at
>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
>> at
>> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.axis2.AxisFault: Transport error: 404 Error: Not
>> Found
>> at
>> org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:329)
>> at
>> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196)
>> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
>> at
>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
>> at
>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
>> at
>> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
>> at
>> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
>> at
>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
>> at
>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.validateDeploymentPolicyNetworkPartition(CloudControllerServiceStub.java:3070)
>> at
>> org.apache.stratos.common.client.CloudControllerServiceClient.validateNetworkPartitionOfDeploymentPolicy(CloudControllerServiceClient.java:201)
>> at
>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:270)
>>
>> Thanks,
>> Raj.
>>
>> On Wed, Nov 18, 2015 at 12:27 PM, Pubudu Gunatilaka <pubu...@wso2.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I did not get the mentioned exception and application came to created
>>> state when I undeploy the application. But instance did not get terminated
>>> and it is running on openstack. I enabled debug logs in cloud controller
>>> and the log is attached.
>>>
>>> Thank you!
>>>
>>> On Wed, Nov 18, 2015 at 11:09 AM, Rajkumar Rajaratnam <
>>> rajkum...@wso2.com> wrote:
>>>
>>>> Hi Akila,
>>>>
>>>> I haven't tested it with ec2/openstack/gce, but the issue you got is
>>>> not bound to any specific iaases, I guess.
>>>>
>>>> According to your stack trace, the cartridge "php-gce" is not found in
>>>> CloudControllerContext#cartridgeTypeToCartridgeMap when Stratos tried to
>>>> terminate the instance. That means, something gone wrong in server restart,
>>>> and the map didn't get populated correctly. But, I have restarted Stratos
>>>> and verified that this map is updated correctly.
>>>>
>>>> If you have complete debug logs after server restart, please attach it
>>>> here. Otherwise, lets wait for Pubudu's test results.
>>>>
>>>> Thanks,
>>>> Raj.
>>>>
>>>> On Wed, Nov 18, 2015 at 10:38 AM, Gayan Gunarathne <gay...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Akila,
>>>>>
>>>>> Seems like cartridge type "php-gce" not available in the cloud
>>>>> controller context. To simplify this did you check whether cartridge "
>>>>> php-gce" is available in that case?
>>>>>
>>>>> Thanks,
>>>>> Gayan
>>>>>
>>>>> On Wed, Nov 18, 2015 at 12:35 AM, Akila Ravihansa Perera <
>>>>> raviha...@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I faced an issue where if I restart the Stratos server after
>>>>>> successfully deploying an app then try to undeploy the app, it will
>>>>>> throw
>>>>>> the following error;
>>>>>>
>>>>>> [2015-11-18 00:31:55,537] ERROR
>>>>>> {org.apache.stratos.cloud.controller.services.impl.InstanceTerminator} -
>>>>>> Instance termination failed! MemberContext
>>>>>> [applicationId=single-cartridge-app-gce, cartridgeType=php-gce,
>>>>>> clusterId=single-cartridge-app-gce.my-php-app-gce.php-gce.domain,
>>>>>> memberId=single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca,
>>>>>> instanceId=null, partition=Partition [id=partition-1, description=null,
>>>>>> isPublic=false, provider=gce, properties=Properties [properties=[Property
>>>>>> [name=region, value=asia-east1], Property [name=zone,
>>>>>> value=asia-east1-a]]]], defaultPrivateIP=null, defaultPublicIP=null,
>>>>>> allocatedIPs=null, publicIPs=null, privateIPs=null,
>>>>>> initTime=1447786797325,
>>>>>> lbClusterId=null, networkPartitionId=network-partition-gce,
>>>>>> kubernetesPodId=null, kubernetesPodName=null,
>>>>>> loadBalancingIPType=Private,
>>>>>> instanceMetadata=null, properties=Properties [properties=[Property
>>>>>> [name=MIN_COUNT, value=1], Property [name=SCALING_DECISION_ID,
>>>>>> value=single-cartridge-app-gce.my-php-app-gce.php-gce.domain-52271c86-c13e-4db1-8499-8c28c149b7b2],
>>>>>> Property [name=EC2_REGION, value=asia-east1], Property
>>>>>> [name=EC2_AVAILABILITY_ZONE, value=asia-east1-a]]]]
>>>>>> org.apache.stratos.cloud.controller.exception.InvalidCartridgeTypeException:
>>>>>> Member termination failed, could not find cartridge in cloud controller
>>>>>> context: [cartridge-type] php-gce [member-id]
>>>>>> single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:307)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Exception in thread "pool-32-thread-3"
>>>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException:
>>>>>> Instance termination failed! MemberContext
>>>>>> [applicationId=single-cartridge-app-gce, cartridgeType=php-gce,
>>>>>> clusterId=single-cartridge-app-gce.my-php-app-gce.php-gce.domain,
>>>>>> memberId=single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca,
>>>>>> instanceId=null, partition=Partition [id=partition-1, description=null,
>>>>>> isPublic=false, provider=gce, properties=Properties [properties=[Property
>>>>>> [name=region, value=asia-east1], Property [name=zone,
>>>>>> value=asia-east1-a]]]], defaultPrivateIP=null, defaultPublicIP=null,
>>>>>> allocatedIPs=null, publicIPs=null, privateIPs=null,
>>>>>> initTime=1447786797325,
>>>>>> lbClusterId=null, networkPartitionId=network-partition-gce,
>>>>>> kubernetesPodId=null, kubernetesPodName=null,
>>>>>> loadBalancingIPType=Private,
>>>>>> instanceMetadata=null, properties=Properties [properties=[Property
>>>>>> [name=MIN_COUNT, value=1], Property [name=SCALING_DECISION_ID,
>>>>>> value=single-cartridge-app-gce.my-php-app-gce.php-gce.domain-52271c86-c13e-4db1-8499-8c28c149b7b2],
>>>>>> Property [name=EC2_REGION, value=asia-east1], Property
>>>>>> [name=EC2_AVAILABILITY_ZONE, value=asia-east1-a]]]]
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:63)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Caused by:
>>>>>> org.apache.stratos.cloud.controller.exception.InvalidCartridgeTypeException:
>>>>>> Member termination failed, could not find cartridge in cloud controller
>>>>>> context: [cartridge-type] php-gce [member-id]
>>>>>> single-cartridge-app-gce.my-php-app-gce.php-gce.domainbead7a03-8589-4c0c-a421-b713491d51ca
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:307)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56)
>>>>>> ... 3 more
>>>>>>
>>>>>>
>>>>>> Has anyone faced this before?
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> WSO2 Inc.; http://wso2.com/
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Gayan Gunarathne
>>>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>>>> Committer & PMC Member, Apache Stratos
>>>>> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Committer & PMC Member, Apache Stratos
>>>> Software Engineer, WSO2
>>>>
>>>> Mobile : +94777568639
>>>> Blog : rajkumarr.com
>>>>
>>>
>>>
>>>
>>> --
>>> *Pubudu Gunatilaka*
>>> Committer and PMC Member - Apache Stratos
>>> Software Engineer
>>> WSO2, Inc.: http://wso2.com
>>> mobile : +94774079049 <%2B94772207163>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
>
> --
> *Pubudu Gunatilaka*
> Committer and PMC Member - Apache Stratos
> Software Engineer
> WSO2, Inc.: http://wso2.com
> mobile : +94774079049 <%2B94772207163>
>
>
--
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2
Mobile : +94777568639
Blog : rajkumarr.com
