Hi Raj,
I think the problem here is that the transport might not be ready even
though the CC is activated. Until the transport ready, AS should not invoke
CC. If we call CC via OSGI service, then this problem would not occur.
Anyway, that is not possible when considering the distributed setup. I
observed earlier that the ntask component usually scheduled the task after
the transport is ready. In that case, CC can only send the CompleteTopology
after ntask component schedule the task. Then, AS will receive Topology and
start the monitors. In that case, i hope that there won't be any issues.
However not sure whether ntask has a dependency to transport.
So, are we receiving the CompleteTopology very early before the transport
is started?
Thanks,
Reka
On Tue, Mar 10, 2015 at 1:26 AM, Rajkumar Rajaratnam <rajkum...@wso2.com>
wrote:
> Hi Reka,
>
> I checked the flow and it is exactly same as you mentioned >> AS makes 1st
> call to CC after CC component is activated.
> Still AS is failing to communicate with CC. This is error message. This
> error message means that the target is not available. But the target URL is
> correct. It means CC service is not ready yet. I will look into it.
>
>
> [2015-03-10 13:34:19,967] INFO
> {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} - WSO2
> Carbon started in 33 sec
> [2015-03-10 13:34:20,075] INFO
> {org.apache.axis2.transport.http.HTTPSender} - Unable to sendViaPost to
> url[https://localhost:9443/services/CloudControllerService/]
> org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found
> at
> org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:311)
> at
> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:194)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
> at
> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
> at
> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
> at
> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
> at
> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.getDeploymentPolicy(CloudControllerServiceStub.java:9172)
> at
> org.apache.stratos.common.client.CloudControllerServiceClient.getDeploymentPolicy(CloudControllerServiceClient.java:227)
> at
> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:258)
> at
> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
> at
> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:789)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [2015-03-10 13:34:20,076] ERROR
> {org.apache.stratos.autoscaler.monitor.MonitorFactory} - Error while
> getting deployment policy from cloud controller [deployment-policy-id]
> deployment-policy-1
> org.apache.axis2.AxisFault: Transport error: 404 Error: Not Found
> at
> org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:311)
> at
> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:194)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
> at
> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
> at
> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
> at
> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
> at
> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.getDeploymentPolicy(CloudControllerServiceStub.java:9172)
> at
> org.apache.stratos.common.client.CloudControllerServiceClient.getDeploymentPolicy(CloudControllerServiceClient.java:227)
> at
> org.apache.stratos.autoscaler.monitor.MonitorFactory.getClusterMonitor(MonitorFactory.java:258)
> at
> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:83)
> at
> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:789)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> Thanks.
>
> On Tue, Mar 10, 2015 at 12:42 PM, Rajkumar Rajaratnam <rajkum...@wso2.com>
> wrote:
>
>>
>>
>> On Tue, Mar 10, 2015 at 12:17 PM, Reka Thirunavukkarasu <r...@wso2.com>
>> wrote:
>>
>>> According to my understanding, autoscaler was only dependent on Topology
>>> in order to trigger anything(startup of monitors for the already deployed
>>> application) in the restart. In that case, even though autoscaler component
>>> starts first, it has to wait until the CompleteTopologyEvent is received.
>>> At the moment autoscaler receives CompleteTopologyEvent, we can assume that
>>> the CC is ready to process.
>>>
>>> It seems that now the flow has changed and autoscaler is no longer
>>> dependent only on the CompleteTopology to trigger the startup of monitors
>>> in the restart. Can't we make the autoscaler to only dependent on the
>>> CompleteTopology rather than directly depending on CC?
>>>
>>
>> Thanks Reka. That will solve the issue. I will have a look at the flow
>> and fix it.
>>
>> Thanks.
>>
>>
>>>
>>> Thanks,
>>> Reka
>>>
>>> On Mon, Mar 9, 2015 at 11:40 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> As already mentioned OSGI dependencies will not work in a distributed
>>>> setup. Instead I prefer a event based mechanism.
>>>>
>>>> On Tue, Mar 10, 2015 at 12:06 PM, Rajkumar Rajaratnam <
>>>> rajkum...@wso2.com> wrote:
>>>>
>>>>> s/same machine/single JVM
>>>>>
>>>>> On Tue, Mar 10, 2015 at 11:51 AM, Rajkumar Rajaratnam <
>>>>> rajkum...@wso2.com> wrote:
>>>>>
>>>>>> Hi Isuru,
>>>>>>
>>>>>> Yes this happening when we are creating monitors by reading
>>>>>> application topology. And I guess enforcing OSGi dependencies among
>>>>>> components will completely break the distributed setup. Since components
>>>>>> are not running in the same machine >> AS will be waiting forever for CC
>>>>>> service to become active.
>>>>>>
>>>>>> As you said, it is better to go with event based solution.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Tue, Mar 10, 2015 at 11:45 AM, Isuru Haththotuwa <
>>>>>> isu...@apache.org> wrote:
>>>>>>
>>>>>>> Hi Raj,
>>>>>>>
>>>>>>> On Tue, Mar 10, 2015 at 11:31 AM, Rajkumar Rajaratnam <
>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Devs,
>>>>>>>>
>>>>>>>> I have found issues in stratos server restart.
>>>>>>>>
>>>>>>>> As you know we don't persist monitors. We read the topology and
>>>>>>>> create monitors when we restart the Stratos. While we are creating
>>>>>>>> monitors, we need to communicate with cloud controller service
>>>>>>>> in-order to
>>>>>>>> do things like getting deployment policy, network partitions,
>>>>>>>> validating
>>>>>>>> them and so on. In the single machine setup, AS component is starting
>>>>>>>> before CC. So when AS tries to communicate with CC, it is failing >>
>>>>>>>> ultimately monitor creation will fail.
>>>>>>>>
>>>>>>> Does this issue come in when we are creating Application Monitors?
>>>>>>> AS starts before CC -> AS tries to restore the Application Monitors from
>>>>>>> the local Applications Toplogy -> tries to communicate with CC -> leads
>>>>>>> to
>>>>>>> the issue?
>>>>>>>
>>>>>>>>
>>>>>>>> What would be solution here? Is there any way to enforce start up
>>>>>>>> orders between components? I know we can use OSGI dependencies to
>>>>>>>> enforce
>>>>>>>> such order. We can make AS component to wait until CC component is
>>>>>>>> activated. But will that solve the problem in distributed setup?
>>>>>>>>
>>>>>>> AFAIK enforcing the bundle startup order will not solve this in a
>>>>>>> distributed setup. How about an event related solution? CC (or any other
>>>>>>> related component) sending an event to say that it is started? To avoid
>>>>>>> the
>>>>>>> deadlock in an event loss, maybe we can add a timeout as well.
>>>>>>>
>>>>>>>>
>>>>>>>> Please share your thoughts on this.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>> Software Engineer, WSO2
>>>>>>>>
>>>>>>>> Mobile : +94777568639
>>>>>>>> Blog : rajkumarr.com
>>>>>>>>
>>>>>>>> --
>>>>>>>> <http://rajkumarr.com>
>>>>>>>> <http://rajkumarr.com>
>>>>>>>> Thanks and Regards,
>>>>>>>>
>>>>>>>> Isuru H.
>>>>>>>> <http://rajkumarr.com>
>>>>>>>> +94 716 358 048 <http://rajkumarr.com>* <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>> * <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>> Software Engineer, WSO2
>>>>>>
>>>>>> Mobile : +94777568639
>>>>>> Blog : rajkumarr.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Committer & PMC Member, Apache Stratos
>>>>> Software Engineer, WSO2
>>>>>
>>>>> Mobile : +94777568639
>>>>> Blog : rajkumarr.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Udara Liyanage
>>>> Software Engineer
>>>> WSO2, Inc.: http://wso2.com
>>>> lean. enterprise. middleware
>>>>
>>>> web: http://udaraliyanage.wordpress.com
>>>> phone: +94 71 443 6897
>>>>
>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Senior Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
>
> --
> Rajkumar Rajaratnam
> Committer & PMC Member, Apache Stratos
> Software Engineer, WSO2
>
> Mobile : +94777568639
> Blog : rajkumarr.com
>
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
