Hi Akila,
On Wed, Oct 7, 2015 at 7:08 PM, Akila Ravihansa Perera <raviha...@wso2.com>
wrote:
> Hi Reka,
>
> Could you please explain the fix that you are planning to do?
>
I have added only Null check to the places where Application Monitor was
invoked. It is required as in the force undeployment, we are removing all
the monitors without considering the monitor hierarchy.
>
> While going through the GroupMonitor class, few issues were found related
> to the logic in handling Thread Executors. The problem is when an
> application is undeployed, execution jobs are never stopped from Thread
> Executor service. The jobs will keep running even after application is been
> undeployed. Have we addressed this case in Thread Executors?
>
This is being handled in the ApplicationBuilder line 427, 302 and 306. We
were stopping the monitor executor service using below code:
/**
* This will stop the scheduler which is running for the monitor
*/
protected void stopScheduler() {
schedulerFuture.cancel(true);
}
If it is not working properly, then might need to check further on this.
When stopScheduler is invoked, if the thread is started, then we don't have
control over it. It will get executed for that iteration and then only it
will get stopped. I have tried a simple sample with debug logs for monitor.
I couldn't see the monitor thread after undeployment. Has this particular
issue occurred during force undeployment?
>
> Also monitoringRunnable implementation does not catch any possible
> exceptions. This is the reason why we are seeing NPE's in the console
> output while running integration tests but not seeing them in
> wso2carbon.log. I think we better catch a generic exception in
> monitoringRunnable run method.
>
+1 for catching generic exception in monitoringRunnable.
Thanks,
Reka
>
> Thanks.
>
> On Wed, Oct 7, 2015 at 6:34 PM, Reka Thirunavukkarasu <r...@wso2.com>
> wrote:
>
>> Hi Imesh,
>>
>> On Wed, Oct 7, 2015 at 7:27 AM, Reka Thirunavukkarasu <r...@wso2.com>
>> wrote:
>>
>>> Hi Imesh,
>>>
>>> On Wed, Oct 7, 2015 at 5:46 AM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> GroupMonitor:
>>>>
>>>> ApplicationMonitor applicationMonitor =
>>>> AutoscalerContext.getInstance().
>>>> getAppMonitor(appId);
>>>>
>>>> [161] //When the application is getting un-deployed, need to avoid
>>>> [162] // checking the minimum count sanctification
>>>> [163] if (!applicationMonitor.isTerminating()) {
>>>>
>>>> In GroupMonitor class the line number 163 is shown above, may be due to
>>>> some reason application monitor is not found when executing the group
>>>> monitor.
>>>>
>>>> @Reka: Do you have any thoughts on this?
>>>>
>>>
>>> Yah..It could happen when force undeployment is invoked. I will check on
>>> this..
>>>
>>
>> I have fixed the possible places which could cause NPE by scanning
>> through the code. I will do a quick round of testing and push the changes
>> to Stratos-4.1.x branch.
>>
>> Thanks,
>> Reka
>>
>>>
>>> Thanks,
>>> Reka
>>>
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Oct 7, 2015 at 1:20 AM, Thanuja Uruththirakodeeswaran <
>>>> thanu...@wso2.com> wrote:
>>>>
>>>>> Hi Akila,
>>>>>
>>>>>
>>>>> On Wed, Oct 7, 2015 at 1:09 AM, Akila Ravihansa Perera <
>>>>> raviha...@wso2.com> wrote:
>>>>>
>>>>>> Hi Thanuja,
>>>>>>
>>>>>> Thanks for the update.
>>>>>>
>>>>>> In both of those cases, did you observe any build failures? If so,
>>>>>> what are the tests that failed?
>>>>>>
>>>>>
>>>>> For the complete stratos build, following test cases are failing with
>>>>> NPE in ClusterMonitor as shown above:
>>>>> Failed tests:
>>>>> ApplicationBurstingTestCase.testApplicationBusting:138
>>>>> GroupStartupOrderTestCase.testTerminationBehavior:194 Application
>>>>> status did not change to Active: [application-id] group-startup-order-test
>>>>> expected:<Active> but was:<Inactive>
>>>>> SampleApplicationStartupTestCase.testApplication:103
>>>>> SampleApplicationsTestCase.testApplication:57 » Runtime
>>>>> SampleApplicationsTestCase.testApplicationList:252 » Runtime
>>>>>
>>>>> SingleClusterScalingTestCase.testDeployApplication:117->assertClusterWithScaleDown:260
>>>>> » ThreadTimeout
>>>>>
>>>>> When running integration test only, there is no test failure and I'm
>>>>> getting NPE in GroupMonitor class.
>>>>>
>>>>>>
>>>>>> I noticed the same behavior as you have explained. When I check the
>>>>>> log file
>>>>>> in
>>>>>> /test-integration/target/<carbon-home>/repository/logs/wso2carbon.log, I
>>>>>> didn't see any NPE's related to Drool files. Only one NPE is thrown from
>>>>>> GroupMonitor class. Can you check whether it is the same for you?
>>>>>>
>>>>>
>>>>> Yes, if I ran integration test directly, NPE was thrown in
>>>>> GroupMonitor class and no drools error were there.
>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Wed, Oct 7, 2015 at 1:05 AM, Akila Ravihansa Perera <
>>>>>> raviha...@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It seems integration tests are failing intermittently regardless of
>>>>>>> NPE's thrown from Drool files. I've added more debug logs to
>>>>>>> troubleshoot
>>>>>>> the issue. I've set the default log level in AS and CC packages to
>>>>>>> DEBUG in
>>>>>>> integration tests. This will help us to identify the root causes of
>>>>>>> integration test failures in Jenkins build.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 6, 2015 at 12:38 PM, Thanuja Uruththirakodeeswaran <
>>>>>>> thanu...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Akila,
>>>>>>>>
>>>>>>>> I'll check this and let you know.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Tue, Oct 6, 2015 at 12:31 PM, Akila Ravihansa Perera <
>>>>>>>> raviha...@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Thanuja,
>>>>>>>>>
>>>>>>>>> I noticed that integration tests are failing in stratos-4.1.x
>>>>>>>>> branch due to an issue in Drools files. It seems there have been
>>>>>>>>> couple of
>>>>>>>>> modifications done for Drools files with new metering dashboard
>>>>>>>>> feature.
>>>>>>>>> Perhaps it is incompatible with autoscaler APIs. Could you look into
>>>>>>>>> that
>>>>>>>>> please?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Akila Ravihansa Perera
>>>>>>>>> WSO2 Inc.; http://wso2.com/
>>>>>>>>>
>>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanuja Uruththirakodeeswaran
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc.;http://wso2.com
>>>>>>>> lean.enterprise.middleware
>>>>>>>>
>>>>>>>> mobile: +94 774363167
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Akila Ravihansa Perera
>>>>>>> WSO2 Inc.; http://wso2.com/
>>>>>>>
>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> WSO2 Inc.; http://wso2.com/
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanuja Uruththirakodeeswaran
>>>>> Software Engineer
>>>>> WSO2 Inc.;http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> mobile: +94 774363167
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Senior Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Senior Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
>
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.; http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
