Hi All,
NPE is not thrown by drool files. I reverted the changes in drool file
which we did for metering service locally and ran the integration test. I'm
getting NPE in same place at GroupMonitor (line - 163) as follows:
2015-10-08 13:42:28 INFO
{org.apache.stratos.integration.tests.application.SampleApplicationStartupTestCase}:150
- Waiting for application status to become INACTIVE
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,758] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,759] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,759] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,761] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,761] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,762] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,762] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,762] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,763] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,763] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,763] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,763] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- A cluster monitor is not found in autoscaler context [cluster]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,764] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,764] DEBUG
{org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,828] DEBUG
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group
monitor is running: [group] my-group6-group-startup-order-test
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 -
Exception in thread "pool-39-thread-93" java.lang.NullPointerException
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.lang.Thread.run(Thread.java:745)
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,852] DEBUG
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group
monitor is running: [group] my-group8-group-startup-order-test
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 -
Exception in thread "pool-39-thread-99" java.lang.NullPointerException
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.lang.Thread.run(Thread.java:745)
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,889] DEBUG
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group
monitor is running: [group] my-group9-group-startup-order-test
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 -
Exception in thread "pool-39-thread-100" java.lang.NullPointerException
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.lang.Thread.run(Thread.java:745)
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,918] DEBUG
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group
monitor is running: [group] my-group6-group-tom2-group-startup-order-test
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 -
Exception in thread "pool-39-thread-97" java.lang.NullPointerException
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.lang.Thread.run(Thread.java:745)
2015-10-08 13:42:35 INFO
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 -
[2015-10-08 13:42:35,941] DEBUG
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group
monitor is running: [group] my-group7-group-startup-order-test
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 -
Exception in thread "pool-39-thread-94" java.lang.NullPointerException
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-10-08 13:42:35 ERROR
{org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at
java.lang.Thread.run(Thread.java:745)
So this NPE is thrown because of drool file changes and we need to debug
ApplicationMonitor and ClusterMonitor classes when undeploying application.
Thanks.
On Wed, Oct 7, 2015 at 10:07 PM, Reka Thirunavukkarasu <r...@wso2.com>
wrote:
> Hi Akila,
>
> On Wed, Oct 7, 2015 at 7:08 PM, Akila Ravihansa Perera <raviha...@wso2.com
> > wrote:
>
>> Hi Reka,
>>
>> Could you please explain the fix that you are planning to do?
>>
> I have added only Null check to the places where Application Monitor was
> invoked. It is required as in the force undeployment, we are removing all
> the monitors without considering the monitor hierarchy.
>
>>
>> While going through the GroupMonitor class, few issues were found related
>> to the logic in handling Thread Executors. The problem is when an
>> application is undeployed, execution jobs are never stopped from Thread
>> Executor service. The jobs will keep running even after application is been
>> undeployed. Have we addressed this case in Thread Executors?
>>
>
> This is being handled in the ApplicationBuilder line 427, 302 and 306. We
> were stopping the monitor executor service using below code:
>
> /**
> * This will stop the scheduler which is running for the monitor
> */
> protected void stopScheduler() {
> schedulerFuture.cancel(true);
> }
>
> If it is not working properly, then might need to check further on this.
> When stopScheduler is invoked, if the thread is started, then we don't have
> control over it. It will get executed for that iteration and then only it
> will get stopped. I have tried a simple sample with debug logs for monitor.
> I couldn't see the monitor thread after undeployment. Has this particular
> issue occurred during force undeployment?
>
>>
>> Also monitoringRunnable implementation does not catch any possible
>> exceptions. This is the reason why we are seeing NPE's in the console
>> output while running integration tests but not seeing them in
>> wso2carbon.log. I think we better catch a generic exception in
>> monitoringRunnable run method.
>>
>
> +1 for catching generic exception in monitoringRunnable.
>
> Thanks,
> Reka
>
>>
>> Thanks.
>>
>> On Wed, Oct 7, 2015 at 6:34 PM, Reka Thirunavukkarasu <r...@wso2.com>
>> wrote:
>>
>>> Hi Imesh,
>>>
>>> On Wed, Oct 7, 2015 at 7:27 AM, Reka Thirunavukkarasu <r...@wso2.com>
>>> wrote:
>>>
>>>> Hi Imesh,
>>>>
>>>> On Wed, Oct 7, 2015 at 5:46 AM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> GroupMonitor:
>>>>>
>>>>> ApplicationMonitor applicationMonitor =
>>>>> AutoscalerContext.getInstance().
>>>>> getAppMonitor(appId);
>>>>>
>>>>> [161] //When the application is getting un-deployed, need to avoid
>>>>> [162] // checking the minimum count sanctification
>>>>> [163] if (!applicationMonitor.isTerminating()) {
>>>>>
>>>>> In GroupMonitor class the line number 163 is shown above, may be due
>>>>> to some reason application monitor is not found when executing the group
>>>>> monitor.
>>>>>
>>>>> @Reka: Do you have any thoughts on this?
>>>>>
>>>>
>>>> Yah..It could happen when force undeployment is invoked. I will check
>>>> on this..
>>>>
>>>
>>> I have fixed the possible places which could cause NPE by scanning
>>> through the code. I will do a quick round of testing and push the changes
>>> to Stratos-4.1.x branch.
>>>
>>> Thanks,
>>> Reka
>>>
>>>>
>>>> Thanks,
>>>> Reka
>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Oct 7, 2015 at 1:20 AM, Thanuja Uruththirakodeeswaran <
>>>>> thanu...@wso2.com> wrote:
>>>>>
>>>>>> Hi Akila,
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 7, 2015 at 1:09 AM, Akila Ravihansa Perera <
>>>>>> raviha...@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Thanuja,
>>>>>>>
>>>>>>> Thanks for the update.
>>>>>>>
>>>>>>> In both of those cases, did you observe any build failures? If so,
>>>>>>> what are the tests that failed?
>>>>>>>
>>>>>>
>>>>>> For the complete stratos build, following test cases are failing with
>>>>>> NPE in ClusterMonitor as shown above:
>>>>>> Failed tests:
>>>>>> ApplicationBurstingTestCase.testApplicationBusting:138
>>>>>> GroupStartupOrderTestCase.testTerminationBehavior:194 Application
>>>>>> status did not change to Active: [application-id]
>>>>>> group-startup-order-test
>>>>>> expected:<Active> but was:<Inactive>
>>>>>> SampleApplicationStartupTestCase.testApplication:103
>>>>>> SampleApplicationsTestCase.testApplication:57 » Runtime
>>>>>> SampleApplicationsTestCase.testApplicationList:252 » Runtime
>>>>>>
>>>>>> SingleClusterScalingTestCase.testDeployApplication:117->assertClusterWithScaleDown:260
>>>>>> » ThreadTimeout
>>>>>>
>>>>>> When running integration test only, there is no test failure and I'm
>>>>>> getting NPE in GroupMonitor class.
>>>>>>
>>>>>>>
>>>>>>> I noticed the same behavior as you have explained. When I check the
>>>>>>> log file
>>>>>>> in
>>>>>>> /test-integration/target/<carbon-home>/repository/logs/wso2carbon.log, I
>>>>>>> didn't see any NPE's related to Drool files. Only one NPE is thrown from
>>>>>>> GroupMonitor class. Can you check whether it is the same for you?
>>>>>>>
>>>>>>
>>>>>> Yes, if I ran integration test directly, NPE was thrown in
>>>>>> GroupMonitor class and no drools error were there.
>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Wed, Oct 7, 2015 at 1:05 AM, Akila Ravihansa Perera <
>>>>>>> raviha...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> It seems integration tests are failing intermittently regardless of
>>>>>>>> NPE's thrown from Drool files. I've added more debug logs to
>>>>>>>> troubleshoot
>>>>>>>> the issue. I've set the default log level in AS and CC packages to
>>>>>>>> DEBUG in
>>>>>>>> integration tests. This will help us to identify the root causes of
>>>>>>>> integration test failures in Jenkins build.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 6, 2015 at 12:38 PM, Thanuja Uruththirakodeeswaran <
>>>>>>>> thanu...@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Akila,
>>>>>>>>>
>>>>>>>>> I'll check this and let you know.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Tue, Oct 6, 2015 at 12:31 PM, Akila Ravihansa Perera <
>>>>>>>>> raviha...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Thanuja,
>>>>>>>>>>
>>>>>>>>>> I noticed that integration tests are failing in stratos-4.1.x
>>>>>>>>>> branch due to an issue in Drools files. It seems there have been
>>>>>>>>>> couple of
>>>>>>>>>> modifications done for Drools files with new metering dashboard
>>>>>>>>>> feature.
>>>>>>>>>> Perhaps it is incompatible with autoscaler APIs. Could you look into
>>>>>>>>>> that
>>>>>>>>>> please?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Akila Ravihansa Perera
>>>>>>>>>> WSO2 Inc.; http://wso2.com/
>>>>>>>>>>
>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanuja Uruththirakodeeswaran
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc.;http://wso2.com
>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>
>>>>>>>>> mobile: +94 774363167
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Akila Ravihansa Perera
>>>>>>>> WSO2 Inc.; http://wso2.com/
>>>>>>>>
>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Akila Ravihansa Perera
>>>>>>> WSO2 Inc.; http://wso2.com/
>>>>>>>
>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanuja Uruththirakodeeswaran
>>>>>> Software Engineer
>>>>>> WSO2 Inc.;http://wso2.com
>>>>>> lean.enterprise.middleware
>>>>>>
>>>>>> mobile: +94 774363167
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Senior Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Reka Thirunavukkarasu
>>>> Senior Software Engineer,
>>>> WSO2, Inc.:http://wso2.com,
>>>> Mobile: +94776442007
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Senior Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>>
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.; http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>
--
Thanuja Uruththirakodeeswaran
Software Engineer
WSO2 Inc.;http://wso2.com
lean.enterprise.middleware
mobile: +94 774363167
