Hi All, NPE is not thrown by drool files. I reverted the changes in drool file which we did for metering service locally and ran the integration test. I'm getting NPE in same place at GroupMonitor (line - 163) as follows:
2015-10-08 13:42:28 INFO {org.apache.stratos.integration.tests.application.SampleApplicationStartupTestCase}:150 - Waiting for application status to become INACTIVE 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,758] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,759] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,759] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,761] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - Member not found in the Topology: [member] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,761] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,762] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,762] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - Member not found in the Topology: [member] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,762] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,763] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,763] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - Member not found in the Topology: [member] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,763] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,763] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - A cluster monitor is not found in autoscaler context [cluster] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,764] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - Member not found in the Topology: [member] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,764] DEBUG {org.apache.stratos.autoscaler.event.receiver.health.AutoscalerHealthStatEventReceiver} - Member not found in the Topology: [member] group-startup-order-test.my-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain5978c03c-0a2e-4023-8cd9-e7657c22c273 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,828] DEBUG {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group monitor is running: [group] my-group6-group-startup-order-test 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - Exception in thread "pool-39-thread-93" java.lang.NullPointerException 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.lang.Thread.run(Thread.java:745) 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,852] DEBUG {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group monitor is running: [group] my-group8-group-startup-order-test 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - Exception in thread "pool-39-thread-99" java.lang.NullPointerException 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.lang.Thread.run(Thread.java:745) 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,889] DEBUG {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group monitor is running: [group] my-group9-group-startup-order-test 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - Exception in thread "pool-39-thread-100" java.lang.NullPointerException 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.lang.Thread.run(Thread.java:745) 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,918] DEBUG {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group monitor is running: [group] my-group6-group-tom2-group-startup-order-test 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - Exception in thread "pool-39-thread-97" java.lang.NullPointerException 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.lang.Thread.run(Thread.java:745) 2015-10-08 13:42:35 INFO {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - [2015-10-08 13:42:35,941] DEBUG {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group monitor is running: [group] my-group7-group-startup-order-test 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - Exception in thread "pool-39-thread-94" java.lang.NullPointerException 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-10-08 13:42:35 ERROR {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at java.lang.Thread.run(Thread.java:745) So this NPE is thrown because of drool file changes and we need to debug ApplicationMonitor and ClusterMonitor classes when undeploying application. Thanks. On Wed, Oct 7, 2015 at 10:07 PM, Reka Thirunavukkarasu <r...@wso2.com> wrote: > Hi Akila, > > On Wed, Oct 7, 2015 at 7:08 PM, Akila Ravihansa Perera <raviha...@wso2.com > > wrote: > >> Hi Reka, >> >> Could you please explain the fix that you are planning to do? >> > I have added only Null check to the places where Application Monitor was > invoked. It is required as in the force undeployment, we are removing all > the monitors without considering the monitor hierarchy. > >> >> While going through the GroupMonitor class, few issues were found related >> to the logic in handling Thread Executors. The problem is when an >> application is undeployed, execution jobs are never stopped from Thread >> Executor service. The jobs will keep running even after application is been >> undeployed. Have we addressed this case in Thread Executors? >> > > This is being handled in the ApplicationBuilder line 427, 302 and 306. We > were stopping the monitor executor service using below code: > > /** > * This will stop the scheduler which is running for the monitor > */ > protected void stopScheduler() { > schedulerFuture.cancel(true); > } > > If it is not working properly, then might need to check further on this. > When stopScheduler is invoked, if the thread is started, then we don't have > control over it. It will get executed for that iteration and then only it > will get stopped. I have tried a simple sample with debug logs for monitor. > I couldn't see the monitor thread after undeployment. Has this particular > issue occurred during force undeployment? > >> >> Also monitoringRunnable implementation does not catch any possible >> exceptions. This is the reason why we are seeing NPE's in the console >> output while running integration tests but not seeing them in >> wso2carbon.log. I think we better catch a generic exception in >> monitoringRunnable run method. >> > > +1 for catching generic exception in monitoringRunnable. > > Thanks, > Reka > >> >> Thanks. >> >> On Wed, Oct 7, 2015 at 6:34 PM, Reka Thirunavukkarasu <r...@wso2.com> >> wrote: >> >>> Hi Imesh, >>> >>> On Wed, Oct 7, 2015 at 7:27 AM, Reka Thirunavukkarasu <r...@wso2.com> >>> wrote: >>> >>>> Hi Imesh, >>>> >>>> On Wed, Oct 7, 2015 at 5:46 AM, Imesh Gunaratne <im...@apache.org> >>>> wrote: >>>> >>>>> GroupMonitor: >>>>> >>>>> ApplicationMonitor applicationMonitor = >>>>> AutoscalerContext.getInstance(). >>>>> getAppMonitor(appId); >>>>> >>>>> [161] //When the application is getting un-deployed, need to avoid >>>>> [162] // checking the minimum count sanctification >>>>> [163] if (!applicationMonitor.isTerminating()) { >>>>> >>>>> In GroupMonitor class the line number 163 is shown above, may be due >>>>> to some reason application monitor is not found when executing the group >>>>> monitor. >>>>> >>>>> @Reka: Do you have any thoughts on this? >>>>> >>>> >>>> Yah..It could happen when force undeployment is invoked. I will check >>>> on this.. >>>> >>> >>> I have fixed the possible places which could cause NPE by scanning >>> through the code. I will do a quick round of testing and push the changes >>> to Stratos-4.1.x branch. >>> >>> Thanks, >>> Reka >>> >>>> >>>> Thanks, >>>> Reka >>>> >>>>> >>>>> Thanks >>>>> >>>>> On Wed, Oct 7, 2015 at 1:20 AM, Thanuja Uruththirakodeeswaran < >>>>> thanu...@wso2.com> wrote: >>>>> >>>>>> Hi Akila, >>>>>> >>>>>> >>>>>> On Wed, Oct 7, 2015 at 1:09 AM, Akila Ravihansa Perera < >>>>>> raviha...@wso2.com> wrote: >>>>>> >>>>>>> Hi Thanuja, >>>>>>> >>>>>>> Thanks for the update. >>>>>>> >>>>>>> In both of those cases, did you observe any build failures? If so, >>>>>>> what are the tests that failed? >>>>>>> >>>>>> >>>>>> For the complete stratos build, following test cases are failing with >>>>>> NPE in ClusterMonitor as shown above: >>>>>> Failed tests: >>>>>> ApplicationBurstingTestCase.testApplicationBusting:138 >>>>>> GroupStartupOrderTestCase.testTerminationBehavior:194 Application >>>>>> status did not change to Active: [application-id] >>>>>> group-startup-order-test >>>>>> expected:<Active> but was:<Inactive> >>>>>> SampleApplicationStartupTestCase.testApplication:103 >>>>>> SampleApplicationsTestCase.testApplication:57 » Runtime >>>>>> SampleApplicationsTestCase.testApplicationList:252 » Runtime >>>>>> >>>>>> SingleClusterScalingTestCase.testDeployApplication:117->assertClusterWithScaleDown:260 >>>>>> » ThreadTimeout >>>>>> >>>>>> When running integration test only, there is no test failure and I'm >>>>>> getting NPE in GroupMonitor class. >>>>>> >>>>>>> >>>>>>> I noticed the same behavior as you have explained. When I check the >>>>>>> log file >>>>>>> in >>>>>>> /test-integration/target/<carbon-home>/repository/logs/wso2carbon.log, I >>>>>>> didn't see any NPE's related to Drool files. Only one NPE is thrown from >>>>>>> GroupMonitor class. Can you check whether it is the same for you? >>>>>>> >>>>>> >>>>>> Yes, if I ran integration test directly, NPE was thrown in >>>>>> GroupMonitor class and no drools error were there. >>>>>> >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> On Wed, Oct 7, 2015 at 1:05 AM, Akila Ravihansa Perera < >>>>>>> raviha...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> It seems integration tests are failing intermittently regardless of >>>>>>>> NPE's thrown from Drool files. I've added more debug logs to >>>>>>>> troubleshoot >>>>>>>> the issue. I've set the default log level in AS and CC packages to >>>>>>>> DEBUG in >>>>>>>> integration tests. This will help us to identify the root causes of >>>>>>>> integration test failures in Jenkins build. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 6, 2015 at 12:38 PM, Thanuja Uruththirakodeeswaran < >>>>>>>> thanu...@wso2.com> wrote: >>>>>>>> >>>>>>>>> Hi Akila, >>>>>>>>> >>>>>>>>> I'll check this and let you know. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> On Tue, Oct 6, 2015 at 12:31 PM, Akila Ravihansa Perera < >>>>>>>>> raviha...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Thanuja, >>>>>>>>>> >>>>>>>>>> I noticed that integration tests are failing in stratos-4.1.x >>>>>>>>>> branch due to an issue in Drools files. It seems there have been >>>>>>>>>> couple of >>>>>>>>>> modifications done for Drools files with new metering dashboard >>>>>>>>>> feature. >>>>>>>>>> Perhaps it is incompatible with autoscaler APIs. Could you look into >>>>>>>>>> that >>>>>>>>>> please? >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>> >>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanuja Uruththirakodeeswaran >>>>>>>>> Software Engineer >>>>>>>>> WSO2 Inc.;http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> >>>>>>>>> mobile: +94 774363167 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Akila Ravihansa Perera >>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>> >>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Akila Ravihansa Perera >>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>> >>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanuja Uruththirakodeeswaran >>>>>> Software Engineer >>>>>> WSO2 Inc.;http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 774363167 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Imesh Gunaratne >>>>> >>>>> Senior Technical Lead, WSO2 >>>>> Committer & PMC Member, Apache Stratos >>>>> >>>> >>>> >>>> >>>> -- >>>> Reka Thirunavukkarasu >>>> Senior Software Engineer, >>>> WSO2, Inc.:http://wso2.com, >>>> Mobile: +94776442007 >>>> >>>> >>>> >>> >>> >>> -- >>> Reka Thirunavukkarasu >>> Senior Software Engineer, >>> WSO2, Inc.:http://wso2.com, >>> Mobile: +94776442007 >>> >>> >>> >> >> >> -- >> Akila Ravihansa Perera >> WSO2 Inc.; http://wso2.com/ >> >> Blog: http://ravihansa3000.blogspot.com >> > > > > -- > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > Mobile: +94776442007 > > > -- Thanuja Uruththirakodeeswaran Software Engineer WSO2 Inc.;http://wso2.com lean.enterprise.middleware mobile: +94 774363167