Hi All, I did several builds with latest changes. Each local build triggered successfully with no test failures. But there are RuntimeExceptions. I tried following scenarios 1. Reverted metering service changes as in commit [1] - build success 2. Build stratos 4.1.x branch with latest commit [2] - build success 3. Build stratos 4.1.x branch without latest commit [2] - tests failed
For the 3rd scenario, I got test failures as below and I've attached part of test log here: Tests run: 21, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 1,499.716 sec <<< FAILURE! - in TestSuite testTerminationBehavior(org.apache.stratos.integration.tests.application.GroupStartupOrderTestCase) Time elapsed: 312.479 sec <<< FAILURE! java.lang.AssertionError: at org.testng.AssertJUnit.fail(AssertJUnit.java:59) at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:33) at org.apache.stratos.integration.tests.application.GroupStartupOrderTestCase.assertCreationOfNodes(GroupStartupOrderTestCase.java:339) at org.apache.stratos.integration.tests.application.GroupStartupOrderTestCase.testTerminationBehavior(GroupStartupOrderTestCase.java:149) testApplication(org.apache.stratos.integration.tests.application.SampleApplicationStartupTestCase) Time elapsed: 0.351 sec <<< FAILURE! java.lang.AssertionError: at org.testng.AssertJUnit.fail(AssertJUnit.java:59) at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) at org.testng.AssertJUnit.assertFalse(AssertJUnit.java:41) at org.testng.AssertJUnit.assertFalse(AssertJUnit.java:49) at org.apache.stratos.integration.tests.application.SampleApplicationStartupTestCase.testApplication(SampleApplicationStartupTestCase.java:105) testApplicationList(org.apache.stratos.integration.tests.application.SampleApplicationsTestCase) Time elapsed: 1.032 sec <<< FAILURE! java.lang.AssertionError: at org.testng.AssertJUnit.fail(AssertJUnit.java:59) at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) at org.testng.AssertJUnit.assertFalse(AssertJUnit.java:41) at org.testng.AssertJUnit.assertFalse(AssertJUnit.java:49) at org.apache.stratos.integration.tests.application.SampleApplicationsTestCase.testApplicationList(SampleApplicationsTestCase.java:350) testApplicationBusting(org.apache.stratos.integration.tests.application.ApplicationBurstingTestCase) Time elapsed: 32.926 sec <<< FAILURE! java.lang.AssertionError: at org.testng.AssertJUnit.fail(AssertJUnit.java:59) at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) at org.testng.AssertJUnit.assertFalse(AssertJUnit.java:41) at org.testng.AssertJUnit.assertFalse(AssertJUnit.java:49) at org.apache.stratos.integration.tests.application.ApplicationBurstingTestCase.testApplicationBusting(ApplicationBurstingTestCase.java:138) Results : Failed tests: ApplicationBurstingTestCase.testApplicationBusting:138 GroupStartupOrderTestCase.testTerminationBehavior:149->assertCreationOfNodes:339 SampleApplicationStartupTestCase.testApplication:105 SampleApplicationsTestCase.testApplicationList:350 Tests run: 21, Failures: 4, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Stratos - Integration Aggregation Module ... SUCCESS [1.360s] [INFO] Apache Stratos - Integration Common Module ........ SUCCESS [6.356s] [INFO] Apache Stratos - Integration Tests ................ FAILURE [25:05.695s] [INFO] ------------------------------------------------------------------------ I continuously ran more integration test on above scenarios and if I get any different result, I'll update in this thread. [1]. https://github.com/Thanu/stratos/commit/631d1a1cc71ade793a6a6c59edabb34bd1594747 [2]. https://github.com/apache/stratos/commit/4bc27f29cb84d86fdaac554ca5a426cb0eb3776f Thanks. On Sun, Oct 11, 2015 at 6:29 PM, Gayan Gunarathne <gay...@wso2.com> wrote: > Seems even application is terminated, some of the group monitors are still > running. Mostly in the nested group scenario. Those group monitors looking > for the application monitor that didn't exist at that time.So I think we > need to recheck the logic of group monitor termination logic in the > application un-deployment. Will check further on this. > > 2015-10-11 18:15:07 ERROR > {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - > Exception in thread "pool-39-thread-36" java.lang.NullPointerException > 2015-10-11 18:15:07 ERROR > {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at > org.apache.stratos.autoscaler.monitor.component.GroupMonitor$1.run(GroupMonitor.java:163) > 2015-10-11 18:15:07 ERROR > {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 2015-10-11 18:15:07 ERROR > {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 2015-10-11 18:15:07 ERROR > {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:73 - at > java.lang.Thread.run(Thread.java:744) > 2015-10-11 18:15:07 INFO > {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - > [2015-10-11 18:15:07,285] DEBUG > {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group > monitor is running: [group] my-group6-group-startup-order-test > > Thanks, > Gayan > > On Sat, Oct 10, 2015 at 3:35 PM, Akila Ravihansa Perera < > raviha...@wso2.com> wrote: > >> This is occurring frequently even in local build. I think this is a >> blocker for 4.1.4 release. >> >> On Fri, Oct 9, 2015 at 9:20 PM, Imesh Gunaratne <im...@apache.org> wrote: >> >>> Thanks Reka! It is important to identify the root cause of this issue. >>> I'm also looking in. >>> >>> On Thu, Oct 8, 2015 at 6:13 PM, Reka Thirunavukkarasu <r...@wso2.com> >>> wrote: >>> >>>> Hi >>>> >>>> After going through the logs of integration, please find the break down >>>> as below. For my-group6-group-tom2-group-startup-order-test, it seems >>>> that the Adder got triggered twice or somehow monitor creation got >>>> triggered twice which cause the memory to inconsistent state. Hence the >>>> particular GroupMonitor got removed somehow. So that the application >>>> hierarchy was incomplete and application was unable to become active. I'm >>>> still investigating on it to find out the root cause. >>>> >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,504] INFO >>>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >>>> Monitor scheduled: [type] cluster [component] >>>> group-startup-order-test.my-group-tom2-tomcat2-group-startup-order-test.tomcat2-group-startup-order-test.domain >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,504] DEBUG >>>> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - >>>> ClusterMonitor task interval set to : [application-id] null [cluster] >>>> group-startup-order-test.my-group7-tomcat-group-startup-order-test.tomcat-group-startup-order-test.domain >>>> [monitor-interval] 90000 >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,504] INFO >>>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >>>> Starting monitor: [type] cluster [component] >>>> group-startup-order-test.my-group-tom2-tomcat2-group-startup-order-test.tomcat2-group-startup-order-test.domain >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,504] ERROR >>>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >>>> An >>>> error occurred while starting monitor: [type] group [component] >>>> my-group6-group-tom2-group-startup-order-test >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> java.lang.NullPointerException >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> org.apache.stratos.autoscaler.monitor.Monitor.addInstance(Monitor.java:149) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> org.apache.stratos.autoscaler.monitor.component.GroupMonitor.createGroupInstanceAndAddToMonitor(GroupMonitor.java:816) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> org.apache.stratos.autoscaler.monitor.component.GroupMonitor.createInstanceAndStartDependencyAtStartup(GroupMonitor.java:898) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getGroupMonitor(MonitorFactory.java:161) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> org.apache.stratos.autoscaler.monitor.MonitorFactory.getMonitor(MonitorFactory.java:81) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor$MonitorAdder.run(ParentComponentMonitor.java:1001) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> at java.lang.Thread.run(Thread.java:745) >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,505] DEBUG >>>> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group >>>> monitor is running: [group] my-group6-group-tom2-group-startup-order-test >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,504] DEBUG >>>> {org.apache.stratos.autoscaler.context.partition.ParentLevelPartitionContext} >>>> - Instance expiry time is set to: 900000 >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,505] DEBUG >>>> {org.apache.stratos.autoscaler.context.partition.ParentLevelPartitionContext} >>>> - Instance obsolete expiry time is set to: 86400000 >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,505] DEBUG >>>> {org.apache.stratos.autoscaler.context.partition.ParentLevelPartitionContext} >>>> - Instance pending termination expiry time is set to: 1800000 >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,505] INFO >>>> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - >>>> [Partition] partition-1has been added for the [Group] >>>> my-group9-group-startup-order-test >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,505] INFO >>>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >>>> Monitor started successfully: [type] group [component] >>>> my-group6-group-tom2-group-startup-order-test [dependents] none >>>> [startup-time] 0 seconds >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,505] INFO >>>> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group >>>> [Instance context] group-startup-order-test-1 has been added to [Group] >>>> my-group9-group-startup-order-test >>>> 2015-10-08 13:49:52 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:49:52,505] INFO >>>> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Starting >>>> the dependencies for the [Group] my-group9-group-startup-order-test >>>> >>>> -------------------------------------- >>>> >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] DEBUG >>>> {org.apache.stratos.autoscaler.status.processor.group.GroupStatusProcessor} >>>> - Checking the status of cluster >>>> group-startup-order-test.my-group-tom2-tomcat3-group-startup-order-test.tomcat3-group-startup-order-test.domain >>>> instance status is: Active >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] DEBUG >>>> {org.apache.stratos.autoscaler.status.processor.group.GroupStatusProcessor} >>>> - Checking the status of cluster >>>> group-startup-order-test.my-group-tom2-tomcat2-group-startup-order-test.tomcat2-group-startup-order-test.domain >>>> instance status is: Active >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] DEBUG >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyManager} - >>>> Write lock released >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] INFO >>>> {org.apache.stratos.autoscaler.status.processor.group.GroupStatusActiveProcessor} >>>> - Sending group instance active for [group] >>>> my-group6-group-tom2-group-startup-order-test [instance] >>>> group-startup-order-test-1 >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] DEBUG >>>> {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - >>>> Handling group activation for the [group-id]: >>>> my-group6-group-tom2-group-startup-order-test in the [application-id] >>>> group-startup-order-test [instance] group-startup-order-test-1 >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] DEBUG >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyManager} - >>>> Write lock acquired >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] WARN >>>> {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - >>>> Group monitor cannot be found: [group-id] >>>> my-group6-group-tom2-group-startup-order-test [application-id] >>>> group-startup-order-test >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] INFO >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>>> Cluster activated adding status started for >>>> group-startup-order-test.my-group-tom2-tomcat3-group-startup-order-test.tomcat3-group-startup-order-test.domain >>>> 2015-10-08 13:51:19 INFO >>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 - >>>> [2015-10-08 13:51:19,100] DEBUG >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyManager} - >>>> Updating topology >>>> >>>> Thanks, >>>> Reka >>>> >>>> >>>> On Thu, Oct 8, 2015 at 4:47 PM, Reka Thirunavukkarasu <r...@wso2.com> >>>> wrote: >>>> >>>>> Hi Akila >>>>> >>>>> On Thu, Oct 8, 2015 at 2:51 PM, Akila Ravihansa Perera < >>>>> raviha...@wso2.com> wrote: >>>>> >>>>>> Hi Thanuja, >>>>>> >>>>>> I was referring to the build fail in [1]. Please see line starting >>>>>> from: >>>>>> >>>>>> 2015-10-02 07:28:32 INFO >>>>>> >>>>>> {org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader}:70 >>>>>> - >>>>>> [2015-10-02 07:28:32,834] ERROR >>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Cannot >>>>>> terminate >>>>>> instance >>>>>> >>>>>> I think this is a separate issue that occurs very rarely. As you >>>>>> noticed, NPE in GroupMonitor occurs when undeploying applications. The >>>>>> problem is group monitors are retrieving the status from other group >>>>>> monitor objects. >>>>>> >>>>>> @Reka: shall we change that logic to read the application status from >>>>>> topology or ApplicationHolder instead of reading from monitors? >>>>>> >>>>> >>>>> We can't read the Application status from Topology as Application will >>>>> not contain any status rather only application instance will contain the >>>>> status. By accessing application instance, we can't identify whether >>>>> undeployment or force undeployment is in-progress. Hence we will have to >>>>> maintain these flags to ApplicationMonitor to identity the undeployment or >>>>> force undeployment. All other status are actually referring from the >>>>> Topology. >>>>> >>>>> Thanks, >>>>> Reka >>>>> >>>>>> >>>>>> [1] >>>>>> https://builds.apache.org/job/Stratos-41x-Commit-Build/68/consoleFull >>>>>> >>>>>> Thanks. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Reka Thirunavukkarasu >>>>> Senior Software Engineer, >>>>> WSO2, Inc.:http://wso2.com, >>>>> Mobile: +94776442007 >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Reka Thirunavukkarasu >>>> Senior Software Engineer, >>>> WSO2, Inc.:http://wso2.com, >>>> Mobile: +94776442007 >>>> >>>> >>>> >>> >>> >>> -- >>> Imesh Gunaratne >>> >>> Senior Technical Lead, WSO2 >>> Committer & PMC Member, Apache Stratos >>> >> >> >> >> -- >> Akila Ravihansa Perera >> WSO2 Inc.; http://wso2.com/ >> >> Blog: http://ravihansa3000.blogspot.com >> > > > > -- > > Gayan Gunarathne > Technical Lead, WSO2 Inc. (http://wso2.com) > Committer & PMC Member, Apache Stratos > email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985> > > > -- Thanuja Uruththirakodeeswaran Software Engineer WSO2 Inc.;http://wso2.com lean.enterprise.middleware mobile: +94 774363167
