H, I hope this is what Alex means the test programmer needs.
The order that is reported is as follows (I included the server test run only): ------------------------------------------------------- T E S T S ------------------------------------------------------- Running com.cloud.alert.AlertControlsUnitTest log4j:WARN No appenders could be found for logger (com.cloud.alert.AlertControlsUnitTest). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.546 sec Running com.cloud.capacity.CapacityManagerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11 sec Running com.cloud.configuration.ConfigurationManagerTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.499 sec Running com.cloud.configuration.ValidateIpRangeTest Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec Running com.cloud.event.EventControlsUnitTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running com.cloud.keystore.KeystoreTest org.apache.cloudstack.api.response.UserVmResponse/null/{"id":"3","securitygroup":[],"nic":[],"tags":[],"affinitygroup":[]} org.apache.cloudstack.api.response.AlertResponse/null/{"id":"100","description":"Hello"} Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.109 sec Running com.cloud.metadata.ResourceMetaDataManagerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.047 sec Running com.cloud.network.CreatePrivateNetworkTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec Running com.cloud.network.dao.NetworkDaoTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running com.cloud.network.DedicateGuestVlanRangesTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec Running com.cloud.network.firewall.FirewallManagerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec Running com.cloud.network.NetworkManagerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec Running com.cloud.network.NetworkModelTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running com.cloud.network.security.SecurityGroupQueueTest Num Vms= 50 Queue size = 50 Num Vms= 20000 Queue size = 20000 time=827 ms Num Vms= 5000 Queue size = 5000 time=2544 ms Num Vms= 1 Queue size = 1 time=0 ms Num Vms= 1000000 Queue size = 1000000 time=1171 ms Num Vms= 1 Queue size = 1 time=1140 ms Total jobs dequeued = 10, num queued=1010 queue current size=1000 Total jobs dequeued = 10, num queued=1008 queue current size=998 Total jobs dequeued = 1, num queued=1001 queue current size=1000 Total jobs dequeued = 10, num queued=1000 queue current size=990 Total jobs dequeued = 10, num queued=10 queue current size=0 Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.745 sec Running com.cloud.network.UpdatePhysicalNetworkTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.312 sec Running com.cloud.resourcelimit.ResourceLimitManagerImplTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running com.cloud.vm.DeploymentPlanningManagerImplTest Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.713 sec Running com.cloud.vm.snapshot.VMSnapshotManagerTest Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.203 sec Running com.cloud.vm.UserVmManagerTest Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.64 sec Running com.cloud.vm.VirtualMachineManagerImplTest Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.297 sec Running com.cloud.vpc.NetworkACLManagerTest Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.406 sec Running com.cloud.vpc.NetworkACLServiceTest Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.203 sec <<< FAILURE! Running com.cloud.vpc.Site2SiteVpnTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.491 sec Running com.cloud.vpc.VpcTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.235 sec Running org.apache.cloudstack.affinity.AffinityApiUnitTest Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.188 sec Running org.apache.cloudstack.lb.ApplicationLoadBalancerTest Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.249 sec Running org.apache.cloudstack.networkoffering.CreateNetworkOfferingTest Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.817 sec Running org.apache.cloudstack.privategw.AclOnPrivateGwTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11 sec Running org.apache.cloudstack.region.gslb.GlobalLoadBalancingRulesServiceImplTest Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec Running org.apache.cloudstack.region.RegionManagerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Results : Tests in error: testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest): ACL item with number 1 already exists in ACL: null -----Original Message----- From: Pranav Saxena [mailto:pranav.sax...@citrix.com] Sent: woensdag 29 mei 2013 5:58 To: dev@cloudstack.apache.org Cc: Alex Huang Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest Thanks Alex for throwing some light on this . I have myself seen this when one of the intenalLb tests were failing on my machine and few others but compiling successfully on Alena's machine but that time I also didn't know how this could have happened and I ended up reporting this to her and then after probably some amendments , unit tests compiled on all the machines. -----Original Message----- From: Alex Huang [mailto:alex.hu...@citrix.com] Sent: Wednesday, May 29, 2013 12:05 AM To: dev@cloudstack.apache.org Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest Unit tests are ran in random order because each unit test should be self-sufficient. I'm saying I've seen cases where unit tests aren't written to be self-sufficient and that's why an unit test fails sometimes and passes sometimes. In those cases, we should fix the unit test and not the order. A common problem is sharing the mocked classes. For example, unit test A tests the set of a mock object and unit test B tests if the field is null but they shared the mock instance. In this case, unit test B passes if it is ran before unit test A but fails if it was ran after. But often times the randomness of how it's ran is actually pseudo-random so it works consistently on someone's laptop but as soon as it gets ran on another machine, the order changes and it fails. So when you see the problem such as this, you should always let the unit test writer know the order in which the unit tests were executed so that the unit test writer can try it. --Alex > -----Original Message----- > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com] > Sent: Monday, May 27, 2013 9:23 AM > To: dev@cloudstack.apache.org > Subject: RE: master build breaks at > com.cloud.vpc.NetworkACLServiceTest > > And if not how do I manipulate the order of tests? > > -----Original Message----- > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com] > Sent: maandag 27 mei 2013 17:02 > To: dev@cloudstack.apache.org > Subject: RE: master build breaks at > com.cloud.vpc.NetworkACLServiceTest > > Alex, > > Can this change out of the box? I had the problem on a clean checkout > of master. > > -----Original Message----- > From: Alex Huang [mailto:alex.hu...@citrix.com] > Sent: maandag 27 mei 2013 15:55 > To: dev@cloudstack.apache.org > Subject: RE: master build breaks at > com.cloud.vpc.NetworkACLServiceTest > > Daan, > > The other possibility is that the test is affected by another test so > that it breaks when the tests are in a certain order. Have you > noticed what's the order of the tests ran when it breaks? > > --Alex > > > -----Original Message----- > > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com] > > Sent: Monday, May 27, 2013 2:37 AM > > To: dev@cloudstack.apache.org > > Subject: RE: master build breaks at > > com.cloud.vpc.NetworkACLServiceTest > > > > Being in the habit of replying on my own mails as I seems to be on > > this list; > > > > This seems to be related to eclipse refresh/build code. Does anybody > > have a solution to it? The problem is not consistently present but > > all the more irritating. > > > > Regards, > > > > -----Original Message----- > > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com] > > Sent: maandag 27 mei 2013 7:49 > > To: dev@cloudstack.apache.org > > Subject: master build breaks at com.cloud.vpc.NetworkACLServiceTest > > > > LS, > > > > Lately I've been getting, > > "com.cloud.exception.InvalidParameterValueException: ACL item with > > number 1 already exists in ACL: null", while building cloudstack. I > > first thought it was my own hack so I don't know when it was > > introduced (it is in a clean master that it happens). The full > > surfire report > follows: > > > > -------------------------------------------------------------------- > > -- > > --------- Test set: com.cloud.vpc.NetworkACLServiceTest > > -------------------------------------------------------------------- > > -- > > --------- Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time > > elapsed: 0.219 sec <<< FAILURE! > > testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest) Time elapsed: > > 0.016 sec <<< ERROR! > > com.cloud.exception.InvalidParameterValueException: ACL item with > > number 1 already exists in ACL: null > > at > > > com.cloud.network.vpc.NetworkACLServiceImpl.createNetworkACLItem(Ne > > tworkACLServiceImpl.java:270) > > at > > > com.cloud.vpc.NetworkACLServiceTest.testCreateACLItem(NetworkACLServ > > iceTest.java:141) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework > > Method.java:45) > > at > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCa > > ll > > able.jav > > a:15) > > at > > org.junit.runners.model.FrameworkMethod.invokeExplosively(Framework > > Method.java:42) > > at > > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMeth > > od.java:20) > > at > > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores > > .j > > ava > > :28) > > at > > > org.springframework.test.context.junit4.statements.RunBeforeTestMethod > > Callbacks.evaluate(RunBeforeTestMethodCallbacks.java:74) > > at > > > org.springframework.test.context.junit4.statements.RunAfterTestMethodC > > allbacks.evaluate(RunAfterTestMethodCallbacks.java:83) > > at > > org.springframework.test.context.junit4.statements.SpringRepeat.eval > > ua > > te( > > SpringRepeat.java:72) > > at > > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runC > > hi > > ld(S > > pringJUnit4ClassRunner.java:231) > > at > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRu > > nn > > er.j > > ava:47) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > > at > > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > > at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > > at > > org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > > at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > > at > > org.springframework.test.context.junit4.statements.RunBeforeTestClas > > sC > > all > > backs.evaluate(RunBeforeTestClassCallbacks.java:61) > > at > > org.springframework.test.context.junit4.statements.RunAfterTestClass > > Ca > > llb > > acks.evaluate(RunAfterTestClassCallbacks.java:71) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > > at > > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run( > > Sp > > ring > > JUnit4ClassRunner.java:174) > > at > > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provid > > er > > .jav > > a:236) > > at > > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit > > 4P > > rov > > ider.java:134) > > at > > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider. > > java > > :113) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray > > (R > > efl > > ectionUtils.java:189) > > at > > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invok > > e( > > Pr > > oviderFactory.java:165) > > at > > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(Prov > > id > > er > > Factory.java:85) > > at > > > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Forke > > d > > Booter.java:103) > > at > > > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:7 > > 4)
