Kishan,
The problem is caused by these two methods. I hope everyone reads this because
I know some of us are new to writing unit tests. The problem is basically as I
described in a previous email. In here, both tests makes use of and shares of
the _networkACLItemDao and the mock is shared because the DAO is a singleton
and is injected. Now, note that testCreateACLItem() invokes the method
findByAclAndNumber() via the _networkAclMgr but it didn't specifically mock the
return value of the method in the test. If this test is ran before
testCreateACLItemDuplicateNumber(), then it will be successful because by
default mocks return null. However, testCreateACLItemDuplicateNumber() mocks
findByAclAndNumber() to return a value, so if testCreateACLItem() is ran after
testCreateACLItemDuplicateNumber(), the test will fail because a value is
returned and the condition inside _networkAclMgr will fire just as Daan found
it.
I was hoping maven prints out the order of the tests, but from Daan's log, it
only printed out the order of the bigger test suite so it's not helpful.
However, it's not too hard to find. Just go to the code with the error, check
the method that fails and go back to the test and see if your test mocked that
method and if it didn't, was it mocked by any other tests.
@Test
public void testCreateACLItem() throws Exception {
Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new VpcVO());
Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenReturn(acl);
Mockito.when(_networkAclMgr.createNetworkACLItem(Mockito.anyInt(),
Mockito.anyInt(), Mockito.anyString(), Mockito.anyList(), Mockito.anyInt(),
Mockito.anyInt(),
Mockito.any(NetworkACLItem.TrafficType.class),
Mockito.anyLong(), Mockito.anyString(), Mockito.anyInt())).thenReturn(new
NetworkACLItemVO());
assertNotNull(_aclService.createNetworkACLItem(createACLItemCmd));
}
@Test(expected = InvalidParameterValueException.class)
public void testCreateACLItemDuplicateNumber() throws Exception {
Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new VpcVO());
Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenReturn(acl);
Mockito.when(_networkACLItemDao.findByAclAndNumber(Mockito.anyLong(),
Mockito.anyInt())).thenReturn(new NetworkACLItemVO());
_aclService.createNetworkACLItem(createACLItemCmd);
}
> -----Original Message-----
> From: Kishan Kavala
> Sent: Wednesday, May 29, 2013 4:48 AM
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
>
> Daan,
> I mocked the method causing the failure (I couldn't replicate the error you
> are seeing though). Please check with latest master and let us know if you
> still see issues.
>
> ~kishan
> ________________________________________
> From: Daan Hoogland [dhoogl...@schubergphilis.com]
> Sent: Wednesday, May 29, 2013 1:49 PM
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
>
> H,
>
> I hope this is what Alex means the test programmer needs.
>
> The order that is reported is as follows (I included the server test run
> only):
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running com.cloud.alert.AlertControlsUnitTest
> log4j:WARN No appenders could be found for logger
> (com.cloud.alert.AlertControlsUnitTest).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.546 sec
> Running
> com.cloud.capacity.CapacityManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11 sec
> Running
> com.cloud.configuration.ConfigurationManagerTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.499 sec
> Running
> com.cloud.configuration.ValidateIpRangeTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec
> Running
> com.cloud.event.EventControlsUnitTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running
> com.cloud.keystore.KeystoreTest
> org.apache.cloudstack.api.response.UserVmResponse/null/{"id":"3","securit
> ygroup":[],"nic":[],"tags":[],"affinitygroup":[]}
> org.apache.cloudstack.api.response.AlertResponse/null/{"id":"100","descrip
> tion":"Hello"}
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.109 sec
> Running
> com.cloud.metadata.ResourceMetaDataManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.047 sec
> Running
> com.cloud.network.CreatePrivateNetworkTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec Running
> com.cloud.network.dao.NetworkDaoTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running
> com.cloud.network.DedicateGuestVlanRangesTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec
> Running
> com.cloud.network.firewall.FirewallManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec Running
> com.cloud.network.NetworkManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec Running
> com.cloud.network.NetworkModelTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running
> com.cloud.network.security.SecurityGroupQueueTest
> Num Vms= 50 Queue size = 50
> Num Vms= 20000 Queue size = 20000 time=827 ms Num Vms= 5000 Queue
> size = 5000 time=2544 ms Num Vms= 1 Queue size = 1 time=0 ms Num Vms=
> 1000000 Queue size = 1000000 time=1171 ms Num Vms= 1 Queue size = 1
> time=1140 ms Total jobs dequeued = 10, num queued=1010 queue current
> size=1000 Total jobs dequeued = 10, num queued=1008 queue current
> size=998 Total jobs dequeued = 1, num queued=1001 queue current
> size=1000 Total jobs dequeued = 10, num queued=1000 queue current
> size=990 Total jobs dequeued = 10, num queued=10 queue current size=0
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.745 sec
> Running
> com.cloud.network.UpdatePhysicalNetworkTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.312 sec
> Running
> com.cloud.resourcelimit.ResourceLimitManagerImplTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running
> com.cloud.vm.DeploymentPlanningManagerImplTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.713 sec
> Running
> com.cloud.vm.snapshot.VMSnapshotManagerTest
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.203 sec
> Running
> com.cloud.vm.UserVmManagerTest Tests run: 10, Failures: 0, Errors: 0,
> Skipped: 0, Time elapsed: 0.64 sec Running
> com.cloud.vm.VirtualMachineManagerImplTest
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.297 sec
> Running
> com.cloud.vpc.NetworkACLManagerTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.406 sec
> Running
> com.cloud.vpc.NetworkACLServiceTest
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.203 sec <<<
> FAILURE!
> Running com.cloud.vpc.Site2SiteVpnTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.491 sec
> Running
> com.cloud.vpc.VpcTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 0.235 sec Running org.apache.cloudstack.affinity.AffinityApiUnitTest
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.188 sec
> Running
> org.apache.cloudstack.lb.ApplicationLoadBalancerTest
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.249 sec
> Running org.apache.cloudstack.networkoffering.CreateNetworkOfferingTest
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.817 sec
> Running org.apache.cloudstack.privategw.AclOnPrivateGwTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11 sec
> Running
> org.apache.cloudstack.region.gslb.GlobalLoadBalancingRulesServiceImplTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec
> Running
> org.apache.cloudstack.region.RegionManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
>
> Results :
>
> Tests in error:
> testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest): ACL item with
> number 1 already exists in ACL: null
>
> -----Original Message-----
> From: Pranav Saxena [mailto:pranav.sax...@citrix.com]
> Sent: woensdag 29 mei 2013 5:58
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
>
> Thanks Alex for throwing some light on this . I have myself seen this when
> one of the intenalLb tests were failing on my machine and few others but
> compiling successfully on Alena's machine but that time I also didn't know
> how this could have happened and I ended up reporting this to her and then
> after probably some amendments , unit tests compiled on all the machines.
>
> -----Original Message-----
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: Wednesday, May 29, 2013 12:05 AM
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
>
> Unit tests are ran in random order because each unit test should be self-
> sufficient. I'm saying I've seen cases where unit tests aren't written to be
> self-sufficient and that's why an unit test fails sometimes and passes
> sometimes.
>
> In those cases, we should fix the unit test and not the order. A common
> problem is sharing the mocked classes. For example, unit test A tests the set
> of a mock object and unit test B tests if the field is null but they shared
> the
> mock instance. In this case, unit test B passes if it is ran before unit
> test A but
> fails if it was ran after.
>
> But often times the randomness of how it's ran is actually pseudo-random so
> it works consistently on someone's laptop but as soon as it gets ran on
> another machine, the order changes and it fails. So when you see the
> problem such as this, you should always let the unit test writer know the
> order in which the unit tests were executed so that the unit test writer can
> try it.
>
> --Alex
>
> > -----Original Message-----
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: Monday, May 27, 2013 9:23 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > And if not how do I manipulate the order of tests?
> >
> > -----Original Message-----
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: maandag 27 mei 2013 17:02
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > Alex,
> >
> > Can this change out of the box? I had the problem on a clean checkout
> > of master.
> >
> > -----Original Message-----
> > From: Alex Huang [mailto:alex.hu...@citrix.com]
> > Sent: maandag 27 mei 2013 15:55
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > Daan,
> >
> > The other possibility is that the test is affected by another test so
> > that it breaks when the tests are in a certain order. Have you
> > noticed what's the order of the tests ran when it breaks?
> >
> > --Alex
> >
> > > -----Original Message-----
> > > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > > Sent: Monday, May 27, 2013 2:37 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: master build breaks at
> > > com.cloud.vpc.NetworkACLServiceTest
> > >
> > > Being in the habit of replying on my own mails as I seems to be on
> > > this list;
> > >
> > > This seems to be related to eclipse refresh/build code. Does anybody
> > > have a solution to it? The problem is not consistently present but
> > > all the more irritating.
> > >
> > > Regards,
> > >
> > > -----Original Message-----
> > > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > > Sent: maandag 27 mei 2013 7:49
> > > To: dev@cloudstack.apache.org
> > > Subject: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> > >
> > > LS,
> > >
> > > Lately I've been getting,
> > > "com.cloud.exception.InvalidParameterValueException: ACL item with
> > > number 1 already exists in ACL: null", while building cloudstack. I
> > > first thought it was my own hack so I don't know when it was
> > > introduced (it is in a clean master that it happens). The full
> > > surfire report
> > follows:
> > >
> > > --------------------------------------------------------------------
> > > --
> > > --------- Test set: com.cloud.vpc.NetworkACLServiceTest
> > > --------------------------------------------------------------------
> > > --
> > > --------- Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time
> > > elapsed: 0.219 sec <<< FAILURE!
> > > testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest) Time
> elapsed:
> > > 0.016 sec <<< ERROR!
> > > com.cloud.exception.InvalidParameterValueException: ACL item with
> > > number 1 already exists in ACL: null
> > > at
> > >
> >
> com.cloud.network.vpc.NetworkACLServiceImpl.createNetworkACLItem(Ne
> > > tworkACLServiceImpl.java:270)
> > > at
> > >
> >
> com.cloud.vpc.NetworkACLServiceTest.testCreateACLItem(NetworkACLServ
> > > iceTest.java:141)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > at
> > >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework
> > > Method.java:45)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCa
> > > ll
> > > able.jav
> > > a:15)
> > > at
> > >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(Framework
> > > Method.java:42)
> > > at
> > >
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMeth
> > > od.java:20)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores
> > > .j
> > > ava
> > > :28)
> > > at
> > >
> >
> org.springframework.test.context.junit4.statements.RunBeforeTestMethod
> > > Callbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
> > > at
> > >
> >
> org.springframework.test.context.junit4.statements.RunAfterTestMethodC
> > > allbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
> > > at
> > > org.springframework.test.context.junit4.statements.SpringRepeat.eval
> > > ua
> > > te(
> > > SpringRepeat.java:72)
> > > at
> > > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runC
> > > hi
> > > ld(S
> > > pringJUnit4ClassRunner.java:231)
> > > at
> > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRu
> > > nn
> > > er.j
> > > ava:47)
> > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> > > at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> > > at
> > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> > > at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> > > at
> > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> > > at
> > > org.springframework.test.context.junit4.statements.RunBeforeTestClas
> > > sC
> > > all
> > > backs.evaluate(RunBeforeTestClassCallbacks.java:61)
> > > at
> > > org.springframework.test.context.junit4.statements.RunAfterTestClass
> > > Ca
> > > llb
> > > acks.evaluate(RunAfterTestClassCallbacks.java:71)
> > > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> > > at
> > > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(
> > > Sp
> > > ring
> > > JUnit4ClassRunner.java:174)
> > > at
> > > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provid
> > > er
> > > .jav
> > > a:236)
> > > at
> > > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit
> > > 4P
> > > rov
> > > ider.java:134)
> > > at
> > > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.
> > > java
> > > :113)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > at
> > > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray
> > > (R
> > > efl
> > > ectionUtils.java:189)
> > > at
> > > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invok
> > > e(
> > > Pr
> > > oviderFactory.java:165)
> > > at
> > > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(Prov
> > > id
> > > er
> > > Factory.java:85)
> > > at
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Forke
> > > d
> > > Booter.java:103)
> > > at
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:7
> > > 4)
