Alex,
Thanks for the mockito lesson. I think a lot of us are now going to improve
their tests.
Can one make mockito forget the when-conditions? I would intuitively say
MockitoAnnotations.initMocks(this); would do this.
Alternative to the answer to that last question would be to follow the javadoc
for when:
//you can set different behavior for consecutive method calls.
//Last stubbing (e.g: thenReturn("foo")) determines the behavior of further
consecutive calls.
when(mock.someMethod("some arg"))
.thenThrow(new RuntimeException())
.thenReturn("foo");
//Alternative, shorter version for consecutive stubbing:
when(mock.someMethod("some arg"))
.thenReturn("one", "two");
//is the same as:
when(mock.someMethod("some arg"))
.thenReturn("one")
.thenReturn("two");
Regards,
Daan (not new to unit-tests sec but to mockito though) Hoogland
-----Original Message-----
From: Alex Huang [mailto:alex.hu...@citrix.com]
Sent: woensdag 29 mei 2013 16:34
To: Kishan Kavala; dev@cloudstack.apache.org
Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
Kishan,
The problem is caused by these two methods. I hope everyone reads this because
I know some of us are new to writing unit tests. The problem is basically as I
described in a previous email. In here, both tests makes use of and shares of
the _networkACLItemDao and the mock is shared because the DAO is a singleton
and is injected. Now, note that testCreateACLItem() invokes the method
findByAclAndNumber() via the _networkAclMgr but it didn't specifically mock the
return value of the method in the test. If this test is ran before
testCreateACLItemDuplicateNumber(), then it will be successful because by
default mocks return null. However, testCreateACLItemDuplicateNumber() mocks
findByAclAndNumber() to return a value, so if testCreateACLItem() is ran after
testCreateACLItemDuplicateNumber(), the test will fail because a value is
returned and the condition inside _networkAclMgr will fire just as Daan found
it.
I was hoping maven prints out the order of the tests, but from Daan's log, it
only printed out the order of the bigger test suite so it's not helpful.
However, it's not too hard to find. Just go to the code with the error, check
the method that fails and go back to the test and see if your test mocked that
method and if it didn't, was it mocked by any other tests.
@Test
public void testCreateACLItem() throws Exception {
Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new VpcVO());
Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenReturn(acl);
Mockito.when(_networkAclMgr.createNetworkACLItem(Mockito.anyInt(),
Mockito.anyInt(), Mockito.anyString(), Mockito.anyList(), Mockito.anyInt(),
Mockito.anyInt(),
Mockito.any(NetworkACLItem.TrafficType.class),
Mockito.anyLong(), Mockito.anyString(), Mockito.anyInt())).thenReturn(new
NetworkACLItemVO());
assertNotNull(_aclService.createNetworkACLItem(createACLItemCmd));
}
@Test(expected = InvalidParameterValueException.class)
public void testCreateACLItemDuplicateNumber() throws Exception {
Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new VpcVO());
Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenReturn(acl);
Mockito.when(_networkACLItemDao.findByAclAndNumber(Mockito.anyLong(),
Mockito.anyInt())).thenReturn(new NetworkACLItemVO());
_aclService.createNetworkACLItem(createACLItemCmd);
}
> -----Original Message-----
> From: Kishan Kavala
> Sent: Wednesday, May 29, 2013 4:48 AM
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at
> com.cloud.vpc.NetworkACLServiceTest
>
> Daan,
> I mocked the method causing the failure (I couldn't replicate the
> error you are seeing though). Please check with latest master and let
> us know if you still see issues.
>
> ~kishan
> ________________________________________
> From: Daan Hoogland [dhoogl...@schubergphilis.com]
> Sent: Wednesday, May 29, 2013 1:49 PM
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at
> com.cloud.vpc.NetworkACLServiceTest
>
> H,
>
> I hope this is what Alex means the test programmer needs.
>
> The order that is reported is as follows (I included the server test run
> only):
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running com.cloud.alert.AlertControlsUnitTest
> log4j:WARN No appenders could be found for logger
> (com.cloud.alert.AlertControlsUnitTest).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info.
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.546
> sec Running com.cloud.capacity.CapacityManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11
> sec Running com.cloud.configuration.ConfigurationManagerTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.499
> sec Running com.cloud.configuration.ValidateIpRangeTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031
> sec Running com.cloud.event.EventControlsUnitTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
> Running com.cloud.keystore.KeystoreTest
> org.apache.cloudstack.api.response.UserVmResponse/null/{"id":"3","secu
> rit ygroup":[],"nic":[],"tags":[],"affinitygroup":[]}
> org.apache.cloudstack.api.response.AlertResponse/null/{"id":"100","des
> crip
> tion":"Hello"}
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.109
> sec Running com.cloud.metadata.ResourceMetaDataManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.047
> sec Running com.cloud.network.CreatePrivateNetworkTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec
> Running com.cloud.network.dao.NetworkDaoTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
> Running com.cloud.network.DedicateGuestVlanRangesTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031
> sec Running com.cloud.network.firewall.FirewallManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec
> Running com.cloud.network.NetworkManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec
> Running com.cloud.network.NetworkModelTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
> Running com.cloud.network.security.SecurityGroupQueueTest
> Num Vms= 50 Queue size = 50
> Num Vms= 20000 Queue size = 20000 time=827 ms Num Vms= 5000 Queue size
> = 5000 time=2544 ms Num Vms= 1 Queue size = 1 time=0 ms Num Vms=
> 1000000 Queue size = 1000000 time=1171 ms Num Vms= 1 Queue size = 1
> time=1140 ms Total jobs dequeued = 10, num queued=1010 queue current
> size=1000 Total jobs dequeued = 10, num queued=1008 queue current
> size=998 Total jobs dequeued = 1, num queued=1001 queue current
> size=1000 Total jobs dequeued = 10, num queued=1000 queue current
> size=990 Total jobs dequeued = 10, num queued=10 queue current size=0
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.745
> sec Running com.cloud.network.UpdatePhysicalNetworkTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.312
> sec Running com.cloud.resourcelimit.ResourceLimitManagerImplTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
> Running com.cloud.vm.DeploymentPlanningManagerImplTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.713
> sec Running com.cloud.vm.snapshot.VMSnapshotManagerTest
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.203
> sec Running com.cloud.vm.UserVmManagerTest Tests run: 10, Failures: 0,
> Errors: 0,
> Skipped: 0, Time elapsed: 0.64 sec Running
> com.cloud.vm.VirtualMachineManagerImplTest
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.297
> sec Running com.cloud.vpc.NetworkACLManagerTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.406
> sec Running com.cloud.vpc.NetworkACLServiceTest
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.203
> sec <<< FAILURE!
> Running com.cloud.vpc.Site2SiteVpnTest Tests run: 1, Failures: 0,
> Errors: 0, Skipped: 0, Time elapsed: 5.491 sec Running
> com.cloud.vpc.VpcTest Tests run: 1, Failures: 0, Errors: 0, Skipped:
> 0, Time
> elapsed: 0.235 sec Running
> org.apache.cloudstack.affinity.AffinityApiUnitTest
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.188
> sec Running org.apache.cloudstack.lb.ApplicationLoadBalancerTest
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.249
> sec Running
> org.apache.cloudstack.networkoffering.CreateNetworkOfferingTest
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.817
> sec Running org.apache.cloudstack.privategw.AclOnPrivateGwTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11
> sec Running
> org.apache.cloudstack.region.gslb.GlobalLoadBalancingRulesServiceImplT
> est Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.032 sec Running org.apache.cloudstack.region.RegionManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
>
> Results :
>
> Tests in error:
> testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest): ACL item
> with number 1 already exists in ACL: null
>
> -----Original Message-----
> From: Pranav Saxena [mailto:pranav.sax...@citrix.com]
> Sent: woensdag 29 mei 2013 5:58
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at
> com.cloud.vpc.NetworkACLServiceTest
>
> Thanks Alex for throwing some light on this . I have myself seen this
> when one of the intenalLb tests were failing on my machine and few
> others but compiling successfully on Alena's machine but that time I
> also didn't know how this could have happened and I ended up reporting
> this to her and then after probably some amendments , unit tests compiled on
> all the machines.
>
> -----Original Message-----
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: Wednesday, May 29, 2013 12:05 AM
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at
> com.cloud.vpc.NetworkACLServiceTest
>
> Unit tests are ran in random order because each unit test should be
> self- sufficient. I'm saying I've seen cases where unit tests aren't
> written to be self-sufficient and that's why an unit test fails
> sometimes and passes sometimes.
>
> In those cases, we should fix the unit test and not the order. A
> common problem is sharing the mocked classes. For example, unit test
> A tests the set of a mock object and unit test B tests if the field is
> null but they shared the mock instance. In this case, unit test B
> passes if it is ran before unit test A but fails if it was ran after.
>
> But often times the randomness of how it's ran is actually
> pseudo-random so it works consistently on someone's laptop but as soon
> as it gets ran on another machine, the order changes and it fails. So
> when you see the problem such as this, you should always let the unit
> test writer know the order in which the unit tests were executed so
> that the unit test writer can try it.
>
> --Alex
>
> > -----Original Message-----
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: Monday, May 27, 2013 9:23 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > And if not how do I manipulate the order of tests?
> >
> > -----Original Message-----
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: maandag 27 mei 2013 17:02
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > Alex,
> >
> > Can this change out of the box? I had the problem on a clean
> > checkout of master.
> >
> > -----Original Message-----
> > From: Alex Huang [mailto:alex.hu...@citrix.com]
> > Sent: maandag 27 mei 2013 15:55
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > Daan,
> >
> > The other possibility is that the test is affected by another test
> > so that it breaks when the tests are in a certain order. Have you
> > noticed what's the order of the tests ran when it breaks?
> >
> > --Alex
> >
> > > -----Original Message-----
> > > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > > Sent: Monday, May 27, 2013 2:37 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: master build breaks at
> > > com.cloud.vpc.NetworkACLServiceTest
> > >
> > > Being in the habit of replying on my own mails as I seems to be on
> > > this list;
> > >
> > > This seems to be related to eclipse refresh/build code. Does
> > > anybody have a solution to it? The problem is not consistently
> > > present but all the more irritating.
> > >
> > > Regards,
> > >
> > > -----Original Message-----
> > > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > > Sent: maandag 27 mei 2013 7:49
> > > To: dev@cloudstack.apache.org
> > > Subject: master build breaks at
> > > com.cloud.vpc.NetworkACLServiceTest
> > >
> > > LS,
> > >
> > > Lately I've been getting,
> > > "com.cloud.exception.InvalidParameterValueException: ACL item with
> > > number 1 already exists in ACL: null", while building cloudstack.
> > > I first thought it was my own hack so I don't know when it was
> > > introduced (it is in a clean master that it happens). The full
> > > surfire report
> > follows:
> > >
> > > ------------------------------------------------------------------
> > > --
> > > --
> > > --------- Test set: com.cloud.vpc.NetworkACLServiceTest
> > > ------------------------------------------------------------------
> > > --
> > > --
> > > --------- Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time
> > > elapsed: 0.219 sec <<< FAILURE!
> > > testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest) Time
> elapsed:
> > > 0.016 sec <<< ERROR!
> > > com.cloud.exception.InvalidParameterValueException: ACL item with
> > > number 1 already exists in ACL: null
> > > at
> > >
> >
> com.cloud.network.vpc.NetworkACLServiceImpl.createNetworkACLItem(Ne
> > > tworkACLServiceImpl.java:270)
> > > at
> > >
> >
> com.cloud.vpc.NetworkACLServiceTest.testCreateACLItem(NetworkACLServ
> > > iceTest.java:141)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > at
> > >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework
> > > Method.java:45)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(Reflective
> > > Ca
> > > ll
> > > able.jav
> > > a:15)
> > > at
> > >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(Framework
> > > Method.java:42)
> > > at
> > >
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMeth
> > > od.java:20)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefor
> > > es
> > > .j
> > > ava
> > > :28)
> > > at
> > >
> >
> org.springframework.test.context.junit4.statements.RunBeforeTestMethod
> > > Callbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
> > > at
> > >
> >
> org.springframework.test.context.junit4.statements.RunAfterTestMethodC
> > > allbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
> > > at
> > > org.springframework.test.context.junit4.statements.SpringRepeat.ev
> > > al
> > > ua
> > > te(
> > > SpringRepeat.java:72)
> > > at
> > > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.ru
> > > nC
> > > hi
> > > ld(S
> > > pringJUnit4ClassRunner.java:231)
> > > at
> > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4Class
> > > Ru
> > > nn
> > > er.j
> > > ava:47)
> > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> > > at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> > > at
> > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> > > at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> > > at
> > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> > > at
> > > org.springframework.test.context.junit4.statements.RunBeforeTestCl
> > > as
> > > sC
> > > all
> > > backs.evaluate(RunBeforeTestClassCallbacks.java:61)
> > > at
> > > org.springframework.test.context.junit4.statements.RunAfterTestCla
> > > ss
> > > Ca
> > > llb
> > > acks.evaluate(RunAfterTestClassCallbacks.java:71)
> > > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> > > at
> > > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.ru
> > > n(
> > > Sp
> > > ring
> > > JUnit4ClassRunner.java:174)
> > > at
> > > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Prov
> > > id
> > > er
> > > .jav
> > > a:236)
> > > at
> > > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUn
> > > it
> > > 4P
> > > rov
> > > ider.java:134)
> > > at
> > > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.
> > > java
> > > :113)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > at
> > > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArr
> > > ay
> > > (R
> > > efl
> > > ectionUtils.java:189)
> > > at
> > > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.inv
> > > ok
> > > e(
> > > Pr
> > > oviderFactory.java:165)
> > > at
> > > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(Pr
> > > ov
> > > id
> > > er
> > > Factory.java:85)
> > > at
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Forke
> > > d
> > > Booter.java:103)
> > > at
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:7
> > > 4)
