Jamie, thanks for taking the time to detail out your proposal. I have no major disagreements. The proposal looks fine to me.
On Thu, May 22, 2014 at 6:21 AM, Jamie Hannaford < jamie.hannaf...@rackspace.com> wrote: > Hey everyone, > > Based on our conversation in irc yesterday, I’m detailing a few > proposals for the way we handle testing. Before that, I want to establish > the terminology so we’re all on the same page. In a software project like > this, there are generally three types of tests: unit tests, integration > tests, end-to-end tests. > > > *End-to-end testing* is the full run-through of a service operation as > an end-user would interact with it. You pass in your identity parameters, > instantiate a service like Swift, and execute an operation. A real HTTP > request is sent over the wire, and a real HTTP response from the server is > received. In other words, it’s a live network test of *all components *from > end to end. Right now, any time we’re communicating with the server API in > our test suite, it’s an *end-to-end test*. There doesn’t need to be many > of these - just enough to test that our API works for end-users. E2E tests > will typically be slow (due to network calls) - but this does not matter. > > > *Integration testing *is like end-to-end testing except no network > connections happen. All it does is test is the *integration *between > modules of the application. So if we want to test Swift operation - we’d > instantiate a context object with identity parameters, then instantiate a > Swift service object, and then after we’ve done the setup, finally test the > operation object. In an integration test, the flow of execution happens > like it would an end-to-end test, but all we’re testing is that different > components work together. This is useful for ensuring that contracts > between interfaces are being satisfied, etc. I don’t think we need to worry > about writing these. > > > *Unit testing* is very different from both of the above. Instead, you > test extremely small “units” of behavior in a particular class. Each test > needs to be fully isolated and have only 1 responsibility (i.e. test one > unit). The class you’re testing should not collaborate with real objects; > instead, you need to pass in mocks. So, if we’re unit testing a Swift > operation, instead of using a real service or transport client - we mock > them and use the mock objects in our tests. If our tested class invokes > methods on this mock, we also need to explicitly define how it does so. For > example, if we’re testing a method that calls `$this->client->foo()` > internally and expects a response, we need to explicitly tell the mocked > client object to return a value when its “foo” method is called. We can > also be more granular and strict: we can say that the method should only be > called with certain arguments, that the method should be called x number of > times. With unit tests, you are defining and testing communication > promises. The point of doing this is that you’re testing HOW your object > communicates in a precise way. Because they’re isolated, there should be > hundreds of unit tests, and they should be very quick. > > > Here’s an example of how you’d mock for unit tests in phpunit: > https://gist.github.com/jamiehannaford/ad7f389466ac5dcafe7a > > There was a proposal made that we should just inject a HTTP client that > doesn’t make real transactions - but that is not feasible because it goes > against what a unit test is. If you were to do this, your tested class > would be executing UNMOCKED methods against a REAL object - making as many > calls as it wants. It would not be isolated, and for that reason it would > not be a unit test. It would be an *integration test* because you’re > forcing your tested class to interact like it would in the wild. Instead, > you should mock every collaborator and explicitly define which calls are > made against them by your tested class. > > There are amazing libraries out there like Prophecy and phpspec which > mocking a whole load easier and more natural - but I assume nobody wants to > move away from phpunit… > > > *Proposal going forward* > > So, here are my proposals for our current library: > > 1. Refactor our unit tests to use mocking instead of real HTTP/network > calls. If a class relies on dependencies or collaborators, they will need > to be mocked. > > 2. As services are added (Swift, Nova, Keystone), end-to-end tests are > added for each. We’d therefore ensure that our SDK is interacting with the > real API as expected. > > 3. Never use the @depends annotation on unit tests, because it makes > them tightly coupled with each other and brittle. A unit test is supposed > to be completely autonomous and independent - it should never depend on the > output of another test. > > 4. Use the “setUp” and “tearDown” helper methods to easily set up test > fixtures > > 5. In our source code, we need to make use of Dependency Injection AS > MUCH AS POSSIBLE because it’s easier to test. If we don’t (choosing to > directly instantiate objects in our code), it introduces tight coupling and > is extremely hard to mock. > > > > Does anybody have any major disagreements with my above proposal? > > > Jamie > > > > Jamie Hannaford > Software Developer III - CH [image: experience Fanatical Support] [image: > LINE] Tel: +41434303908Mob: +41791009767 [image: Rackspace] > > > > Rackspace International GmbH a company registered in the Canton of Zurich, > Switzerland (company identification number CH-020.4.047.077-1) whose > registered office is at Pfingstweidstrasse 60, 8005 Zurich, Switzerland. > Rackspace International GmbH privacy policy can be viewed at > www.rackspace.co.uk/legal/swiss-privacy-policy > - > Rackspace Hosting Australia PTY LTD a company registered in the state of > Victoria, Australia (company registered number ACN 153 275 524) whose > registered office is at Suite 3, Level 7, 210 George Street, Sydney, NSW > 2000, Australia. Rackspace Hosting Australia PTY LTD privacy policy can be > viewed at www.rackspace.com.au/company/legal-privacy-statement.php > - > Rackspace US, Inc, 5000 Walzem Road, San Antonio, Texas 78218, United > States of America > Rackspace US, Inc privacy policy can be viewed at > www.rackspace.com/information/legal/privacystatement > - > Rackspace Limited is a company registered in England & Wales (company > registered number 03897010) whose registered office is at 5 Millington > Road, Hyde Park Hayes, Middlesex UB3 4AZ. > Rackspace Limited privacy policy can be viewed at > www.rackspace.co.uk/legal/privacy-policy > - > Rackspace Benelux B.V. is a company registered in the Netherlands (company > KvK nummer 34276327) whose registered office is at Teleportboulevard 110, > 1043 EJ Amsterdam. > Rackspace Benelux B.V privacy policy can be viewed at > www.rackspace.nl/juridisch/privacy-policy > - > Rackspace Asia Limited is a company registered in Hong Kong (Company no: > 1211294) whose registered office is at 9/F, Cambridge House, Taikoo Place, > 979 King's Road, Quarry Bay, Hong Kong. > Rackspace Asia Limited privacy policy can be viewed at > www.rackspace.com.hk/company/legal-privacy-statement.php > - > This e-mail message (including any attachments or embedded documents) is > intended for the exclusive and confidential use of the individual or entity > to which this message is addressed, and unless otherwise expressly > indicated, is confidential and privileged information of Rackspace. Any > dissemination, distribution or copying of the enclosed material is > prohibited. If you receive this transmission in error, please notify us > immediately by e-mail at ab...@rackspace.com and delete the original > message. Your cooperation is appreciated. >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev