Thanks for asking about this, it is a topic that needs a bit of thought. What you'll find, I suspect, is that there is little agreement on a topic like this :) But here is my take.
First off, we need to be careful with terminology. An acceptance test is a test that is worked out with a stakeholder in order to come to a common understanding of what it means for the work to be done. Acceptance tests can deal with the system at any level of integration so long as they clearly express the desires of that stakeholder. Beyond acceptance testing you have tests that incorporate different amounts of the program's ecosystem. A unit test tries to test very small components in isolation (individual methods, classes, or small packages of classes). A fairly common definition of a unit test is that they are fast (on the order of 100 per second, or more) and not dependent on the disk or network (to reduce brittleness and need for setup of environments). After unit tests you get into the realm of component integration tests. These tests take multiple components and bring them together and test that they can communicate with each other. These tests can probably touch a disk, but for simplicity of setup, should probably not touch networks. Scaling up from component integration you get to systems integration. Here you are taking whole products and testing that they work together. These are often the slowest and hardest to control tests. They can bring immense value, but they also come with a large cost. The infrastructure needed for running them can be very costly to maintain and diagnosing a failure (be it infrastructure, bad test, or actual bug) can take a long time. A rule of thumb that I've been told was to aim for a 100 to 10 to 1 ratio of unit to component to system tests. So after all of that, what are the criteria I use? If I can come up with a reasonable test for it at the unit level then that is what I do (even if someone wants to consider it "acceptance", it is a matter of writing the code cleanly). This should be the case if there are no interactions needed with external products and "non-quirky" interactions with the underlying OS (or whatever platform). If that is the case and you still find yourself uncomfortable with writing the test, then there is a code design problem in which components and platforms cannot be sufficiently separated from each other. When we get into the level of multiple components interacting or more complicated interactions with the platform (which is really just another form of components interacting), then there should be a couple component integration tests to make sure that the happy path and maybe a few possible sad paths are exercised. A reason to not jump to this level immediately is speed of execution, possible complexity of setup, and possible impossibility of setup when you get into sad path testing. System tests are reserved for mainly happy path interactions between products. If you can do sad path testing at this point, then good, but usually it will be so complicated and hard to maintain, that you are better off coming up with ways to redesign your code to be able to gain confidence of those conditions at the unit level. On Jun 26, 2012, at 4:08 PM, Ken Barber wrote: > Hey all, > > Sorry - last newbie question for the night. > > I wanted to see what peoples thoughts were around when an acceptance > test is required, there was a debate on a ticket recently and two > people told me it wasn't necessary to add acceptance tests for a > particular case - and my mother told me when enough people tell you > that you are wrong, you probably should start questioning yourself > :-). The particular case itself is irrelevant here, what I'm more > interested in is discussing peoples views on when an acceptance test > is necessary or optional. > > So - when do people think its necessary? Is there a rule people > follow? I'm really after objective arguments here, if they exist. I'm > not sure if all the devs in Puppetlabs are on the same page, but some > pointers around _when_ its a good idea from anyone who has an opinion > would really help me out. > > Thanks. > > ken. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
