On Wed, Nov 2, 2011 at 5:12 PM, Nicolas Spiegelberg <nspiegelb...@fb.com> wrote: > I don't think tying the maturity of a patch merely to a metric as simple > as the presence of a unit test. Unit tests are great, but they're a means > instead of an ends to quality. I've seen super-buggy code that had a > number of unit tests. I've seen code that passed the unit test, but the > test assumptions were wrong. I've seen code that failed a unit test > because the unit test was horribly written. There's a lot of metrics that > make a feature more palatable. How are you using this? Can you give > examples? Is it in production or just something you wanted? How actively > is this being used? What sort of scaling is being tested here? These are > questions that contributors should be available to answer if they submit a > large diff that impacts a critical section of code and will be a necessary > design decision for many future features.
I phrased this as "write sufficient tests" since I mostly agree with the above. It's possible to have lots of unit tests which show nothing, or few tests which convince people that it works. The measure isn't binary, even though the QA bot will give a "+1 tests included" for including a single test. I don't think it's OK to only have manual tests. Manual tests (or stable production deployments) prove that the feature is stable _today_ but do not show that the feature is stable tomorrow or the next week. Unless we had a QA team and a list of text files checked in with all of the manual tests needed to certify a release, we'd end up releasing things with good chances of regressions. Adding a system test suite would do us some good here. Accumulo has a very nice one which we could work on cloning (though the APIs are different we can borrow the test scenarios). -Todd -- Todd Lipcon Software Engineer, Cloudera
