On Thu, 10 Mar 2011 11:09:21 +1300, Michael Hope <michael.h...@linaro.org>
wrote:
> > * How frequently do problems get through review that are found by the
> > test suite?
>
> Ideally the test suite should have been run before the review and have
> shown no regressions. That's how upstream does it. GCC is
> complicated so a review will miss subtle things.
Sure, but reviews shouldn't be trying to catch every single problem that
may exist.
If you don't have an idea of how much benefit test-before-merge has over
just review-before-merge then it's going to be hard to estimate the risk
of changing that scheme.
> > * How frequently do problems get past the testsuite?
>
> Being pessamistic, say every second patch causes a failure found in
> the field not found in the testsuite.
That indicates that getting the code out there for (non-production) use
would actually be worthwhile, rather than trying too hard to ensure
everything goes through the testsuite. (Unless every patch has 10
problems that are not found in review but found by the testsuite.)
> > * Would you be confident in your ability to have quality releases if
> > there wasn't a testsuite run of each change before it lands in
> > trunk?
>
> Not a quality release, no, but I have good confidence in the patches
> in general. It's unlikely that a commit would break trunk.
What sense are you using "break" here?
It sounds like prompt review->integration branch->beta testers
|
`->test suite->trunk
would be best if you don't think that you can have a quality release if
you merge-before-test.
If you feel that you are equipped to recover from problems found by
running the test suite against trunk before doing a release then it may
be better to avoid the overhead and merge to trunk after prompt review,
and deal with test failures when they happen.
Thanks,
James
_______________________________________________
linaro-dev mailing list
linaro-dev@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-dev
