On Feb 18, 2008 8:03 PM, Brett Porter <[EMAIL PROTECTED]> wrote:

> On 19/02/2008, at 3:15 AM, John Casey wrote:
> Things like:
> - running the tests twice by necessity


Having being burned by this little foible...

I often see people complain that the code coverage tools runs the unit tests
twice.

I often see people want to only run the tests once...

We have had the following:

* Unit tests that only pass on a multi-processor system when running with
code coverage tools (due to synchronization problems. the code coverage's
instrumented byte code by necessity has to synchronize updating method
coverage, which forces the vm's to pick up some variable changes that it
would not ordinarily see)

* Unit tests that only fail when running with code coverage tools (one
example is the tests we wrote after we found the synchronization logic error
in our code)

Then there is the fun of is the code coverage valid if the unit tests fail?
I say that the coverage is valid if all the un-instrumented tests pass, but
if the instrumented tests fail you need to examine the failures case by
case.

My rule of thumb is to _always_ run the tests twice, once with coverage and
once without. The test failure during the second (instrumented) execution
should not fail the build.

- not being able to reuse the enhanced classes easily
> - lack of consistency between build time checks and reporting
>

Reply via email to