Jörg Jahnke wrote:
Hi,
the reason why the Wiki page speaks of mandatory tests I have mentioned
in a previous mail:
Jörg Jahnke schrieb:
The problem with such tests not being mandatory is that, sooner or
later, some tests would break. That again would lead to a state where
the user of the tests could not be sure whether a broken test-case
means that he introduced a bug or whether he just encountered an old
problem that broke the test-cases before. He would have to start a
tedious search to find out the cause of the problem - just like the
testers have to do nowadays. And then people would simply not use the
tests because the efforts are too high...
Ause just informed me about another solution that might remove the need
to have the test run on every CWS i.e. we wouldn't need to have the
tests mandatory. His idea is to run the tests on the Master Workspace
prior to announcing the CWS as "ready for CWS use". If a test fails then
this would result in a P1 issue that has to be fixed before the MWS can
be used by everyone. Very similar to how we handle it for the Smoketest
on the MWS nowadays.
Additionally the list of tests to run would be checked in to CVS, so
that we could disable a tests for every user on a given milestone if a
fix cannot be done in time.
That way a developer could get an _optional_ means at hand of doing
regression tests, with no obligation to always run these tests. If the
developer feels that he should run the tests, then he could do so and
invest the (machine) time. If he thinks that the tests will be no
additional help, he just does not run them.
Of course the question then is how often such a regression happens. If
we have to expect to have half a dozen P1 bugs each milestone due to the
mass of regressions, then the "mandatory for every CWS" seems the better
solution to me. But if we expect to have such a P1 bug from the
automatic tests only once every 2 or 3 milestones (or hopefully even
less often), then this seems an acceptable way to me.
Does that make sense?
Are you really asking for my opinion? ;-)
From my point of view this does not make sense at all. It is never a
good idea to shift testing from childworkspaces to master builds. We
would spend quite some time with P1 issues which should have been
detected on the responsible CWS beforehand. This whole discussion is
about testing CWSs before integration to prevent regressions instead of
detection regressions afterwards, isn't it?
BTW, what you describe is quite different to current Smoketest handling
in that doing Smoketest is mandatory on every CWS.
Rüdiger
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
