Daniel John Debrunner wrote:
Vemund Ostgaard wrote:
So, I think it would be good to work towards tests being executed in
environments that are as similar as possible, and in the same order,
etc. when using different mechanisms that are intended to be equivalent.
On the other hand having the tests run differently has exposed bugs
that might not have been seen if the different mechanisms were made
as identical as possible.
True. Still it would be good if we were clear on whether differences
exist by design to achieve a specific purpose like finding more bugs, or
if its unintentional, to avoid misunderstandings. Ideally I think the
mechanisms themselves (like ant or other tools) should affect the tests
as little as possible, but if we want variation to expose more bugs for
instance it could achieved with different targets or suites that are
clearly intended for that purpose. I guess it is mostly about
documenting what is intended to be equivalent and what is different by
design.
Vemund
