I just realized that CTest only prioritized failed tests in parallel
test runs since it only modified COSTs in that case. So Casey was
quite
right noticing that there is a behavior change here.
I created a topic "restore-serial-failed-test-order" which limits
that
behavior to parallel test runs.
Should anyone prefer that behavior I could merge it to next.
I think that would be wise for backwards compatibility. The recent
email discussions about CMake's /STACK argument change without a policy
highlight the importance of this.
Pro/con the current behavior in next:
+ Consistent sorting for serial and parallel test runs
- There may be users who expect a fixed order of tests in serial runs
- There may be tests in CMake that expect a fixed order of tests in
serial runs(?)
+ When bug fixing or implementing (in case of TDD) the previously
failed
tests are those that a user is waiting for the most so it may make
sense
to have them run first (I assume that is why it was implemented for
parallel testing)
It's not the tests in CMake that we have to worry about. We can always
adapt the test suite of CMake to account for new behavior if necessary.
It's the tests in all the other projects in the world, that have not
taken advantage of parallel testing yet that you will have to be
concerned about. (For a notable example, the CDash project cannot use
parallel testing. It's tests are highly dependent on order, and can
only be done serially as the tests in its suite do not use the DEPENDS
property to specify order, they rely on the default ordering. You can
verify this by running a CDash dashboard with any proposed build of
CMake/ctest...)
In serial testing, the default ordering of the tests should match the
order of the add_test calls encountered during CMakeLists processing.
HTH,
David C.
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
