I just began to make ctest replace autotools for the program I develop, but I still have some problems while running ctest. Computer: Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz RAM 3,37 GO System: Microsoft Windows XP Version 2002 Service Pack 3 Environment: Cygwin 1.7.17(0.262/5/3) cmake version: 2.8.9 Context: Application made using cmake. Testing by ctest. Tests are run either by the command: make test or ctest Example: ------------------------------------------------------------------------ $ ctest Test project /cygdrive/d/F053022/Programs/middle-1.4/build Start 1: t_proto_child 1/30 Test #1: t_proto_child .................... Passed 0.09 sec Start 2: t_CsvReader ... 18/30 Test #18: test_05 .......................... Passed 0.28 sec Start 19: test_06 19/30 Test #19: test_06 .......................... Passed 0.23 sec Start 20: test_07 20/30 Test #20: test_07 ..........................***Not Run 0.25 sec Start 21: test_08 21/30 Test #21: test_08 ..........................***Not Run 0.27 sec Start 22: test_09 22/30 Test #22: test_09 .......................... Passed 0.33 sec Start 23: test_10 ... 25/30 Test #25: test_12 .......................... Passed 0.33 sec Start 26: test_13 26/30 Test #26: test_13 .......................... Passed 0.31 sec Start 27: test_14 27/30 Test #27: test_14 ..........................***Not Run 0.37 sec Start 28: test_15 28/30 Test #28: test_15 .......................... Passed 0.44 sec Start 29: test_16 29/30 Test #29: test_16 .......................... Passed 0.53 sec Start 30: test_17 30/30 Test #30: test_17 ..........................***Not Run 0.39 sec 87% tests passed, 4 tests failed out of 30 Total Test time (real) = 7.86 sec The following tests FAILED: 20 - test_07 (BAD_COMMAND) 21 - test_08 (BAD_COMMAND) 27 - test_14 (BAD_COMMAND) 30 - test_17 (BAD_COMMAND) Errors while running CTest ------------------------------------------------------------------------ The tests that fail make it randomly. That is, for instance, if I run, for the test test_07 that failed: ctest -R test_07 the test may succeed. Or if I run make test again, the set of tests that fail may be different. All test_nn test are built on the same model. for instance: add_test (test_07 ${PROJECT_SOURCE_DIR}/test/test_driver -N 07 -S 0 -P test_parser) As the tests were run originally from automake, an environment variable is set, at end of CMakeLists.txt file by the command: set_tests_properties ( test_01 test_02 test_03 test_04 test_05 test_06 test_07 test_08 test_09 test_10 test_11 test_12 test_13 test_14 test_15 test_16 test_17 PROPERTIES ENVIRONMENT srcdir=${PROJECT_SOURCE_DIR}/test) A random failure of the tested program is discarded, because the tested program is activated through a shell, "test_driver". This shell outputs a trace upon entry and exit. The final trace on exit is produced by the following script lines (korn shell). If the trace produced contains "PASS", I don't see how the exit status could differ from 0: if [[ ${errors} -gt 0 ]] ; then MSG=FAIL else MSG=PASS fi echo "${MSG}" >> trc_${NUM}.txt exit ${errors}
All tests mentioned as "Not Run" are actually run, since a corresponding trace output file is found for each test. In all cases, the trace output is normal, indicating correct entry and a normal status on exit, just before the exit command, whith the 0 parameter that would normally correspond to success. Thus, the problem seems to come from ctest itself. ------------------------------------------------------------------------ On the internet, typing "cmake random BAD_COMMAND error" gets some stories, but with no interesting answer. Is this king of behavior already known ? Does that correspond to a bug ? Is there a known workaround ? andre
-- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake