[CMake] How to set _default_ timeout for the ctest command?
For the lapack CMake-based build and test system, some of the ctests can exceed 1500 seconds (the default limit) for special conditions (e.g, a quadruple-precision build of lapack that can be very slow). To reduce such issues without forcing users to use the ctest --timeout option, the lapack developers would like to increase the default limit substantially above 1500 seconds. Does anyone here know how to do that? I guessed that setting CTEST_TEST_TIMEOUT might do that, but when I experimentally set that to a small number (actually 1) (just before the first add_test command of another project I am completely familiar with (PLplot, natch)) the ctests that ran beyond that one second limit did not error out (i.e., I presume they were still using the 1500 second default). Note, the ctests I am referring to are simply run using the ctest command from the command line and are not submitted to a dashboard server. I presume another way to do it would be to set the TIMEOUT test property for every ctest, but that constitutes a maintanence issue, i.e., it is a pretty clumsy way to do it. So any ideas for setting the default timeout (either higher or lower than 1500) for ctests would be welcome. Alan __ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __ Linux-powered Science __ -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] How to set _default_ timeout for the ctest command?
Hi Alan, Just by experimenting, I discovered that you also need to set DART_TESTING_TIMEOUT. I'm not sure what the relationship (if any) is between CTEST_TEST_TIMEOUT and DART_TESTING_TIMEOUT. I set them both in the file CTestConfig.cmake. I get the feeling that DART_TESTING_TIMEOUT is there for less than historic reasons. Maybe someone from Kitware can shed a light on this. Regards, Marcel Loose. On 09/12/15 07:33, Alan W. Irwin wrote: > For the lapack CMake-based build and test system, some of the ctests > can exceed 1500 seconds (the default limit) for special conditions > (e.g, a quadruple-precision build of lapack that can be very slow). To > reduce such issues without forcing users to use the ctest --timeout > option, the lapack developers would like to > increase the default limit substantially above 1500 seconds. Does > anyone here know how to do that? > > I guessed that setting CTEST_TEST_TIMEOUT might do that, but when I > experimentally set that to a small number (actually 1) (just before > the first add_test command of another project I am completely familiar > with (PLplot, natch)) the ctests that ran beyond that one second limit > did not error out (i.e., I presume they were still using the 1500 > second default). Note, the ctests I am referring to are simply run > using the ctest command from the command line and are not submitted to > a dashboard server. > > I presume another way to do it would be to set the TIMEOUT test property > for every ctest, but that constitutes a maintanence issue, i.e., it > is a pretty clumsy way to do it. > > So any ideas for setting the default timeout (either higher or lower > than 1500) for ctests would be welcome. > > Alan > __ > Alan W. Irwin > > Astronomical research affiliation with Department of Physics and > Astronomy, > University of Victoria (astrowww.phys.uvic.ca). > > Programming affiliations with the FreeEOS equation-of-state > implementation for stellar interiors (freeeos.sf.net); the Time > Ephemerides project (timeephem.sf.net); PLplot scientific plotting > software package (plplot.sf.net); the libLASi project > (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); > and the Linux Brochure Project (lbproject.sf.net). > __ > > Linux-powered Science > __ <>-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] How to set _default_ timeout for the ctest command?
On 2015-12-09 09:23+0100 Marcel Loose wrote: Hi Alan, Just by experimenting, I discovered that you also need to set DART_TESTING_TIMEOUT. I'm not sure what the relationship (if any) is between CTEST_TEST_TIMEOUT and DART_TESTING_TIMEOUT. I set them both in the file CTestConfig.cmake. I get the feeling that DART_TESTING_TIMEOUT is there for less than historic reasons. Maybe someone from Kitware can shed a light on this. Thanks, Marcel, for trying those experiments. I tried some additional experiments and found that all that is needed is to set the CTEST_TEST_TIMEOUT value before CTest is included in my top-level CMakeLists.txt file. (My previous experiment set that value after the include, but before the first add_test command.) Note that I do not use CTestConfig.cmake at all. So the current experimental logic is # Control the default timeout for all ctests. set(CTEST_TEST_TIMEOUT 1) include(CTest) Those commands generate the following timeout results in the build tree. software@raven> find . -type f |xargs grep -i timeout ./CMakeCache.txt:DART_TESTING_TIMEOUT:STRING=1 ./CMakeCache.txt://ADVANCED property for variable: DART_TESTING_TIMEOUT ./CMakeCache.txt:DART_TESTING_TIMEOUT-ADVANCED:INTERNAL=1 ./DartConfiguration.tcl:# TimeOut is the amount of time in seconds to wait for processes ./DartConfiguration.tcl:# to complete during testing. After TimeOut seconds, the ./DartConfiguration.tcl:TimeOut: 1 ./DartConfiguration.tcl:# For CTest submissions that timeout, these options So for this case, DART_TESTING_TIMEOUT is automatically set to the specified CTEST_TEST_TIMEOUT value. (If CTEST_TEST_TIMEOUT is not set, the cached value of DART_TESTING_TIMEOUT is 1500, and the value of the timeout in DartConfiguration.tcl is also set to 1500). Afterwards, ctest timed out at 1 second for all tests that exceeded that value, but it turns out that is a hard-coded value which cannot be superseded by the ctest --timeout option. So this approach is not really ideal for the lapack developers who will want to increase the default limit by only an order of magnitude or less to substantially reduce the probability someone with a slow computer or special circumstances will have to override using the ctest --timeout option, but that change likely will not reduce that probability to zero so keeping the ability to override the lapack timeout default by the ctest --timeout option is still likely imporatant to them. @CMake developers: The documentation of CTEST_TEST_TIMEOUT (and DART_TESTING_TIMEOUT) includes the following language: "The default timeout for each test if not specified by the `TIMEOUT` test property. Also, the documentation of the ctest --timeout option says: `--timeout ` Set a global timeout on all tests. This option will set a global timeout on all tests that do not already have a timeout set on them. So I guess it could be argued that CTEST_TEST_TIMEOUT does impose a specific timeout on all tests, and that is why the --timeout option is being ignored for this case. But that is not particularly useful behaviour, and it would be far better in my opinion for the --timeout option to work for all tests where the TIMEOUT test property has not been specifically set by the project's build system. I would appreciate some further investigation of these matters by a knowledgable CMake developer, and if it turns out this issue is actually caused by a cmake or ctest bug, then I would be willing to write up the appropriate bug report if it turns out an instantaneous fix could not be made. I also tried the experiment of dropping the setting of CTEST_TEST_TIMEOUT and using set_tests_properties to set the TIMEOUT for individual commands, but consistent with the above wording of the documentation, the ctest --timeout option was ignored for this case. I am OK with that limitation so long (see above) as some change is made so CTEST_TEST_TIMEOUT is interpreted exactly the same as the 1500 default timeout, i.e., the CTEST_TEST_TIMEOUT value can be overridden by the ctest --timeout option. Alan __ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __ Linux-powered Science __ -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http:
Re: [CMake] How to set _default_ timeout for the ctest command?
On 12/09/2015 10:06 PM, Alan W. Irwin wrote: > Also, the documentation of the ctest --timeout option says: > > `--timeout ` > Set a global timeout on all tests. > > This option will set a global timeout on all tests that do not > already have a timeout set on them. Yes, one can see that implemented here: https://cmake.org/gitweb?p=cmake.git;a=blob;f=Source/CTest/cmCTestTestHandler.cxx;hb=v3.4.1#l1112 https://cmake.org/gitweb?p=cmake.git;a=blob;f=Source/cmCTest.cxx#l2108 Here is a commit that set the order of precedence: Local test timeouts will now always take precedence over the global --timeout option. https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=c32088cf >From the history of that code it looks like --timeout was added with no consideration of its relationship to CTEST_TEST_TIMEOUT. Some other CTest options in the same scope as CTEST_TEST_TIMEOUT can be overridden by the command line, so I agree that --timeout should override it too. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
