Am Dienstag, 23. August 2016, 10:06:01 schrieb Craig Scott: > Cheeky way to get me more involved in contributing, but okay, I'll bite. ;) > Switching discussion to the dev list. > > So how would you want the feature to work? I'd suggest an initial set of > requirements something like the following: > > - Need to support the ability to define multiple setup and/or tear down > tasks. > - It should be possible to specify dependencies between setup tasks and > between tear down tasks. > - Individual tests need to be able to indicate which setup and/or tear > down tasks they require, similar to the way DEPENDS is used to specify > dependencies between test cases. > - When using ctest --rerun-failed, ctest should automatically invoke any > setup or tear down tasks required by the test cases that will be re-run. > - Setup or tear down tasks which reference executable targets should > substitute the actual built executable just like how add_custom_command() > does.
-need a way to mark if 2 tests with the same setup/teardown can share those or if they need to run for every of them -the default for each test is "no s/t", which means it can't be run with any of the above in parallel (especially for compatibillity)[1] -need a way to tell if a test doesn't care about those 1) think of a database connector test: the test that will check what happens if no DB is present will fail if the setup step "start DB" was run, but not the teardown > Some open questions: > > - Should setup and tear down tasks be defined in pairs, or should they > completely independent (this would still require the ability to specify a > dependency of a tear down task on a setup task)? The test could be "shutdown daemon" or "delete files" so I would keep them separated. They may be created by the same command, so they could be batched anyway. > - Should the setup and tear down tasks be defined by a new CTest/CMake > command or extend an existing mechanism (e.g. add_custom_command())? Don't clutter existing commands more than needed. If it's something new, then create a new command (they could still share C++ code). If it's basically the same as anything existing at the end then use that. > - If no test case has a dependency on a setup or tear down task, should > that task be skipped? Perhaps tasks need to have a flag which indicates > whether they always run or only if a test case depends on it. Keep backward compatibility. I.e. if I now add a new test with s/t, then every other test should still run (and succeed) as before. > - What terminology to use? Things like GoogleTest use terms like test > *fixtures* for this sort of thing. The terms setup and tear down are a > bit imprecise and cumbersome, so we would probably need something better > than those. > - Would it make sense for the ctest command line to support disabling > setup and/or tear down steps? I can see some potential scenarios where > this may be desirable, but maybe this is getting too ambitious for a > starting set of requirements. IMHO that doesn't make sense. One could think about an option to disable the s/t merging, i.e. that they are really called alone for every test. > - What should happen if a setup or tear down task fails? How would > failure be detected? How would such failures impact things like a CDash > test report, etc.? Then the test fails, just the same as it now does when it can't find the executable. Eike
signature.asc
Description: This is a digitally signed message part.
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
