I suspect that per David's suggestion, CTest would essentially do what you're doing but in a more thorough manner (based on the test framework used). So essentially CTest will need:
- A new command line parameter that specifies the test framework - Functionality to parse the contents of source files provided to the target the test is assigned to I'll try to look into this when I can. David, if you have more specific requirements or pointers please do share them here. Thanks for the input everyone. On Tue, Jan 27, 2015 at 5:08 PM, Fraser Hutchison <fraser.hutchi...@gmail.com> wrote: > We've run into this exact chicken and egg problem with both gtest and Catch > at my work too. In both cases, we've worked round the problem by writing > fairly fragile CMake code which parses the C++ files comprising a test > executable and ultimately calling add_test for each discovered test case. > > gtest is harder to handle than Catch in that it provides macros for handling > value-parameterised and type-parameterised tests. It would be great if > CTest were able to handle these popular test libraries automatically. I'm > really short of time of the moment, but if there are any pointers from > experts on how this might be best handled by CTest, I could perhaps start to > have a poke about if there are no other takers? > > In the meantime, here's a link to our Catch-handling code: > https://github.com/Fraser999/MaidSafe/blob/846ce37c23bd86ee261e439919dfd7000a8f9372/cmake_modules/add_catch_tests.cmake > > The file's commented at the top. It probably doesn't cover all Catch > macros, and I'd be amazed if it handled all C++ coding styles properly, but > it worked fairly reliably for us. We don't use Catch any more, so I'm not > even sure if it works with the current version of Catch. For comparison (of > the complexity) here's our gtest equivalent: > https://github.com/Fraser999/MaidSafe/blob/72af3a0def2f03af7320696f5ea3d241c5af9bdc/cmake_modules/add_gtests.cmake > (it's old code and could be made more efficient I'm sure - but again it > works fine for us). > > Cheers, > Fraser. > > > > On 27/01/2015 17:14, David Cole via CMake wrote: >> >> The best thing to do would be to make CTest aware of unit test >> frameworks like this, and have it be automatic. >> >> Support for cppunit, gtest and Catch (and any other popular ones) >> would all be appreciated by many, I'm sure. >> >> Got time to dig into ctest a little bit? >> >> >> D >> >> >> >> On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey >> <rcdailey.li...@gmail.com> wrote: >>> >>> I wouldn't say this is an artificial problem. let me go into more detail. >>> >>> A single test can define any number of test cases. These are unknown >>> to CMake scripts, as they are written and defined in code. The ideal >>> solution is to have the test program output a list of all test cases >>> and based on that list, automatically generate the add_test commands. >>> That way I don't have to worry about hard-coding test case names in >>> the build scripts. When a new test case is added to the test code (no >>> new files would be added), "magically" a new unit test appears to >>> CMake. >>> >>> But none of this can be done unless the unit test executables are >>> built. Thus the chicken and egg problem. To do it as you suggest would >>> require me to hard-code the test cases. Example: >>> >>> In MyTest.cpp: >>> >>> #include <catch.hpp> >>> TEST_CASE("test1") {} >>> TEST_CASE("test2") {} >>> >>> In CMakeLists.txt: >>> >>> add_executable( MyTest MyTest.cpp ) >>> add_test( NAME MyTest_test1 COMMAND test1 ) >>> add_test( NAME MyTest_test2 COMMAND test2 ) >>> >>> Am I correctly understanding what you have in mind? If so, can you >>> think of a way to automate this a little better? >>> >>> On Tue, Jan 27, 2015 at 7:18 AM, David Cole <dlrd...@aol.com> wrote: >>>> >>>> It's only a chicken and egg problem if you constrain yourself >>>> artificially. >>>> >>>> When you define a new unit test source file, there are two things you >>>> need to do: >>>> (1) add it to its executable >>>> (2) write an add_test line that invokes that specific unit test >>>> >>>> You could possibly encapsulate those actions into an add_unit_test >>>> macro so that it only seems like you have one thing to do, but >>>> certainly, it's possible. >>>> >>>> You must be doing #1 already, so you just have to make sure #2 always >>>> happens whenever #1 happens. >>>> >>>> Both eggs. >>>> >>>> >>>> D >>>> >>>> >>>> >>>> >>>> On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey >>>> <rcdailey.li...@gmail.com> wrote: >>>>> >>>>> I believe so, but then you run into the chicken and egg problem: >>>>> >>>>> I need to fully build all unit tests in order to know all the test >>>>> cases, but I need to define the tests at the generation phase (before >>>>> build). >>>>> >>>>> I'm just not sure how to handle this. The best i can think of is to >>>>> generate 1 test executable per CPP file that we write in our unit >>>>> tests (generally 1 CPP file per class that is unit tested). Any ideas? >>>>> >>>>> On Mon, Jan 26, 2015 at 10:05 AM, David Cole <dlrd...@aol.com> wrote: >>>>>> >>>>>> Can you run a command that executes one test case at a time, or do you >>>>>> have to run all 50 when you execute the tests for a library? >>>>>> >>>>>> ctest just runs whatever you give it with add_test -- if you want to >>>>>> filter a collection of unit tests to run only a single unit test, then >>>>>> the unit test framework you're using would have to support that. Does >>>>>> Catch allow you to run the tests individually? >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey >>>>>> <rcdailey.li...@gmail.com> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm using Catch as my unit test framework: >>>>>>> https://github.com/philsquared/Catch >>>>>>> >>>>>>> Is it possible for CTest to report each TEST_CASE block as an >>>>>>> individual test? My understanding is that CTest will only treat each >>>>>>> executable as a test. However, my test structure is as follows: >>>>>>> >>>>>>> 1 library >>>>>>> 1 test project >>>>>>> >>>>>>> Each test project has 1 test CPP file per each class in the library. >>>>>>> This way I implement tests for classes in a corresponding CPP file. >>>>>>> >>>>>>> Each CPP file contains multiple test cases (defined by TEST_CASE >>>>>>> macro). >>>>>>> >>>>>>> The resulting output of `ctest -T Test` shows only 1 test, even >>>>>>> though >>>>>>> that may be around 50 test cases. I'd like CMake to show the >>>>>>> pass/fail >>>>>>> status of each one. Is this possible? >>>>>>> -- >>>>>>> >>>>>>> Powered by www.kitware.com >>>>>>> >>>>>>> Please keep messages on-topic and check the CMake FAQ at: >>>>>>> http://www.cmake.org/Wiki/CMake_FAQ >>>>>>> >>>>>>> Kitware offers various services to support the CMake community. For >>>>>>> more information on each offering, please visit: >>>>>>> >>>>>>> CMake Support: http://cmake.org/cmake/help/support.html >>>>>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html >>>>>>> CMake Training Courses: http://cmake.org/cmake/help/training.html >>>>>>> >>>>>>> Visit other Kitware open-source projects at >>>>>>> http://www.kitware.com/opensource/opensource.html >>>>>>> >>>>>>> Follow this link to subscribe/unsubscribe: >>>>>>> http://public.kitware.com/mailman/listinfo/cmake > > -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
