Hi all. I've run into this problem as well - I think first-class support for this in CTest would be a most useful feature.
And I would like to nominate Boost.Test into the list of frameworks considered for support, if possible. Petr On Wed, Jan 28, 2015 at 1:38 AM, Robert Dailey <rcdailey.li...@gmail.com> wrote: > I suspect that per David's suggestion, CTest would essentially do what > you're doing but in a more thorough manner (based on the test > framework used). So essentially CTest will need: > > - A new command line parameter that specifies the test framework > - Functionality to parse the contents of source files provided to the > target the test is assigned to > > I'll try to look into this when I can. David, if you have more > specific requirements or pointers please do share them here. Thanks > for the input everyone. > > On Tue, Jan 27, 2015 at 5:08 PM, Fraser Hutchison > <fraser.hutchi...@gmail.com> wrote: > > We've run into this exact chicken and egg problem with both gtest and > Catch > > at my work too. In both cases, we've worked round the problem by writing > > fairly fragile CMake code which parses the C++ files comprising a test > > executable and ultimately calling add_test for each discovered test case. > > > > gtest is harder to handle than Catch in that it provides macros for > handling > > value-parameterised and type-parameterised tests. It would be great if > > CTest were able to handle these popular test libraries automatically. > I'm > > really short of time of the moment, but if there are any pointers from > > experts on how this might be best handled by CTest, I could perhaps > start to > > have a poke about if there are no other takers? > > > > In the meantime, here's a link to our Catch-handling code: > > > https://github.com/Fraser999/MaidSafe/blob/846ce37c23bd86ee261e439919dfd7000a8f9372/cmake_modules/add_catch_tests.cmake > > > > The file's commented at the top. It probably doesn't cover all Catch > > macros, and I'd be amazed if it handled all C++ coding styles properly, > but > > it worked fairly reliably for us. We don't use Catch any more, so I'm > not > > even sure if it works with the current version of Catch. For comparison > (of > > the complexity) here's our gtest equivalent: > > > https://github.com/Fraser999/MaidSafe/blob/72af3a0def2f03af7320696f5ea3d241c5af9bdc/cmake_modules/add_gtests.cmake > > (it's old code and could be made more efficient I'm sure - but again it > > works fine for us). > > > > Cheers, > > Fraser. > > > > > > > > On 27/01/2015 17:14, David Cole via CMake wrote: > >> > >> The best thing to do would be to make CTest aware of unit test > >> frameworks like this, and have it be automatic. > >> > >> Support for cppunit, gtest and Catch (and any other popular ones) > >> would all be appreciated by many, I'm sure. > >> > >> Got time to dig into ctest a little bit? > >> > >> > >> D > >> > >> > >> > >> On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey > >> <rcdailey.li...@gmail.com> wrote: > >>> > >>> I wouldn't say this is an artificial problem. let me go into more > detail. > >>> > >>> A single test can define any number of test cases. These are unknown > >>> to CMake scripts, as they are written and defined in code. The ideal > >>> solution is to have the test program output a list of all test cases > >>> and based on that list, automatically generate the add_test commands. > >>> That way I don't have to worry about hard-coding test case names in > >>> the build scripts. When a new test case is added to the test code (no > >>> new files would be added), "magically" a new unit test appears to > >>> CMake. > >>> > >>> But none of this can be done unless the unit test executables are > >>> built. Thus the chicken and egg problem. To do it as you suggest would > >>> require me to hard-code the test cases. Example: > >>> > >>> In MyTest.cpp: > >>> > >>> #include <catch.hpp> > >>> TEST_CASE("test1") {} > >>> TEST_CASE("test2") {} > >>> > >>> In CMakeLists.txt: > >>> > >>> add_executable( MyTest MyTest.cpp ) > >>> add_test( NAME MyTest_test1 COMMAND test1 ) > >>> add_test( NAME MyTest_test2 COMMAND test2 ) > >>> > >>> Am I correctly understanding what you have in mind? If so, can you > >>> think of a way to automate this a little better? > >>> > >>> On Tue, Jan 27, 2015 at 7:18 AM, David Cole <dlrd...@aol.com> wrote: > >>>> > >>>> It's only a chicken and egg problem if you constrain yourself > >>>> artificially. > >>>> > >>>> When you define a new unit test source file, there are two things you > >>>> need to do: > >>>> (1) add it to its executable > >>>> (2) write an add_test line that invokes that specific unit test > >>>> > >>>> You could possibly encapsulate those actions into an add_unit_test > >>>> macro so that it only seems like you have one thing to do, but > >>>> certainly, it's possible. > >>>> > >>>> You must be doing #1 already, so you just have to make sure #2 always > >>>> happens whenever #1 happens. > >>>> > >>>> Both eggs. > >>>> > >>>> > >>>> D > >>>> > >>>> > >>>> > >>>> > >>>> On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey > >>>> <rcdailey.li...@gmail.com> wrote: > >>>>> > >>>>> I believe so, but then you run into the chicken and egg problem: > >>>>> > >>>>> I need to fully build all unit tests in order to know all the test > >>>>> cases, but I need to define the tests at the generation phase (before > >>>>> build). > >>>>> > >>>>> I'm just not sure how to handle this. The best i can think of is to > >>>>> generate 1 test executable per CPP file that we write in our unit > >>>>> tests (generally 1 CPP file per class that is unit tested). Any > ideas? > >>>>> > >>>>> On Mon, Jan 26, 2015 at 10:05 AM, David Cole <dlrd...@aol.com> > wrote: > >>>>>> > >>>>>> Can you run a command that executes one test case at a time, or do > you > >>>>>> have to run all 50 when you execute the tests for a library? > >>>>>> > >>>>>> ctest just runs whatever you give it with add_test -- if you want to > >>>>>> filter a collection of unit tests to run only a single unit test, > then > >>>>>> the unit test framework you're using would have to support that. > Does > >>>>>> Catch allow you to run the tests individually? > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey > >>>>>> <rcdailey.li...@gmail.com> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> I'm using Catch as my unit test framework: > >>>>>>> https://github.com/philsquared/Catch > >>>>>>> > >>>>>>> Is it possible for CTest to report each TEST_CASE block as an > >>>>>>> individual test? My understanding is that CTest will only treat > each > >>>>>>> executable as a test. However, my test structure is as follows: > >>>>>>> > >>>>>>> 1 library > >>>>>>> 1 test project > >>>>>>> > >>>>>>> Each test project has 1 test CPP file per each class in the > library. > >>>>>>> This way I implement tests for classes in a corresponding CPP file. > >>>>>>> > >>>>>>> Each CPP file contains multiple test cases (defined by TEST_CASE > >>>>>>> macro). > >>>>>>> > >>>>>>> The resulting output of `ctest -T Test` shows only 1 test, even > >>>>>>> though > >>>>>>> that may be around 50 test cases. I'd like CMake to show the > >>>>>>> pass/fail > >>>>>>> status of each one. Is this possible? > >>>>>>> -- > >>>>>>> > >>>>>>> Powered by www.kitware.com > >>>>>>> > >>>>>>> Please keep messages on-topic and check the CMake FAQ at: > >>>>>>> http://www.cmake.org/Wiki/CMake_FAQ > >>>>>>> > >>>>>>> Kitware offers various services to support the CMake community. For > >>>>>>> more information on each offering, please visit: > >>>>>>> > >>>>>>> CMake Support: http://cmake.org/cmake/help/support.html > >>>>>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html > >>>>>>> CMake Training Courses: http://cmake.org/cmake/help/training.html > >>>>>>> > >>>>>>> Visit other Kitware open-source projects at > >>>>>>> http://www.kitware.com/opensource/opensource.html > >>>>>>> > >>>>>>> Follow this link to subscribe/unsubscribe: > >>>>>>> http://public.kitware.com/mailman/listinfo/cmake > > > > > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > http://public.kitware.com/mailman/listinfo/cmake >
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
