Re: [CMake] CTest + Catch Framework
I'm sure it's possible, as long as CDash can handle the flood of test results. CTest is responsible for taking add_test (or whatever the new CMake command will be called) calls and transforming them into Test.xml files for sending to CDash. As long as you can read the output of running the unit tests, and decide how to split it up into test results, you can map it however you like. The beauty of unwritten code is that it always does exactly what you want it to do. Until you write it. :-) On Wed, Jan 28, 2015 at 11:57 AM, Robert Dailey wrote: > @David > > I agree with your point about unit test initialization. Perhaps > instead of running a test multiple times, we can inspect the output of > a test that is run 1 time and assuming it spits out success/failure > details for individual test cases, we can simply parse that output > into appropriate CText XML so that it appears as if it was instead run > once for each test case? > > I'm being fairly abstract here as I'm not sure of the implementation > details of CTest or if this is even possible. But it's a thought. Let > me know what you think. > > On Wed, Jan 28, 2015 at 6:43 AM, David Cole wrote: >> I have two more suggestions for whoever takes on this work. >> >> One: add a new CMake command add_unit_test_container (or >> add_unit_tester or a better name ... ?) which names the DLL or >> executable file containing the unit tests. Give it properties to >> indicate how to extract a list of tests from it (perhaps just a "type" >> that maps to internal code that knows how to do this for each >> type...), and how to run *all* the tests, and how to run an individual >> test by name. >> >> Two: consider performance, both of running all the unit tests and of >> submitting/processing results on CDash, ... frequently, some unit test >> containers involve "expensive" setup and teardown, and that's why it's >> a good thing to have many tests in one container -- so the slow stuff >> only has to run once for 100 tests... (or more!) >> >> Unless you really really need this functionality, you might want to >> test out CDash first to make sure it can handle the number of tests >> you want to send it. Done properly, unit tests will test every single >> C++ method in your code... If VTK were to do that, there would be >> **tens of thousands** of test results being sent to CDash for each row >> on the dashboard. >> >> OK, and [Three:], make it a goal to add support for all of the >> following (and make it easily extensible for more): >> >> - gtest >> - cppunit >> - Catch >> - Boost.Test >> - Microsoft::VisualStudio::CppUnitTestFramework >> >> I am currently involved with a project where we're using >> Microsoft::VisualStudio::CppUnitTestFramework for unit tests, and we >> run vstest.console.exe to run all the unit tests at once for each >> library we test. Then we submit to a CDash dashboard as >> "Release.UnitTests.LibraryName" and "Debug.UnitTests.LibraryName" -- >> if a single test fails, you just have to inspect the output and read >> which test failed. >> >> Unit tests, in general, should run nearly instantaneously, and ALL >> your developers should run them and verify passing unit tests before >> every push to the repo. If they're not doing that already, then new >> ctest features ain't gonna help you... >> >> >> Cheers, >> David C. >> >> On Wed, Jan 28, 2015 at 4:19 AM, Nils Gladitz wrote: >>> On 28.01.2015 01:38, Robert Dailey wrote: I suspect that per David's suggestion, CTest would essentially do what you're doing but in a more thorough manner (based on the test framework used). So essentially CTest will need: - A new command line parameter that specifies the test framework - Functionality to parse the contents of source files provided to the target the test is assigned to I'll try to look into this when I can. David, if you have more specific requirements or pointers please do share them here. Thanks for the input everyone. >>> >>> CMake generates "CTestTestfile.cmake" files which contain the code that >>> defines tests during execution of ctest itself. >>> >>> Those "CTestTestfile.cmake" files could use execute_process() to query test >>> binaries for their contained tests and then add tests appropriately. >>> With that approach there would be no need to parse source files. >>> >>> This can already be done in a hackish way by using the TEST_INCLUDE_FILE[1] >>> directory property. >>> file(GENERATE) can be used to create per-test-binary/per-configuration >>> snippets with executable locations ($) which then can be >>> globbed for and included by CTestTestfile.cmake. >>> >>> I would still opt for implementing support for this in cmake itself though >>> since the entire logic could be contained in the generated >>> "CTestTestfile.cmake"s. >>> Perhaps with a couple of properties that would define how to e.g. query the >>> list of tests and the required parameter to run a specific test; that way it
Re: [CMake] CTest + Catch Framework
@David I agree with your point about unit test initialization. Perhaps instead of running a test multiple times, we can inspect the output of a test that is run 1 time and assuming it spits out success/failure details for individual test cases, we can simply parse that output into appropriate CText XML so that it appears as if it was instead run once for each test case? I'm being fairly abstract here as I'm not sure of the implementation details of CTest or if this is even possible. But it's a thought. Let me know what you think. On Wed, Jan 28, 2015 at 6:43 AM, David Cole wrote: > I have two more suggestions for whoever takes on this work. > > One: add a new CMake command add_unit_test_container (or > add_unit_tester or a better name ... ?) which names the DLL or > executable file containing the unit tests. Give it properties to > indicate how to extract a list of tests from it (perhaps just a "type" > that maps to internal code that knows how to do this for each > type...), and how to run *all* the tests, and how to run an individual > test by name. > > Two: consider performance, both of running all the unit tests and of > submitting/processing results on CDash, ... frequently, some unit test > containers involve "expensive" setup and teardown, and that's why it's > a good thing to have many tests in one container -- so the slow stuff > only has to run once for 100 tests... (or more!) > > Unless you really really need this functionality, you might want to > test out CDash first to make sure it can handle the number of tests > you want to send it. Done properly, unit tests will test every single > C++ method in your code... If VTK were to do that, there would be > **tens of thousands** of test results being sent to CDash for each row > on the dashboard. > > OK, and [Three:], make it a goal to add support for all of the > following (and make it easily extensible for more): > > - gtest > - cppunit > - Catch > - Boost.Test > - Microsoft::VisualStudio::CppUnitTestFramework > > I am currently involved with a project where we're using > Microsoft::VisualStudio::CppUnitTestFramework for unit tests, and we > run vstest.console.exe to run all the unit tests at once for each > library we test. Then we submit to a CDash dashboard as > "Release.UnitTests.LibraryName" and "Debug.UnitTests.LibraryName" -- > if a single test fails, you just have to inspect the output and read > which test failed. > > Unit tests, in general, should run nearly instantaneously, and ALL > your developers should run them and verify passing unit tests before > every push to the repo. If they're not doing that already, then new > ctest features ain't gonna help you... > > > Cheers, > David C. > > On Wed, Jan 28, 2015 at 4:19 AM, Nils Gladitz wrote: >> On 28.01.2015 01:38, Robert Dailey wrote: >>> >>> I suspect that per David's suggestion, CTest would essentially do what >>> you're doing but in a more thorough manner (based on the test >>> framework used). So essentially CTest will need: >>> >>> - A new command line parameter that specifies the test framework >>> - Functionality to parse the contents of source files provided to the >>> target the test is assigned to >>> >>> I'll try to look into this when I can. David, if you have more >>> specific requirements or pointers please do share them here. Thanks >>> for the input everyone. >>> >> >> CMake generates "CTestTestfile.cmake" files which contain the code that >> defines tests during execution of ctest itself. >> >> Those "CTestTestfile.cmake" files could use execute_process() to query test >> binaries for their contained tests and then add tests appropriately. >> With that approach there would be no need to parse source files. >> >> This can already be done in a hackish way by using the TEST_INCLUDE_FILE[1] >> directory property. >> file(GENERATE) can be used to create per-test-binary/per-configuration >> snippets with executable locations ($) which then can be >> globbed for and included by CTestTestfile.cmake. >> >> I would still opt for implementing support for this in cmake itself though >> since the entire logic could be contained in the generated >> "CTestTestfile.cmake"s. >> Perhaps with a couple of properties that would define how to e.g. query the >> list of tests and the required parameter to run a specific test; that way it >> could be made to work irregardless of which test framework is being used. >> >> Nils >> >> [1] http://www.cmake.org/cmake/help/v3.1/prop_dir/TEST_INCLUDE_FILE.html >> >> -- >> >> Powered by www.kitware.com >> >> Please keep messages on-topic and check the CMake FAQ at: >> http://www.cmake.org/Wiki/CMake_FAQ >> >> Kitware offers various services to support the CMake community. For more >> information on each offering, please visit: >> >> CMake Support: http://cmake.org/cmake/help/support.html >> CMake Consulting: http://cmake.org/cmake/help/consulting.html >> CMake Training Courses: http://cmake.org/cmake/help/training.html >> >> Visit other Kitware op
Re: [CMake] CTest + Catch Framework
I have two more suggestions for whoever takes on this work. One: add a new CMake command add_unit_test_container (or add_unit_tester or a better name ... ?) which names the DLL or executable file containing the unit tests. Give it properties to indicate how to extract a list of tests from it (perhaps just a "type" that maps to internal code that knows how to do this for each type...), and how to run *all* the tests, and how to run an individual test by name. Two: consider performance, both of running all the unit tests and of submitting/processing results on CDash, ... frequently, some unit test containers involve "expensive" setup and teardown, and that's why it's a good thing to have many tests in one container -- so the slow stuff only has to run once for 100 tests... (or more!) Unless you really really need this functionality, you might want to test out CDash first to make sure it can handle the number of tests you want to send it. Done properly, unit tests will test every single C++ method in your code... If VTK were to do that, there would be **tens of thousands** of test results being sent to CDash for each row on the dashboard. OK, and [Three:], make it a goal to add support for all of the following (and make it easily extensible for more): - gtest - cppunit - Catch - Boost.Test - Microsoft::VisualStudio::CppUnitTestFramework I am currently involved with a project where we're using Microsoft::VisualStudio::CppUnitTestFramework for unit tests, and we run vstest.console.exe to run all the unit tests at once for each library we test. Then we submit to a CDash dashboard as "Release.UnitTests.LibraryName" and "Debug.UnitTests.LibraryName" -- if a single test fails, you just have to inspect the output and read which test failed. Unit tests, in general, should run nearly instantaneously, and ALL your developers should run them and verify passing unit tests before every push to the repo. If they're not doing that already, then new ctest features ain't gonna help you... Cheers, David C. On Wed, Jan 28, 2015 at 4:19 AM, Nils Gladitz wrote: > On 28.01.2015 01:38, Robert Dailey wrote: >> >> I suspect that per David's suggestion, CTest would essentially do what >> you're doing but in a more thorough manner (based on the test >> framework used). So essentially CTest will need: >> >> - A new command line parameter that specifies the test framework >> - Functionality to parse the contents of source files provided to the >> target the test is assigned to >> >> I'll try to look into this when I can. David, if you have more >> specific requirements or pointers please do share them here. Thanks >> for the input everyone. >> > > CMake generates "CTestTestfile.cmake" files which contain the code that > defines tests during execution of ctest itself. > > Those "CTestTestfile.cmake" files could use execute_process() to query test > binaries for their contained tests and then add tests appropriately. > With that approach there would be no need to parse source files. > > This can already be done in a hackish way by using the TEST_INCLUDE_FILE[1] > directory property. > file(GENERATE) can be used to create per-test-binary/per-configuration > snippets with executable locations ($) which then can be > globbed for and included by CTestTestfile.cmake. > > I would still opt for implementing support for this in cmake itself though > since the entire logic could be contained in the generated > "CTestTestfile.cmake"s. > Perhaps with a couple of properties that would define how to e.g. query the > list of tests and the required parameter to run a specific test; that way it > could be made to work irregardless of which test framework is being used. > > Nils > > [1] http://www.cmake.org/cmake/help/v3.1/prop_dir/TEST_INCLUDE_FILE.html > > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsu
Re: [CMake] CTest + Catch Framework
On 28.01.2015 01:38, Robert Dailey wrote: I suspect that per David's suggestion, CTest would essentially do what you're doing but in a more thorough manner (based on the test framework used). So essentially CTest will need: - A new command line parameter that specifies the test framework - Functionality to parse the contents of source files provided to the target the test is assigned to I'll try to look into this when I can. David, if you have more specific requirements or pointers please do share them here. Thanks for the input everyone. CMake generates "CTestTestfile.cmake" files which contain the code that defines tests during execution of ctest itself. Those "CTestTestfile.cmake" files could use execute_process() to query test binaries for their contained tests and then add tests appropriately. With that approach there would be no need to parse source files. This can already be done in a hackish way by using the TEST_INCLUDE_FILE[1] directory property. file(GENERATE) can be used to create per-test-binary/per-configuration snippets with executable locations ($) which then can be globbed for and included by CTestTestfile.cmake. I would still opt for implementing support for this in cmake itself though since the entire logic could be contained in the generated "CTestTestfile.cmake"s. Perhaps with a couple of properties that would define how to e.g. query the list of tests and the required parameter to run a specific test; that way it could be made to work irregardless of which test framework is being used. Nils [1] http://www.cmake.org/cmake/help/v3.1/prop_dir/TEST_INCLUDE_FILE.html -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] CTest + Catch Framework
Hi all.
I've run into this problem as well - I think first-class support for this
in CTest would be a most useful feature.
And I would like to nominate Boost.Test into the list of frameworks
considered for support, if possible.
Petr
On Wed, Jan 28, 2015 at 1:38 AM, Robert Dailey
wrote:
> I suspect that per David's suggestion, CTest would essentially do what
> you're doing but in a more thorough manner (based on the test
> framework used). So essentially CTest will need:
>
> - A new command line parameter that specifies the test framework
> - Functionality to parse the contents of source files provided to the
> target the test is assigned to
>
> I'll try to look into this when I can. David, if you have more
> specific requirements or pointers please do share them here. Thanks
> for the input everyone.
>
> On Tue, Jan 27, 2015 at 5:08 PM, Fraser Hutchison
> wrote:
> > We've run into this exact chicken and egg problem with both gtest and
> Catch
> > at my work too. In both cases, we've worked round the problem by writing
> > fairly fragile CMake code which parses the C++ files comprising a test
> > executable and ultimately calling add_test for each discovered test case.
> >
> > gtest is harder to handle than Catch in that it provides macros for
> handling
> > value-parameterised and type-parameterised tests. It would be great if
> > CTest were able to handle these popular test libraries automatically.
> I'm
> > really short of time of the moment, but if there are any pointers from
> > experts on how this might be best handled by CTest, I could perhaps
> start to
> > have a poke about if there are no other takers?
> >
> > In the meantime, here's a link to our Catch-handling code:
> >
> https://github.com/Fraser999/MaidSafe/blob/846ce37c23bd86ee261e439919dfd7000a8f9372/cmake_modules/add_catch_tests.cmake
> >
> > The file's commented at the top. It probably doesn't cover all Catch
> > macros, and I'd be amazed if it handled all C++ coding styles properly,
> but
> > it worked fairly reliably for us. We don't use Catch any more, so I'm
> not
> > even sure if it works with the current version of Catch. For comparison
> (of
> > the complexity) here's our gtest equivalent:
> >
> https://github.com/Fraser999/MaidSafe/blob/72af3a0def2f03af7320696f5ea3d241c5af9bdc/cmake_modules/add_gtests.cmake
> > (it's old code and could be made more efficient I'm sure - but again it
> > works fine for us).
> >
> > Cheers,
> > Fraser.
> >
> >
> >
> > On 27/01/2015 17:14, David Cole via CMake wrote:
> >>
> >> The best thing to do would be to make CTest aware of unit test
> >> frameworks like this, and have it be automatic.
> >>
> >> Support for cppunit, gtest and Catch (and any other popular ones)
> >> would all be appreciated by many, I'm sure.
> >>
> >> Got time to dig into ctest a little bit?
> >>
> >>
> >> D
> >>
> >>
> >>
> >> On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey
> >> wrote:
> >>>
> >>> I wouldn't say this is an artificial problem. let me go into more
> detail.
> >>>
> >>> A single test can define any number of test cases. These are unknown
> >>> to CMake scripts, as they are written and defined in code. The ideal
> >>> solution is to have the test program output a list of all test cases
> >>> and based on that list, automatically generate the add_test commands.
> >>> That way I don't have to worry about hard-coding test case names in
> >>> the build scripts. When a new test case is added to the test code (no
> >>> new files would be added), "magically" a new unit test appears to
> >>> CMake.
> >>>
> >>> But none of this can be done unless the unit test executables are
> >>> built. Thus the chicken and egg problem. To do it as you suggest would
> >>> require me to hard-code the test cases. Example:
> >>>
> >>> In MyTest.cpp:
> >>>
> >>> #include
> >>> TEST_CASE("test1") {}
> >>> TEST_CASE("test2") {}
> >>>
> >>> In CMakeLists.txt:
> >>>
> >>> add_executable( MyTest MyTest.cpp )
> >>> add_test( NAME MyTest_test1 COMMAND test1 )
> >>> add_test( NAME MyTest_test2 COMMAND test2 )
> >>>
> >>> Am I correctly understanding what you have in mind? If so, can you
> >>> think of a way to automate this a little better?
> >>>
> >>> On Tue, Jan 27, 2015 at 7:18 AM, David Cole wrote:
>
> It's only a chicken and egg problem if you constrain yourself
> artificially.
>
> When you define a new unit test source file, there are two things you
> need to do:
> (1) add it to its executable
> (2) write an add_test line that invokes that specific unit test
>
> You could possibly encapsulate those actions into an add_unit_test
> macro so that it only seems like you have one thing to do, but
> certainly, it's possible.
>
> You must be doing #1 already, so you just have to make sure #2 always
> happens whenever #1 happens.
>
> Both eggs.
>
>
> D
>
>
>
>
> On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey
>
Re: [CMake] CTest + Catch Framework
I suspect that per David's suggestion, CTest would essentially do what
you're doing but in a more thorough manner (based on the test
framework used). So essentially CTest will need:
- A new command line parameter that specifies the test framework
- Functionality to parse the contents of source files provided to the
target the test is assigned to
I'll try to look into this when I can. David, if you have more
specific requirements or pointers please do share them here. Thanks
for the input everyone.
On Tue, Jan 27, 2015 at 5:08 PM, Fraser Hutchison
wrote:
> We've run into this exact chicken and egg problem with both gtest and Catch
> at my work too. In both cases, we've worked round the problem by writing
> fairly fragile CMake code which parses the C++ files comprising a test
> executable and ultimately calling add_test for each discovered test case.
>
> gtest is harder to handle than Catch in that it provides macros for handling
> value-parameterised and type-parameterised tests. It would be great if
> CTest were able to handle these popular test libraries automatically. I'm
> really short of time of the moment, but if there are any pointers from
> experts on how this might be best handled by CTest, I could perhaps start to
> have a poke about if there are no other takers?
>
> In the meantime, here's a link to our Catch-handling code:
> https://github.com/Fraser999/MaidSafe/blob/846ce37c23bd86ee261e439919dfd7000a8f9372/cmake_modules/add_catch_tests.cmake
>
> The file's commented at the top. It probably doesn't cover all Catch
> macros, and I'd be amazed if it handled all C++ coding styles properly, but
> it worked fairly reliably for us. We don't use Catch any more, so I'm not
> even sure if it works with the current version of Catch. For comparison (of
> the complexity) here's our gtest equivalent:
> https://github.com/Fraser999/MaidSafe/blob/72af3a0def2f03af7320696f5ea3d241c5af9bdc/cmake_modules/add_gtests.cmake
> (it's old code and could be made more efficient I'm sure - but again it
> works fine for us).
>
> Cheers,
> Fraser.
>
>
>
> On 27/01/2015 17:14, David Cole via CMake wrote:
>>
>> The best thing to do would be to make CTest aware of unit test
>> frameworks like this, and have it be automatic.
>>
>> Support for cppunit, gtest and Catch (and any other popular ones)
>> would all be appreciated by many, I'm sure.
>>
>> Got time to dig into ctest a little bit?
>>
>>
>> D
>>
>>
>>
>> On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey
>> wrote:
>>>
>>> I wouldn't say this is an artificial problem. let me go into more detail.
>>>
>>> A single test can define any number of test cases. These are unknown
>>> to CMake scripts, as they are written and defined in code. The ideal
>>> solution is to have the test program output a list of all test cases
>>> and based on that list, automatically generate the add_test commands.
>>> That way I don't have to worry about hard-coding test case names in
>>> the build scripts. When a new test case is added to the test code (no
>>> new files would be added), "magically" a new unit test appears to
>>> CMake.
>>>
>>> But none of this can be done unless the unit test executables are
>>> built. Thus the chicken and egg problem. To do it as you suggest would
>>> require me to hard-code the test cases. Example:
>>>
>>> In MyTest.cpp:
>>>
>>> #include
>>> TEST_CASE("test1") {}
>>> TEST_CASE("test2") {}
>>>
>>> In CMakeLists.txt:
>>>
>>> add_executable( MyTest MyTest.cpp )
>>> add_test( NAME MyTest_test1 COMMAND test1 )
>>> add_test( NAME MyTest_test2 COMMAND test2 )
>>>
>>> Am I correctly understanding what you have in mind? If so, can you
>>> think of a way to automate this a little better?
>>>
>>> On Tue, Jan 27, 2015 at 7:18 AM, David Cole wrote:
It's only a chicken and egg problem if you constrain yourself
artificially.
When you define a new unit test source file, there are two things you
need to do:
(1) add it to its executable
(2) write an add_test line that invokes that specific unit test
You could possibly encapsulate those actions into an add_unit_test
macro so that it only seems like you have one thing to do, but
certainly, it's possible.
You must be doing #1 already, so you just have to make sure #2 always
happens whenever #1 happens.
Both eggs.
D
On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey
wrote:
>
> I believe so, but then you run into the chicken and egg problem:
>
> I need to fully build all unit tests in order to know all the test
> cases, but I need to define the tests at the generation phase (before
> build).
>
> I'm just not sure how to handle this. The best i can think of is to
> generate 1 test executable per CPP file that we write in our unit
> tests (generally 1 CPP file per class that is unit tested). Any ideas?
>
> On Mon, Jan 26, 2015 at 10:05 AM, David Cole wrote
Re: [CMake] CTest + Catch Framework
We've run into this exact chicken and egg problem with both gtest and
Catch at my work too. In both cases, we've worked round the problem by
writing fairly fragile CMake code which parses the C++ files comprising
a test executable and ultimately calling add_test for each discovered
test case.
gtest is harder to handle than Catch in that it provides macros for
handling value-parameterised and type-parameterised tests. It would be
great if CTest were able to handle these popular test libraries
automatically. I'm really short of time of the moment, but if there are
any pointers from experts on how this might be best handled by CTest, I
could perhaps start to have a poke about if there are no other takers?
In the meantime, here's a link to our Catch-handling code:
https://github.com/Fraser999/MaidSafe/blob/846ce37c23bd86ee261e439919dfd7000a8f9372/cmake_modules/add_catch_tests.cmake
The file's commented at the top. It probably doesn't cover all Catch
macros, and I'd be amazed if it handled all C++ coding styles properly,
but it worked fairly reliably for us. We don't use Catch any more, so
I'm not even sure if it works with the current version of Catch. For
comparison (of the complexity) here's our gtest equivalent:
https://github.com/Fraser999/MaidSafe/blob/72af3a0def2f03af7320696f5ea3d241c5af9bdc/cmake_modules/add_gtests.cmake
(it's old code and could be made more efficient I'm sure - but again it
works fine for us).
Cheers,
Fraser.
On 27/01/2015 17:14, David Cole via CMake wrote:
The best thing to do would be to make CTest aware of unit test
frameworks like this, and have it be automatic.
Support for cppunit, gtest and Catch (and any other popular ones)
would all be appreciated by many, I'm sure.
Got time to dig into ctest a little bit?
D
On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey
wrote:
I wouldn't say this is an artificial problem. let me go into more detail.
A single test can define any number of test cases. These are unknown
to CMake scripts, as they are written and defined in code. The ideal
solution is to have the test program output a list of all test cases
and based on that list, automatically generate the add_test commands.
That way I don't have to worry about hard-coding test case names in
the build scripts. When a new test case is added to the test code (no
new files would be added), "magically" a new unit test appears to
CMake.
But none of this can be done unless the unit test executables are
built. Thus the chicken and egg problem. To do it as you suggest would
require me to hard-code the test cases. Example:
In MyTest.cpp:
#include
TEST_CASE("test1") {}
TEST_CASE("test2") {}
In CMakeLists.txt:
add_executable( MyTest MyTest.cpp )
add_test( NAME MyTest_test1 COMMAND test1 )
add_test( NAME MyTest_test2 COMMAND test2 )
Am I correctly understanding what you have in mind? If so, can you
think of a way to automate this a little better?
On Tue, Jan 27, 2015 at 7:18 AM, David Cole wrote:
It's only a chicken and egg problem if you constrain yourself artificially.
When you define a new unit test source file, there are two things you
need to do:
(1) add it to its executable
(2) write an add_test line that invokes that specific unit test
You could possibly encapsulate those actions into an add_unit_test
macro so that it only seems like you have one thing to do, but
certainly, it's possible.
You must be doing #1 already, so you just have to make sure #2 always
happens whenever #1 happens.
Both eggs.
D
On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey wrote:
I believe so, but then you run into the chicken and egg problem:
I need to fully build all unit tests in order to know all the test
cases, but I need to define the tests at the generation phase (before
build).
I'm just not sure how to handle this. The best i can think of is to
generate 1 test executable per CPP file that we write in our unit
tests (generally 1 CPP file per class that is unit tested). Any ideas?
On Mon, Jan 26, 2015 at 10:05 AM, David Cole wrote:
Can you run a command that executes one test case at a time, or do you
have to run all 50 when you execute the tests for a library?
ctest just runs whatever you give it with add_test -- if you want to
filter a collection of unit tests to run only a single unit test, then
the unit test framework you're using would have to support that. Does
Catch allow you to run the tests individually?
On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey
wrote:
Hi,
I'm using Catch as my unit test framework:
https://github.com/philsquared/Catch
Is it possible for CTest to report each TEST_CASE block as an
individual test? My understanding is that CTest will only treat each
executable as a test. However, my test structure is as follows:
1 library
1 test project
Each test project has 1 test CPP file per each class in the library.
This way I implement tests for classes in a corresponding CPP file.
Each CPP file contains multiple test cases (
Re: [CMake] CTest + Catch Framework
The best thing to do would be to make CTest aware of unit test
frameworks like this, and have it be automatic.
Support for cppunit, gtest and Catch (and any other popular ones)
would all be appreciated by many, I'm sure.
Got time to dig into ctest a little bit?
D
On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey
wrote:
> I wouldn't say this is an artificial problem. let me go into more detail.
>
> A single test can define any number of test cases. These are unknown
> to CMake scripts, as they are written and defined in code. The ideal
> solution is to have the test program output a list of all test cases
> and based on that list, automatically generate the add_test commands.
> That way I don't have to worry about hard-coding test case names in
> the build scripts. When a new test case is added to the test code (no
> new files would be added), "magically" a new unit test appears to
> CMake.
>
> But none of this can be done unless the unit test executables are
> built. Thus the chicken and egg problem. To do it as you suggest would
> require me to hard-code the test cases. Example:
>
> In MyTest.cpp:
>
> #include
> TEST_CASE("test1") {}
> TEST_CASE("test2") {}
>
> In CMakeLists.txt:
>
> add_executable( MyTest MyTest.cpp )
> add_test( NAME MyTest_test1 COMMAND test1 )
> add_test( NAME MyTest_test2 COMMAND test2 )
>
> Am I correctly understanding what you have in mind? If so, can you
> think of a way to automate this a little better?
>
> On Tue, Jan 27, 2015 at 7:18 AM, David Cole wrote:
>> It's only a chicken and egg problem if you constrain yourself artificially.
>>
>> When you define a new unit test source file, there are two things you
>> need to do:
>> (1) add it to its executable
>> (2) write an add_test line that invokes that specific unit test
>>
>> You could possibly encapsulate those actions into an add_unit_test
>> macro so that it only seems like you have one thing to do, but
>> certainly, it's possible.
>>
>> You must be doing #1 already, so you just have to make sure #2 always
>> happens whenever #1 happens.
>>
>> Both eggs.
>>
>>
>> D
>>
>>
>>
>>
>> On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey
>> wrote:
>>> I believe so, but then you run into the chicken and egg problem:
>>>
>>> I need to fully build all unit tests in order to know all the test
>>> cases, but I need to define the tests at the generation phase (before
>>> build).
>>>
>>> I'm just not sure how to handle this. The best i can think of is to
>>> generate 1 test executable per CPP file that we write in our unit
>>> tests (generally 1 CPP file per class that is unit tested). Any ideas?
>>>
>>> On Mon, Jan 26, 2015 at 10:05 AM, David Cole wrote:
Can you run a command that executes one test case at a time, or do you
have to run all 50 when you execute the tests for a library?
ctest just runs whatever you give it with add_test -- if you want to
filter a collection of unit tests to run only a single unit test, then
the unit test framework you're using would have to support that. Does
Catch allow you to run the tests individually?
On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey
wrote:
> Hi,
>
> I'm using Catch as my unit test framework:
> https://github.com/philsquared/Catch
>
> Is it possible for CTest to report each TEST_CASE block as an
> individual test? My understanding is that CTest will only treat each
> executable as a test. However, my test structure is as follows:
>
> 1 library
> 1 test project
>
> Each test project has 1 test CPP file per each class in the library.
> This way I implement tests for classes in a corresponding CPP file.
>
> Each CPP file contains multiple test cases (defined by TEST_CASE macro).
>
> The resulting output of `ctest -T Test` shows only 1 test, even though
> that may be around 50 test cases. I'd like CMake to show the pass/fail
> status of each one. Is this possible?
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.
Re: [CMake] CTest + Catch Framework
I wouldn't say this is an artificial problem. let me go into more detail.
A single test can define any number of test cases. These are unknown
to CMake scripts, as they are written and defined in code. The ideal
solution is to have the test program output a list of all test cases
and based on that list, automatically generate the add_test commands.
That way I don't have to worry about hard-coding test case names in
the build scripts. When a new test case is added to the test code (no
new files would be added), "magically" a new unit test appears to
CMake.
But none of this can be done unless the unit test executables are
built. Thus the chicken and egg problem. To do it as you suggest would
require me to hard-code the test cases. Example:
In MyTest.cpp:
#include
TEST_CASE("test1") {}
TEST_CASE("test2") {}
In CMakeLists.txt:
add_executable( MyTest MyTest.cpp )
add_test( NAME MyTest_test1 COMMAND test1 )
add_test( NAME MyTest_test2 COMMAND test2 )
Am I correctly understanding what you have in mind? If so, can you
think of a way to automate this a little better?
On Tue, Jan 27, 2015 at 7:18 AM, David Cole wrote:
> It's only a chicken and egg problem if you constrain yourself artificially.
>
> When you define a new unit test source file, there are two things you
> need to do:
> (1) add it to its executable
> (2) write an add_test line that invokes that specific unit test
>
> You could possibly encapsulate those actions into an add_unit_test
> macro so that it only seems like you have one thing to do, but
> certainly, it's possible.
>
> You must be doing #1 already, so you just have to make sure #2 always
> happens whenever #1 happens.
>
> Both eggs.
>
>
> D
>
>
>
>
> On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey
> wrote:
>> I believe so, but then you run into the chicken and egg problem:
>>
>> I need to fully build all unit tests in order to know all the test
>> cases, but I need to define the tests at the generation phase (before
>> build).
>>
>> I'm just not sure how to handle this. The best i can think of is to
>> generate 1 test executable per CPP file that we write in our unit
>> tests (generally 1 CPP file per class that is unit tested). Any ideas?
>>
>> On Mon, Jan 26, 2015 at 10:05 AM, David Cole wrote:
>>> Can you run a command that executes one test case at a time, or do you
>>> have to run all 50 when you execute the tests for a library?
>>>
>>> ctest just runs whatever you give it with add_test -- if you want to
>>> filter a collection of unit tests to run only a single unit test, then
>>> the unit test framework you're using would have to support that. Does
>>> Catch allow you to run the tests individually?
>>>
>>>
>>>
>>> On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey
>>> wrote:
Hi,
I'm using Catch as my unit test framework:
https://github.com/philsquared/Catch
Is it possible for CTest to report each TEST_CASE block as an
individual test? My understanding is that CTest will only treat each
executable as a test. However, my test structure is as follows:
1 library
1 test project
Each test project has 1 test CPP file per each class in the library.
This way I implement tests for classes in a corresponding CPP file.
Each CPP file contains multiple test cases (defined by TEST_CASE macro).
The resulting output of `ctest -T Test` shows only 1 test, even though
that may be around 50 test cases. I'd like CMake to show the pass/fail
status of each one. Is this possible?
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] CTest + Catch Framework
Hello, This looks a little like google test. For a project I just created a CMake macro which grepped through the test files for the test cases and adds them one by one. Certainly features could be added to better support text fixtures. Hope this gives you some inspiration: https://github.com/SimpleITK/SimpleITK/blob/7aeb952d25c0142e66087edce89e708db2d59a1a/Testing/Unit/CMakeLists.txt#L270-L310 Brad On Jan 27, 2015, at 8:18 AM, David Cole via CMake wrote: > It's only a chicken and egg problem if you constrain yourself artificially. > > When you define a new unit test source file, there are two things you > need to do: > (1) add it to its executable > (2) write an add_test line that invokes that specific unit test > > You could possibly encapsulate those actions into an add_unit_test > macro so that it only seems like you have one thing to do, but > certainly, it's possible. > > You must be doing #1 already, so you just have to make sure #2 always > happens whenever #1 happens. > > Both eggs. > > > D > > > > > On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey > wrote: >> I believe so, but then you run into the chicken and egg problem: >> >> I need to fully build all unit tests in order to know all the test >> cases, but I need to define the tests at the generation phase (before >> build). >> >> I'm just not sure how to handle this. The best i can think of is to >> generate 1 test executable per CPP file that we write in our unit >> tests (generally 1 CPP file per class that is unit tested). Any ideas? >> >> On Mon, Jan 26, 2015 at 10:05 AM, David Cole wrote: >>> Can you run a command that executes one test case at a time, or do you >>> have to run all 50 when you execute the tests for a library? >>> >>> ctest just runs whatever you give it with add_test -- if you want to >>> filter a collection of unit tests to run only a single unit test, then >>> the unit test framework you're using would have to support that. Does >>> Catch allow you to run the tests individually? >>> >>> >>> >>> On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey >>> wrote: Hi, I'm using Catch as my unit test framework: https://github.com/philsquared/Catch Is it possible for CTest to report each TEST_CASE block as an individual test? My understanding is that CTest will only treat each executable as a test. However, my test structure is as follows: 1 library 1 test project Each test project has 1 test CPP file per each class in the library. This way I implement tests for classes in a corresponding CPP file. Each CPP file contains multiple test cases (defined by TEST_CASE macro). The resulting output of `ctest -T Test` shows only 1 test, even though that may be around 50 test cases. I'd like CMake to show the pass/fail status of each one. Is this possible? -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] CTest + Catch Framework
It's only a chicken and egg problem if you constrain yourself artificially. When you define a new unit test source file, there are two things you need to do: (1) add it to its executable (2) write an add_test line that invokes that specific unit test You could possibly encapsulate those actions into an add_unit_test macro so that it only seems like you have one thing to do, but certainly, it's possible. You must be doing #1 already, so you just have to make sure #2 always happens whenever #1 happens. Both eggs. D On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey wrote: > I believe so, but then you run into the chicken and egg problem: > > I need to fully build all unit tests in order to know all the test > cases, but I need to define the tests at the generation phase (before > build). > > I'm just not sure how to handle this. The best i can think of is to > generate 1 test executable per CPP file that we write in our unit > tests (generally 1 CPP file per class that is unit tested). Any ideas? > > On Mon, Jan 26, 2015 at 10:05 AM, David Cole wrote: >> Can you run a command that executes one test case at a time, or do you >> have to run all 50 when you execute the tests for a library? >> >> ctest just runs whatever you give it with add_test -- if you want to >> filter a collection of unit tests to run only a single unit test, then >> the unit test framework you're using would have to support that. Does >> Catch allow you to run the tests individually? >> >> >> >> On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey >> wrote: >>> Hi, >>> >>> I'm using Catch as my unit test framework: >>> https://github.com/philsquared/Catch >>> >>> Is it possible for CTest to report each TEST_CASE block as an >>> individual test? My understanding is that CTest will only treat each >>> executable as a test. However, my test structure is as follows: >>> >>> 1 library >>> 1 test project >>> >>> Each test project has 1 test CPP file per each class in the library. >>> This way I implement tests for classes in a corresponding CPP file. >>> >>> Each CPP file contains multiple test cases (defined by TEST_CASE macro). >>> >>> The resulting output of `ctest -T Test` shows only 1 test, even though >>> that may be around 50 test cases. I'd like CMake to show the pass/fail >>> status of each one. Is this possible? >>> -- >>> >>> Powered by www.kitware.com >>> >>> Please keep messages on-topic and check the CMake FAQ at: >>> http://www.cmake.org/Wiki/CMake_FAQ >>> >>> Kitware offers various services to support the CMake community. For more >>> information on each offering, please visit: >>> >>> CMake Support: http://cmake.org/cmake/help/support.html >>> CMake Consulting: http://cmake.org/cmake/help/consulting.html >>> CMake Training Courses: http://cmake.org/cmake/help/training.html >>> >>> Visit other Kitware open-source projects at >>> http://www.kitware.com/opensource/opensource.html >>> >>> Follow this link to subscribe/unsubscribe: >>> http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] CTest + Catch Framework
I believe so, but then you run into the chicken and egg problem: I need to fully build all unit tests in order to know all the test cases, but I need to define the tests at the generation phase (before build). I'm just not sure how to handle this. The best i can think of is to generate 1 test executable per CPP file that we write in our unit tests (generally 1 CPP file per class that is unit tested). Any ideas? On Mon, Jan 26, 2015 at 10:05 AM, David Cole wrote: > Can you run a command that executes one test case at a time, or do you > have to run all 50 when you execute the tests for a library? > > ctest just runs whatever you give it with add_test -- if you want to > filter a collection of unit tests to run only a single unit test, then > the unit test framework you're using would have to support that. Does > Catch allow you to run the tests individually? > > > > On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey > wrote: >> Hi, >> >> I'm using Catch as my unit test framework: >> https://github.com/philsquared/Catch >> >> Is it possible for CTest to report each TEST_CASE block as an >> individual test? My understanding is that CTest will only treat each >> executable as a test. However, my test structure is as follows: >> >> 1 library >> 1 test project >> >> Each test project has 1 test CPP file per each class in the library. >> This way I implement tests for classes in a corresponding CPP file. >> >> Each CPP file contains multiple test cases (defined by TEST_CASE macro). >> >> The resulting output of `ctest -T Test` shows only 1 test, even though >> that may be around 50 test cases. I'd like CMake to show the pass/fail >> status of each one. Is this possible? >> -- >> >> Powered by www.kitware.com >> >> Please keep messages on-topic and check the CMake FAQ at: >> http://www.cmake.org/Wiki/CMake_FAQ >> >> Kitware offers various services to support the CMake community. For more >> information on each offering, please visit: >> >> CMake Support: http://cmake.org/cmake/help/support.html >> CMake Consulting: http://cmake.org/cmake/help/consulting.html >> CMake Training Courses: http://cmake.org/cmake/help/training.html >> >> Visit other Kitware open-source projects at >> http://www.kitware.com/opensource/opensource.html >> >> Follow this link to subscribe/unsubscribe: >> http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] CTest + Catch Framework
Can you run a command that executes one test case at a time, or do you have to run all 50 when you execute the tests for a library? ctest just runs whatever you give it with add_test -- if you want to filter a collection of unit tests to run only a single unit test, then the unit test framework you're using would have to support that. Does Catch allow you to run the tests individually? On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey wrote: > Hi, > > I'm using Catch as my unit test framework: > https://github.com/philsquared/Catch > > Is it possible for CTest to report each TEST_CASE block as an > individual test? My understanding is that CTest will only treat each > executable as a test. However, my test structure is as follows: > > 1 library > 1 test project > > Each test project has 1 test CPP file per each class in the library. > This way I implement tests for classes in a corresponding CPP file. > > Each CPP file contains multiple test cases (defined by TEST_CASE macro). > > The resulting output of `ctest -T Test` shows only 1 test, even though > that may be around 50 test cases. I'd like CMake to show the pass/fail > status of each one. Is this possible? > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
