While I was suspecting something like this about CTest, your explanation helps
a lot.
I have integrated UnitTest++ into my flow, and it is working quite well. It
does also return a 0 if all tests pass, so one could integrate with CTest/CDash
if desirable.
Thanks
Date: Fri, 9 Jul 2010 14:54:46 -0700
From: chillery-cm...@lambda.nu
To: b...@askmonty.org
CC: cmake@cmake.org
Subject: Re: [CMake] CTest examples
For CTest, running a test means running an executable and seeing whether the
result of that process is 0 (success) or not 0 (failure). It is also possible
to scan the output of the executable for certain regular expressions to
determine success or failure.
It's a very simple and somewhat limited mechanism. It does depend on you
writing your own test driver, which makes sense because CMake can't possibly
know how to test your code.
As an example, Zorba is an XQuery (xml query language) processor. The majority
of our tests are actual XQueries stored in text files with the extension .xq,
along with corresponding expected results in .xml files. We have a testdriver
program which executes the query, checks the results, and returns 0 or 1.
Our CMakeLists looks vaguely like this:
add_executable(testdriver ${TESTDRIVER_SRCS})
file(GLOB_RECURSE TESTFILES "${CMAKE_CURRENT_SOURCE_DIR}" "*.xq")
foreach(TESTFILE ${TESTFILES})
add_test(${TESTFILE} testdriver ${TESTFILE})
endforeach()
This creates one test for each .xq file. The name of the test is the name of
the .xq file, and that same name is passed as an argument to testdriver.
This works well, and lets us use ctest's functionality like -R to run certain
tests by name. It also results in nice easy-to-read listings on CDash because
there's one line for each test, making it easy to examine failures. (Although
given that we have over 20,000 tests, some pages on CDash take a looooong time
to load...) The main downside is speed, since every single test has to start up
a whole Zorba process.
When you get to unit testing, however, IMHO things don't work as well. CMake
does offer the create_test_sourcelist() command, and you should look into it.
We use that in a couple places to create a single test driver from several
separate small unit test drivers. However, you still need to write a loop in
CMakeLists to add multiple tests using that test driver.
In another project I work on, I've been using CxxTest for a unit testing
framework. This is a really nice setup as it makes it very trivial to write
many unit tests. However, the integration with ctest is poor. There's a
reasonably good FindCxxTest script shipped with CMake that makes it easy to get
CxxTest suites running; in a nutshell:
find_package(CxxTest)
cxx_add_test(my_unit_test my_unit_test.cpp
"${CMAKE_CURRENT_SOURCE_DIR}/my_unit_test.h")
Unit tests in CxxTest are written as classes in a .h file; cxx_add_test() runs
CxxTest to generate the named .cpp file from that, then builds it into the
named test driver and adds a CTest test for it. So far so good. However, the
main benefit of CxxTest is grouping similar test cases into a single .h file,
probably with similar setup and so forth. Unfortunately cxx_add_test() only
adds one test to CTest, and that one test runs all the test cases in the .h
file. So you lose the ability to run certain tests with ctest -R, and the
reports on CDash will just show whether ALL the tests in my_unit_test succeeded
or not.
The fundamental problem is that ctest requires a test case to be a program
execution, and has no facilities to allow a program to represent multiple
tests. So either you pay the cost in speed, or you lose some of the flexibility
ctest and cdash offer. (To be fair, even if this were fixed in ctest, CxxTest
would also need some modifications as it currently doesn't let you run subsets
of tests either.)
Ceej
aka Chris Hillery
On Fri, Jul 9, 2010 at 4:37 AM, Bo Thorsen <b...@askmonty.org> wrote:
Hi people,
I have converted a set of applications to cmake and cpack, and now have my eyes
set on ctest.
I'd like to hear if someone here has some good advice, or links to good advice,
on how to structure tests. I'm searching for help on how to put different tests
into what executables. On how to handle multiple tests on each classes, on how
to best structure the test of the static libraries (all of those are part of
the source tree) that are linked in to the application. And on how to test
classes from the main application itself.
I have read the ctest FAQ, documentation etc. and still don't know anything
that help me write the actual test code.
>From the looks of it, ctest only provides the framework to run a test, no help
>is given to write the code of the tests themselves, is this right? I have
>previously been using cppunit, and it looks like this will still be useful.
To sum it up, I'm looking for real life advice on what you guys have done with
ctest. This information seem almost completely missing on the net, where all
searches on ctest leads to useless presentation on ctest features.
Cheers,
Bo Thorsen.
Monty Program AB.
--
MariaDB: MySQL replacement
Community developed. Feature enhanced. Backward compatible.
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
