On 9/15/2017 8:22 AM, Brad King wrote:
On 09/14/2017 11:33 PM, Edward Diener wrote:
Boost Build has tests for running an application successfully or not,
for compiling one or more source files successfully or not, and for
building one or more source files into an exe or not. These tests in
Boost Build are the run/run-fail, compile/compile-fail, and
link/link-fail rules.
Good summary. One major difference is that Boost Build directly controls
the entire build graph and is natively aware of expected failure cases.
CMake needs to generate an external build system e.g. as Visual Studio
`.vcxproj` files with no good way to represent expected failure directly.
This is a technical reason CMake has never gained this functionality,
but perhaps it could be overcome with a careful design. See below.
I am a relative beginner with CMake although pretty knowledgeable about
Boost Build and Visual Studio, so please take my comments based on those
criteria. A gui environment like Visual Studio does provide
functionality to just compile one or more source files, and it also
provides functionality to just build an executable from one or more
source files without actually running that executable. I do not know how
CMake "picks up" the results of its add_test when the build engine is
Visual Studio, but however it does so it seems it could also "pick up"
the result of a compile or of a compile/link in Visual Studio also,
although I do not know how difficult this would be to do.
CMake/CTest has the exact equivalent to the run/run-fail rule in its
add_test framework
While `add_test` can do run/run-fail it doesn't do so given just a
source file like Boost Build does. `add_test` expects to be given a
command referring to some executable that already exists or is built
elsewhere in the project via `add_executable`. Therefore some work
would be needed even to reproduce Boost's run/run-fail infrastructure.
Boost Build's version of 'run/run-fail' does build an executable from
the source file(s) that are part of its 'run/run-fail' rule. Giving a
target name for that executable is optional, else the target name is
taken from the first source file mentioned. I am not asking that CMake's
'add_test' completely duplicate Boost Build's 'run/run-fail' rules and I
do not mind the way that CMake works using an 'add_executable' as long
as the 'add_executable' for testing can be turned off as a normal build
target. In general I am not asking that CMake completely duplicate
exactly Boost Build in these testing scenarios, but that it offer
similar functionality so that the basic functionality of the Boost Build
'run', 'compile', and 'link' rules have a CMake equivalent. I think the
current 'add_executable', 'add_test' is a good enough equivalent to
Boost Build's 'run/run-fail' rule. I would just like to see a built-in
equivalent to testing under CMake which involves only a successful
compilation or not, equivalent to Boost Build's 'compile/compile-fail'
rule and a built-in equivalent to testing which involves only a
successful build or not of a non-build target equivalent to Boost
Build's 'link/link-fail' rule. I think both of the latter are valid
testing methodologies which do not involve an actual run-time test.
Hopefully CMake developers will get the message and make the necessary
update to CMake/CTest.
We understand the value of the tests but simply don't have the need for
them in our own use cases. This is a non-technical reason CMake does
not have this functionality, but it doesn't stand in the way of others
contributing a solution.
Who are "we" in your previous paragraph ? I believe that you as a
developer understand that compile-time testing is a reality in the C++
world of programming, although I am not sure what other computer
languages CMake supports and if compile-time testing exists in those
other languages. Outside of Boost, which pushes the bounds of C++
library development and for whom, due to Boost's basis of heavily
relying on template programming, and where compile-time testing has long
been a reality, other C++ programmers have occasionally mentioned a need
for compile-time testing under CMake to be added to CMake's run-time
testing infrastructure. I am just suggesting that this should not be a
popularity contest but just a recognition that compile-time testing
should be supported by CMake within the same generally testing framework
in which CMake supports run-time testing.
I freely admit that I am a basically a CMake beginner even as an
end-user, and certainly not a CMake developer and do not know what
adding compile-time testing would entail at the CMake developer level.
But as a Boost library contributor, who has designed compile-time tests
for my own supported libraries in Boost, and who has seen compile-time
tests for many other Boost libraries, the fact that Boost Build contains
support for compile-time testing and CMake does not is disconcerting to
me if Boost is going to successfully move from Boost Build to CMake.
Is there any consensus that these other two types of tests might be
valuable for CMake/CTest, or any way to make this happen ?
I agree they would be useful.
From some quick brainstorming, one idea to deal with the technical
challenge raised above is to add a new "BUILD_TEST" target property
to tell CMake that a target is meant only to be built during testing.
We allow dependencies among BUILD_TEST targets on each other and on
normal targets, but not from normal targets to BUILD_TEST targets.
That will give a separable part of the build graph to generate only
for testing. Then add_test can create tests that try to build one
of the BUILD_TEST targets and expect one of the compile/compile-fail,
link/link-fail, or run/run-fail cases.
For generators like Visual Studio we could lave the BUILD_TEST
targets out of the main `.sln` file so they wouldn't show up when
users load the projects. Instead they would be placed in a second
`.sln` file used only for testing.
Further design work is likely needed but this is a start.
---------------------------------------------------------------------
IMO it will be better to first prototype this externally to gain some
experience with it and work with existing CMake releases. One can do
this by using the `export()` command to make all targets in the build
tree available to externally-built projects (which is useful on its own).
With that a few approaches can be used to drive testing as an external
build.
Then have a custom target that configures the test-only part of the
project externally but doesn't build it. The test-only project is
externally-built and can load the targets file created by `export()`
above. Then use `add_test` to create tests that drive the build on
parts of the external test-only project and check for appropriate
success/failure and output matching.
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
