Re: [cmake-developers] Adding compile and build type tests to CMake/CTest
On 9/18/2017 8:16 AM, Brad King wrote: On 09/15/2017 09:22 AM, Edward Diener wrote: A gui environment like Visual Studio does provide functionality to just compile one or more source files, and it also provides functionality to just build an executable from one or more source files without actually running that executable. Yes. We just need it to be generated separately from the main build system so that "Build Solution" or our ALL_BUILD.vcxproj don't try to build the tests. Some work will be needed on the CMake side to do this in a built-in way. Do you happen to know if/how it is possible to take a .vcxproj with multiple source files (ClCompile entries) and use MSBuild to compile just one of the sources from the command-line? The IDE must do it somehow with the right-click "Compile" option. Similarly in Xcode? I do not know much about MSBuild since I have normally used the IDE. I would just like to see a built-in equivalent to testing under CMake which involves only a successful compilation or not, equivalent to Boost Build's 'compile/compile-fail' rule and a built-in equivalent to testing which involves only a successful build or not of a non-build target equivalent to Boost Build's 'link/link-fail' rule. I think both of the latter are valid testing methodologies which do not involve an actual run-time test. Yes, having a built-in solution would be nice. Currently these can only be achieved indirectly by writing tests that configure projects that are not configured by CMake as part of the main project. BTW, where is the corresponding discussion of this problem on the Boost side taking place? When the original announcement from the Boost Steering Committee was made a few months back that Boost would be moving to CMake there were numerous discussions about converting jam files to CMake and how this can be done, largely on the Boost developers mailing list. I brought up issues about how to duplicate Boost Build testing under CMake. Since then there has been little further discussion about this or indication by the Boost Steering Committee how this was to be done, on the developers mailing list. One implementation popularly considered is on Github as https://github.com/boost-cmake/bcm, but that implementation of a pure compile test is a kludge IMO, which led to this discussion here. Thanks, -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Adding compile and build type tests to CMake/CTest
On 09/15/2017 09:22 AM, Edward Diener wrote: > A gui environment like Visual Studio does provide > functionality to just compile one or more source files, and it also > provides functionality to just build an executable from one or more > source files without actually running that executable. Yes. We just need it to be generated separately from the main build system so that "Build Solution" or our ALL_BUILD.vcxproj don't try to build the tests. Some work will be needed on the CMake side to do this in a built-in way. Do you happen to know if/how it is possible to take a .vcxproj with multiple source files (ClCompile entries) and use MSBuild to compile just one of the sources from the command-line? The IDE must do it somehow with the right-click "Compile" option. Similarly in Xcode? > I would just like to see a built-in > equivalent to testing under CMake which involves only a successful > compilation or not, equivalent to Boost Build's 'compile/compile-fail' > rule and a built-in equivalent to testing which involves only a > successful build or not of a non-build target equivalent to Boost > Build's 'link/link-fail' rule. I think both of the latter are valid > testing methodologies which do not involve an actual run-time test. Yes, having a built-in solution would be nice. Currently these can only be achieved indirectly by writing tests that configure projects that are not configured by CMake as part of the main project. BTW, where is the corresponding discussion of this problem on the Boost side taking place? Thanks, -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Adding compile and build type tests to CMake/CTest
On 9/15/2017 8:22 AM, Brad King wrote: On 09/14/2017 11:33 PM, Edward Diener wrote: Boost Build has tests for running an application successfully or not, for compiling one or more source files successfully or not, and for building one or more source files into an exe or not. These tests in Boost Build are the run/run-fail, compile/compile-fail, and link/link-fail rules. Good summary. One major difference is that Boost Build directly controls the entire build graph and is natively aware of expected failure cases. CMake needs to generate an external build system e.g. as Visual Studio `.vcxproj` files with no good way to represent expected failure directly. This is a technical reason CMake has never gained this functionality, but perhaps it could be overcome with a careful design. See below. I am a relative beginner with CMake although pretty knowledgeable about Boost Build and Visual Studio, so please take my comments based on those criteria. A gui environment like Visual Studio does provide functionality to just compile one or more source files, and it also provides functionality to just build an executable from one or more source files without actually running that executable. I do not know how CMake "picks up" the results of its add_test when the build engine is Visual Studio, but however it does so it seems it could also "pick up" the result of a compile or of a compile/link in Visual Studio also, although I do not know how difficult this would be to do. CMake/CTest has the exact equivalent to the run/run-fail rule in its add_test framework While `add_test` can do run/run-fail it doesn't do so given just a source file like Boost Build does. `add_test` expects to be given a command referring to some executable that already exists or is built elsewhere in the project via `add_executable`. Therefore some work would be needed even to reproduce Boost's run/run-fail infrastructure. Boost Build's version of 'run/run-fail' does build an executable from the source file(s) that are part of its 'run/run-fail' rule. Giving a target name for that executable is optional, else the target name is taken from the first source file mentioned. I am not asking that CMake's 'add_test' completely duplicate Boost Build's 'run/run-fail' rules and I do not mind the way that CMake works using an 'add_executable' as long as the 'add_executable' for testing can be turned off as a normal build target. In general I am not asking that CMake completely duplicate exactly Boost Build in these testing scenarios, but that it offer similar functionality so that the basic functionality of the Boost Build 'run', 'compile', and 'link' rules have a CMake equivalent. I think the current 'add_executable', 'add_test' is a good enough equivalent to Boost Build's 'run/run-fail' rule. I would just like to see a built-in equivalent to testing under CMake which involves only a successful compilation or not, equivalent to Boost Build's 'compile/compile-fail' rule and a built-in equivalent to testing which involves only a successful build or not of a non-build target equivalent to Boost Build's 'link/link-fail' rule. I think both of the latter are valid testing methodologies which do not involve an actual run-time test. Hopefully CMake developers will get the message and make the necessary update to CMake/CTest. We understand the value of the tests but simply don't have the need for them in our own use cases. This is a non-technical reason CMake does not have this functionality, but it doesn't stand in the way of others contributing a solution. Who are "we" in your previous paragraph ? I believe that you as a developer understand that compile-time testing is a reality in the C++ world of programming, although I am not sure what other computer languages CMake supports and if compile-time testing exists in those other languages. Outside of Boost, which pushes the bounds of C++ library development and for whom, due to Boost's basis of heavily relying on template programming, and where compile-time testing has long been a reality, other C++ programmers have occasionally mentioned a need for compile-time testing under CMake to be added to CMake's run-time testing infrastructure. I am just suggesting that this should not be a popularity contest but just a recognition that compile-time testing should be supported by CMake within the same generally testing framework in which CMake supports run-time testing. I freely admit that I am a basically a CMake beginner even as an end-user, and certainly not a CMake developer and do not know what adding compile-time testing would entail at the CMake developer level. But as a Boost library contributor, who has designed compile-time tests for my own supported libraries in Boost, and who has seen compile-time tests for many other Boost libraries, the fact that Boost Build contains support for compile-time testing and CMake does not is disconcerting
Re: [cmake-developers] Adding compile and build type tests to CMake/CTest
On 09/14/2017 11:33 PM, Edward Diener wrote: > Boost Build has tests for running an application successfully or not, > for compiling one or more source files successfully or not, and for > building one or more source files into an exe or not. These tests in > Boost Build are the run/run-fail, compile/compile-fail, and > link/link-fail rules. Good summary. One major difference is that Boost Build directly controls the entire build graph and is natively aware of expected failure cases. CMake needs to generate an external build system e.g. as Visual Studio `.vcxproj` files with no good way to represent expected failure directly. This is a technical reason CMake has never gained this functionality, but perhaps it could be overcome with a careful design. See below. > CMake/CTest has the exact equivalent to the run/run-fail rule in its > add_test framework While `add_test` can do run/run-fail it doesn't do so given just a source file like Boost Build does. `add_test` expects to be given a command referring to some executable that already exists or is built elsewhere in the project via `add_executable`. Therefore some work would be needed even to reproduce Boost's run/run-fail infrastructure. > Hopefully CMake developers will get the message and make the necessary > update to CMake/CTest. We understand the value of the tests but simply don't have the need for them in our own use cases. This is a non-technical reason CMake does not have this functionality, but it doesn't stand in the way of others contributing a solution. > Is there any consensus that these other two types of tests might be > valuable for CMake/CTest, or any way to make this happen ? I agree they would be useful. >From some quick brainstorming, one idea to deal with the technical challenge raised above is to add a new "BUILD_TEST" target property to tell CMake that a target is meant only to be built during testing. We allow dependencies among BUILD_TEST targets on each other and on normal targets, but not from normal targets to BUILD_TEST targets. That will give a separable part of the build graph to generate only for testing. Then add_test can create tests that try to build one of the BUILD_TEST targets and expect one of the compile/compile-fail, link/link-fail, or run/run-fail cases. For generators like Visual Studio we could lave the BUILD_TEST targets out of the main `.sln` file so they wouldn't show up when users load the projects. Instead they would be placed in a second `.sln` file used only for testing. Further design work is likely needed but this is a start. - IMO it will be better to first prototype this externally to gain some experience with it and work with existing CMake releases. One can do this by using the `export()` command to make all targets in the build tree available to externally-built projects (which is useful on its own). With that a few approaches can be used to drive testing as an external build. Then have a custom target that configures the test-only part of the project externally but doesn't build it. The test-only project is externally-built and can load the targets file created by `export()` above. Then use `add_test` to create tests that drive the build on parts of the external test-only project and check for appropriate success/failure and output matching. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] Adding compile and build type tests to CMake/CTest
I posted these messages to the users list but Bill Hoffman suggested I post them to the developers list: Boost Build has tests for running an application successfully or not, for compiling one or more source files successfully or not, and for building one or more source files into an exe or not. These tests in Boost Build are the run/run-fail, compile/compile-fail, and link/link-fail rules. CMake/CTest has the exact equivalent to the run/run-fail rule in its add_test framework, but there is not add_test equivalent to the other two sets of rules. It sure would be nice, when Boost transitions to using CMake/CTest instead of Boost Build, if CMake/CTest had the equivalent of the other two sets of types of test in its add_test framework. To put it succinctly CMake should adding compile-time testing so that when some compilation succeeds the test is successful and if the compilation fails the test is not successful, with the proviso that you can reverse the result as a compile should fail type of test. Similarly a build type testing, without having to run anything should be added along the same lines. In modern C++ it is perfectly feasible, especially with template programming, to do compile time testing, invoking a compile-time static assert as a compile-time failure. Boost has had this for years and modern C++ has it as part of the latest version of the C++ standard. CMake needs to update itself to the reality that pure compile-time testing is a reality for modern C++ and should update itself accordingly. Only having run-time testing is an artifact of the past. Hopefully CMake developers will get the message and make the necessary update to CMake/CTest. Is there any consensus that these other two types of tests might be valuable for CMake/CTest, or any way to make this happen ? -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
