Re: [cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Daniel Pfeifer]

On Wed, Oct 5, 2016 at 12:54 PM, Nils Gladitz  wrote:
> On 04.10.2016 11:20, Tobias Hunger wrote:
>>
>> On Mo, 2016-10-03 at 14:25 -0400, Brad King wrote:

 The list we need would thus be built up to contain all COMMAND
 arguments to add_test that are also build targets in CMake.
>>>
>>> The list would also need to include targets mentioned in generator
>>> expressions such as `$`.  This will almost certainly
>>> be better done as a CMake feature implemented in C++.  This was actually
>>> an early design philosophy: do the hard stuff in C++ so that complex
>>> logic is not needed in CMake-language code.
>>
>> Access to the list of defined tests would be nice to have in the
>> server-mode,
>> too:-)
>>
>> If some work is done in that direction, please keep server-mode in mind
>> when
>> designing the APIs. I would love to reuse them to expose that information
>> to
>> clients.
>
>
> This might be rare but the tests that cmake knows about might differ from
> the tests that ctest knows about given that tests can be dynamically defined
> through TEST_INCLUDE_FILE[1].
>
> I've never used it that way but I was considering e.g. querying test
> binaries (produced at build time) for the tests they define and making those
> available dynamically when ctest processes CTestTestfile.cmake. This is in
> contrast to e.g. FindGTests.cmake's GTEST_ADD_TESTS() which tries to
> accomplish the same thing at configuration time by parsing test sources
> instead.

+1. I wrote about that approach here:
http://purplekarrot.net/blog/cmake-and-test-suites.html
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Nils Gladitz]

On 04.10.2016 11:20, Tobias Hunger wrote:

On Mo, 2016-10-03 at 14:25 -0400, Brad King wrote:

The list we need would thus be built up to contain all COMMAND
arguments to add_test that are also build targets in CMake.

The list would also need to include targets mentioned in generator
expressions such as `$`.  This will almost certainly
be better done as a CMake feature implemented in C++.  This was actually
an early design philosophy: do the hard stuff in C++ so that complex
logic is not needed in CMake-language code.

Access to the list of defined tests would be nice to have in the server-mode,
too:-)

If some work is done in that direction, please keep server-mode in mind when
designing the APIs. I would love to reuse them to expose that information to
clients.


This might be rare but the tests that cmake knows about might differ 
from the tests that ctest knows about given that tests can be 
dynamically defined through TEST_INCLUDE_FILE[1].


I've never used it that way but I was considering e.g. querying test 
binaries (produced at build time) for the tests they define and making 
those available dynamically when ctest processes CTestTestfile.cmake. 
This is in contrast to e.g. FindGTests.cmake's GTEST_ADD_TESTS() which 
tries to accomplish the same thing at configuration time by parsing test 
sources instead.


[1] https://cmake.org/cmake/help/v3.6/prop_dir/TEST_INCLUDE_FILE.html

Nils
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Tobias Hunger]

On Mo, 2016-10-03 at 14:25 -0400, Brad King wrote:
> > The list we need would thus be built up to contain all COMMAND
> > arguments to add_test that are also build targets in CMake.
> 
> The list would also need to include targets mentioned in generator
> expressions such as `$`.  This will almost certainly
> be better done as a CMake feature implemented in C++.  This was actually
> an early design philosophy: do the hard stuff in C++ so that complex
> logic is not needed in CMake-language code.

Access to the list of defined tests would be nice to have in the server-mode,
too:-)

If some work is done in that direction, please keep server-mode in mind when
designing the APIs. I would love to reuse them to expose that information to
clients.

Best Regards,
Tobias

-- 
Tobias Hunger, Senior Software Engineer | The Qt Company
The Qt Company GmbH, Rudower Chaussee 13, D-12489 Berlin
Geschäftsführer: Mika Pälsi, Juha Varelius, Mika Harjuaho. Sitz der
Gesellschaft: Berlin, Registergericht: Amtsgericht Charlottenburg, HRB 144331 B
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Clifford Yapp]

On Mon, Oct 3, 2016 at 2:25 PM, Brad King  wrote:
> On 10/03/2016 12:14 PM, Clifford Yapp wrote:
>> so what we need is not actually the test names themselves,
>> but a list of the executables used to run tests that are also defined
>> as build targets.
> [snip]
>> The list we need would thus be built up to contain all COMMAND
>> arguments to add_test that are also build targets in CMake.
>
> The list would also need to include targets mentioned in generator
> expressions such as `$`.  This will almost certainly
> be better done as a CMake feature implemented in C++.  This was actually
> an early design philosophy: do the hard stuff in C++ so that complex
> logic is not needed in CMake-language code.

Sounds like a plan.  I must confess I'm not familiar with the use of
generator expressions yet (we've been holding off requiring CMake 3,
but I see Debian stable has now updated so it's probably time) but
I'll start catching up on my homework since I think they'll be needed
to "modernize" some of our features that are still using the LOCATION
property.

>> dependency of the custom target "check".  Then, when the custom target
>> "check" executes, it will first make sure all of the test executables
>> (and their dependencies) have  been built.
>
> Why do you need a "check" target instead of the CMake-defined "test"
> target?

Primarily because we have experimental CTest targets that are in
various stages of development, and not ready for prime time.  check is
the target we use when the test is ready to be used as a determining
factor in whether something has broken.

> One could teach cmGlobalGenerator::AddGlobalTarget_Test an option to add
> a dependency on the "all" target from "test".  Most projects do not have
> so many non-test targets that they cannot afford to build "all" before
> testing.

While we could "afford" to, we have quite a large number of doc
targets (for example) that aren't needed for make test so it would be
a noticeable increase.

> Alternatively the implementation could directly add dependencies
> on targets referenced by tests.  That may be a bit trickier though.

That would be ideal, if it can be done.

Cheers,
CY
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Brad King]

On 10/03/2016 12:14 PM, Clifford Yapp wrote:
> so what we need is not actually the test names themselves,
> but a list of the executables used to run tests that are also defined
> as build targets.
[snip]
> The list we need would thus be built up to contain all COMMAND
> arguments to add_test that are also build targets in CMake.

The list would also need to include targets mentioned in generator
expressions such as `$`.  This will almost certainly
be better done as a CMake feature implemented in C++.  This was actually
an early design philosophy: do the hard stuff in C++ so that complex
logic is not needed in CMake-language code.

> dependency of the custom target "check".  Then, when the custom target
> "check" executes, it will first make sure all of the test executables
> (and their dependencies) have  been built.

Why do you need a "check" target instead of the CMake-defined "test"
target?

One could teach cmGlobalGenerator::AddGlobalTarget_Test an option to add
a dependency on the "all" target from "test".  Most projects do not have
so many non-test targets that they cannot afford to build "all" before
testing.  Alternatively the implementation could directly add dependencies
on targets referenced by tests.  That may be a bit trickier though.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Clifford Yapp]

On Mon, Oct 3, 2016 at 11:21 AM, Brad King  wrote:
> On 10/03/2016 11:17 AM, Clifford Yapp wrote:
>> One of the functions we are wrapping in our build with CMake's "_"
>> prefix debugging mechanism is add_test.  Looking at why we do that, it
>> should be possible to avoid wrapping if we can get access in CMake to
>> the list of all test build targets defined with add_test.
>> Essentially, we need to add them to some custom target dependencies,
>> and it should be OK to do that after all tests are added if we can get
>> access to the list of targets within CMake.
>>
>> Is there a property somewhere that exposes this list?
>
> I don't recall one off the top of my head.  Even if there were,
> the `set_property(TEST ...)` only works on tests in the invoking
> directory.  There is no global scope for test names.

Hmm.  OK, so what we need is not actually the test names themselves,
but a list of the executables used to run tests that are also defined
as build targets.  So, for example, we have a test that is named
bu_basename_usr that runs the executable test_bu_basename.  What we
actually need is for the build target test_bu_basename to be a
dependency of the custom target "check".  Then, when the custom target
"check" executes, it will first make sure all of the text executables
(and their dependencies) have  been built.

The list we need would thus be built up to contain all COMMAND
arguments to add_test that are also build targets in CMake.  Does that
make sense?

> Some projects use a `myproj_add_test()` wrapper around their
> `add_test()` calls specifically to provide customizations.

I suspect that would run into issues similar to those we're seeing now
trying to pass empty strings from the wrapper to the children, unless
perhaps a macro wrapper would work where a function wrapper would
not...:  https://gitlab.kitware.com/cmake/cmake/issues/16341

Cheers,
CY
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Brad King]

On 10/03/2016 11:17 AM, Clifford Yapp wrote:
> One of the functions we are wrapping in our build with CMake's "_"
> prefix debugging mechanism is add_test.  Looking at why we do that, it
> should be possible to avoid wrapping if we can get access in CMake to
> the list of all test build targets defined with add_test.
> Essentially, we need to add them to some custom target dependencies,
> and it should be OK to do that after all tests are added if we can get
> access to the list of targets within CMake.
> 
> Is there a property somewhere that exposes this list?

I don't recall one off the top of my head.  Even if there were,
the `set_property(TEST ...)` only works on tests in the invoking
directory.  There is no global scope for test names.

Some projects use a `myproj_add_test()` wrapper around their
`add_test()` calls specifically to provide customizations.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] Is there a way in CMake to get a list of all CTest tests added with add_test?

[Clifford Yapp]

One of the functions we are wrapping in our build with CMake's "_"
prefix debugging mechanism is add_test.  Looking at why we do that, it
should be possible to avoid wrapping if we can get access in CMake to
the list of all test build targets defined with add_test.
Essentially, we need to add them to some custom target dependencies,
and it should be OK to do that after all tests are added if we can get
access to the list of targets within CMake.

Is there a property somewhere that exposes this list?

Thanks,
CY
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers