Re: [cmake-developers] [CMake] Setup/tear down steps for CTest
Am Dienstag, 23. August 2016, 10:06:01 schrieb Craig Scott: > Cheeky way to get me more involved in contributing, but okay, I'll bite. ;) > Switching discussion to the dev list. > > So how would you want the feature to work? I'd suggest an initial set of > requirements something like the following: > >- Need to support the ability to define multiple setup and/or tear down >tasks. >- It should be possible to specify dependencies between setup tasks and >between tear down tasks. >- Individual tests need to be able to indicate which setup and/or tear >down tasks they require, similar to the way DEPENDS is used to specify >dependencies between test cases. >- When using ctest --rerun-failed, ctest should automatically invoke any >setup or tear down tasks required by the test cases that will be re-run. >- Setup or tear down tasks which reference executable targets should >substitute the actual built executable just like how add_custom_command() > does. -need a way to mark if 2 tests with the same setup/teardown can share those or if they need to run for every of them -the default for each test is "no s/t", which means it can't be run with any of the above in parallel (especially for compatibillity)[1] -need a way to tell if a test doesn't care about those 1) think of a database connector test: the test that will check what happens if no DB is present will fail if the setup step "start DB" was run, but not the teardown > Some open questions: > >- Should setup and tear down tasks be defined in pairs, or should they >completely independent (this would still require the ability to specify a > dependency of a tear down task on a setup task)? The test could be "shutdown daemon" or "delete files" so I would keep them separated. They may be created by the same command, so they could be batched anyway. >- Should the setup and tear down tasks be defined by a new CTest/CMake >command or extend an existing mechanism (e.g. add_custom_command())? Don't clutter existing commands more than needed. If it's something new, then create a new command (they could still share C++ code). If it's basically the same as anything existing at the end then use that. >- If no test case has a dependency on a setup or tear down task, should >that task be skipped? Perhaps tasks need to have a flag which indicates >whether they always run or only if a test case depends on it. Keep backward compatibility. I.e. if I now add a new test with s/t, then every other test should still run (and succeed) as before. >- What terminology to use? Things like GoogleTest use terms like test >*fixtures* for this sort of thing. The terms setup and tear down are a >bit imprecise and cumbersome, so we would probably need something better >than those. >- Would it make sense for the ctest command line to support disabling >setup and/or tear down steps? I can see some potential scenarios where > this may be desirable, but maybe this is getting too ambitious for a > starting set of requirements. IMHO that doesn't make sense. One could think about an option to disable the s/t merging, i.e. that they are really called alone for every test. >- What should happen if a setup or tear down task fails? How would >failure be detected? How would such failures impact things like a CDash >test report, etc.? Then the test fails, just the same as it now does when it can't find the executable. Eike signature.asc Description: This is a digitally signed message part. -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Unix makefile: concurrent build configurations
Hi Steve! On Do, 2016-08-18 at 15:37 -0500, Steve Lorimer wrote: > New to this list, so apologies if it's been discussed before. > > I'd like to know why it's not possible to have both debug and release mode > builds configured at the same time for Unix makefiles? You can have an arbitrary number of configurations at once, all nicely separated from each other in their own build directory. Nobody stops you from having several build directories after all. > I am aware that some generators (eg VisualStudio) do allow this behaviour. > Why can I not have the same behaviour with Unix makefiles? IMHO VisualStudio is rather broken in that regard. Please do not let the insanity pass over to Makefiles! :-) Best Regards, Tobias -- Tobias Hunger, Senior Software Engineer | The Qt Company The Qt Company GmbH, Rudower Chaussee 13, D-12489 Berlin Geschäftsführer: Mika Pälsi, Juha Varelius, Mika Harjuaho. Sitz der Gesellschaft: Berlin, Registergericht: Amtsgericht Charlottenburg, HRB 144331 B -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] [CMake] Setup/tear down steps for CTest
Cheeky way to get me more involved in contributing, but okay, I'll bite. ;) Switching discussion to the dev list. So how would you want the feature to work? I'd suggest an initial set of requirements something like the following: - Need to support the ability to define multiple setup and/or tear down tasks. - It should be possible to specify dependencies between setup tasks and between tear down tasks. - Individual tests need to be able to indicate which setup and/or tear down tasks they require, similar to the way DEPENDS is used to specify dependencies between test cases. - When using ctest --rerun-failed, ctest should automatically invoke any setup or tear down tasks required by the test cases that will be re-run. - Setup or tear down tasks which reference executable targets should substitute the actual built executable just like how add_custom_command() does. Some open questions: - Should setup and tear down tasks be defined in pairs, or should they completely independent (this would still require the ability to specify a dependency of a tear down task on a setup task)? - Should the setup and tear down tasks be defined by a new CTest/CMake command or extend an existing mechanism (e.g. add_custom_command())? - If no test case has a dependency on a setup or tear down task, should that task be skipped? Perhaps tasks need to have a flag which indicates whether they always run or only if a test case depends on it. - What terminology to use? Things like GoogleTest use terms like test *fixtures* for this sort of thing. The terms setup and tear down are a bit imprecise and cumbersome, so we would probably need something better than those. - Would it make sense for the ctest command line to support disabling setup and/or tear down steps? I can see some potential scenarios where this may be desirable, but maybe this is getting too ambitious for a starting set of requirements. - What should happen if a setup or tear down task fails? How would failure be detected? How would such failures impact things like a CDash test report, etc.? I think that's probably enough to kick off discussions for now. On Sun, Aug 21, 2016 at 11:41 PM, David Cole wrote: > The best thing to do would be to add the feature to ctest, and > contribute to the CMake community. > > I, too, use the "run this test first" and "that test last" technique, > and set up DEPENDS property values to ensure ordering when all tests > are run in parallel. However, as you noted, this does not work to run > subsets of tests reliably. For me, I am able to live with the partial > solution because the vast majority of my tests can be run > independently anyhow and we usually do run all the tests, but a setup > / teardown step for the whole suite would be a welcome addition to > ctest. > > Looking forward to your patch... :-) > > > David C. > > > On Sat, Aug 20, 2016 at 8:32 PM, Craig Scott > wrote: > > Let's say a project defines a bunch of tests which require setup and tear > > down steps before/after all the tests are run (not each individual test, > I'm > > talking here about one setup before all tests are run and one tear down > > after all tests have finished). While this could be done by a script > driving > > CTest itself, it is less desirable since different platforms may need > > different driver scripts and this seems like something CTest should be > able > > to handle itself (if the setup/tear down steps use parts of the build, > that > > only strengthens the case to have them handled by CMake/CTest directly). > > > > It is possible to abuse the DEPENDS test property and define setup and > tear > > down "tests" which are not really tests but which perform the necessary > > steps. While this mostly works, it is not ideal and in particular it > doesn't > > work with CTest's --rerun-failed option. I'm wondering if there's > currently > > a better way of telling CMake/CTest about a setup step which must be run > > before some particular set of test cases and a tear down step after they > are > > all done. The tear down step needs to be performed regardless of whether > any > > of the real test cases pass or fail. > > > > The motivating case is to start up and shutdown a service which a (subset > > of) test cases need running. That service is expensive to set up and > hence > > it isn't feasible to start it up and shut it down for every test case > > individually. > > > > Any ideas? > > > > -- > > Craig Scott > > Melbourne, Australia > > http://crascit.com > > > > -- > > > > Powered by www.kitware.com > > > > Please keep messages on-topic and check the CMake FAQ at: > > http://www.cmake.org/Wiki/CMake_FAQ > > > > Kitware offers various services to support the CMake community. For more > > information on each offering, please visit: > > > > CMake Support: http://cmake.org/cmake/help/support.html > > CMake Consulting: http://cmake.org/cmake/help/consult
Re: [cmake-developers] Security in CMake
Hi Chuck,
> Is this intended to run on Linux?
Yes. And thanks for the pointing out to SELinux. I'll add it to my checklist.
---
The system is on very early stages now, so its parts are changing
rapidly and I'm able to consider different approaches to its
subsystems (including security).
But I probably confused all of you with the words 'package manager'.
It's the package manager only in the narrow sense. It's not trying to
be another apt, yum etc. Sorry that I didn't provide much details, but
the original topic is very precise and I think I'll return to it a bit
later when the functionality of the system will be in more stable
state.
On 22 August 2016 at 20:17, Chuck Atkins wrote:
> Hi Egor,
> Is this intended to run on Linux? If so, I think you're FAR better off
> leveraging an existing security framework like SELinux, since it's actually
> designed from the ground up to enforce these types of controls. You could
> define a label that you place on the executables run by the package manager
> and then enforce whatever restrictions and controls you feel are
> appropriate. This will let you do things like block network access got the
> specific CMake executable used by the package system. It allows the CMake
> scripts to leverage all the available language and command features but
> deny, ant a system level, actions you deem unsafe or harmful.
>
> --
> Chuck Atkins
> Staff R&D Engineer, Scientific Computing
> Kitware, Inc.
>
>
> On Sun, Aug 21, 2016 at 2:02 PM, Egor Pugin wrote:
>>
>> > What is the attack you want to stop? What are bad scripts and commands
>> > in this context?
>>
>> I wrote them in the first message. For example,
>> - any cmake commands that use COMMAND keyword (execute_process(COMMAND
>> ...), add_custom_{command|target}(...) etc. This will deny any user
>> scripts, programs (wget, curl, rm, ...).
>> - download commands (CMake's builtin curl) - they can fill the drives
>> with trash.
>>
>> > CMake runs lots of commands all the time. Most can be changed by a user,
>> > many are changed by the generator based on environment and whatnot. Any of
>> > these may be bad commands -- based on configuration.
>>
>> Yes, and it should deny only stuff above in small CMakeLists.txt part
>> that will be protected with some other commands or policies.
>>
>> > But if CMake gets a secure mode for your generator and if that is merged
>> > upstream, then I need to know about that when reading or writing
>> > CMakeLists.txt.
>>
>> For the moment I'm just asking about possibility of implementation of
>> these features. Any decision will go from CMake guys, not from me. So,
>> you shouldn't ask me about it. :)
>>
>> > Generated code is safe only as long as you very tightly control the
>> > environment CMake runs in.
>>
>> I don't care what is around, what is in user env. This is his
>> responsibility. I'm just worrying for my parts of CMake stuff.
>>
>> On 21 August 2016 at 20:43, Tobias Hunger wrote:
>> > Hi Egor,
>> >
>> > Am 21.08.2016 12:34 schrieb "Egor Pugin" :
>> >>
>> >> > What are the attack scenarios you want to defend against? What should
>> >> > not be possible in your system that currently is in CMake?
>> >>
>> >> At least downloading or executing bad scripts and commands.
>> >
>> > What is the attack you want to stop? What are bad scripts and commands
>> > in
>> > this context?
>> >
>> > CMake runs lots of commands all the time. Most can be changed by a user,
>> > many are changed by the generator based on environment and whatnot. Any
>> > of
>> > these may be bad commands -- based on configuration.
>> >
>> > Downloading can be done using internal commands or by running e.g. wget
>> > or
>> > curl, both of which are pretty widely available on developer machines.
>> >
>> >> > That forces me to keep more state in my head when reading
>> >> > CMakeLists.txt
>> >> > files.
>> >>
>> >> CMake files are generated in my system. That's what I mean when I said
>> >> 'based on CMake'.
>> >
>> > Sure. But if CMake gets a secure mode for your generator and if that is
>> > merged upstream, then I need to know about that when reading or writing
>> > CMakeLists.txt.
>> >
>> >> It's like compiler compiler like yacc, bison, lex, flex. They are
>> >> producing output not for human readers, but for computer parsers.
>> >> And that's why generated code is safe and insertions from users are
>> >> not.
>> >
>> > Generated code is safe only as long as you very tightly control the
>> > environment CMake runs in.
>> >
>> >> Also in the most cases there's no any insertions at all, so it's rare
>> >> case.
>> >
>> > I'm sure you know what you are doing:)
>> >
>> > Best regards,
>> > Tobias
>>
>>
>>
>> --
>> Egor Pugin
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http
Re: [cmake-developers] Security in CMake
Hi Egor,
Is this intended to run on Linux? If so, I think you're FAR better off
leveraging an existing security framework like SELinux, since it's actually
designed from the ground up to enforce these types of controls. You could
define a label that you place on the executables run by the package manager
and then enforce whatever restrictions and controls you feel are
appropriate. This will let you do things like block network access got the
specific CMake executable used by the package system. It allows the CMake
scripts to leverage all the available language and command features but
deny, ant a system level, actions you deem unsafe or harmful.
--
Chuck Atkins
Staff R&D Engineer, Scientific Computing
Kitware, Inc.
<%28518%29%20881-1183>
On Sun, Aug 21, 2016 at 2:02 PM, Egor Pugin wrote:
> > What is the attack you want to stop? What are bad scripts and commands
> in this context?
>
> I wrote them in the first message. For example,
> - any cmake commands that use COMMAND keyword (execute_process(COMMAND
> ...), add_custom_{command|target}(...) etc. This will deny any user
> scripts, programs (wget, curl, rm, ...).
> - download commands (CMake's builtin curl) - they can fill the drives
> with trash.
>
> > CMake runs lots of commands all the time. Most can be changed by a user,
> many are changed by the generator based on environment and whatnot. Any of
> these may be bad commands -- based on configuration.
>
> Yes, and it should deny only stuff above in small CMakeLists.txt part
> that will be protected with some other commands or policies.
>
> > But if CMake gets a secure mode for your generator and if that is merged
> upstream, then I need to know about that when reading or writing
> CMakeLists.txt.
>
> For the moment I'm just asking about possibility of implementation of
> these features. Any decision will go from CMake guys, not from me. So,
> you shouldn't ask me about it. :)
>
> > Generated code is safe only as long as you very tightly control the
> environment CMake runs in.
>
> I don't care what is around, what is in user env. This is his
> responsibility. I'm just worrying for my parts of CMake stuff.
>
> On 21 August 2016 at 20:43, Tobias Hunger wrote:
> > Hi Egor,
> >
> > Am 21.08.2016 12:34 schrieb "Egor Pugin" :
> >>
> >> > What are the attack scenarios you want to defend against? What should
> >> > not be possible in your system that currently is in CMake?
> >>
> >> At least downloading or executing bad scripts and commands.
> >
> > What is the attack you want to stop? What are bad scripts and commands in
> > this context?
> >
> > CMake runs lots of commands all the time. Most can be changed by a user,
> > many are changed by the generator based on environment and whatnot. Any
> of
> > these may be bad commands -- based on configuration.
> >
> > Downloading can be done using internal commands or by running e.g. wget
> or
> > curl, both of which are pretty widely available on developer machines.
> >
> >> > That forces me to keep more state in my head when reading
> CMakeLists.txt
> >> > files.
> >>
> >> CMake files are generated in my system. That's what I mean when I said
> >> 'based on CMake'.
> >
> > Sure. But if CMake gets a secure mode for your generator and if that is
> > merged upstream, then I need to know about that when reading or writing
> > CMakeLists.txt.
> >
> >> It's like compiler compiler like yacc, bison, lex, flex. They are
> >> producing output not for human readers, but for computer parsers.
> >> And that's why generated code is safe and insertions from users are not.
> >
> > Generated code is safe only as long as you very tightly control the
> > environment CMake runs in.
> >
> >> Also in the most cases there's no any insertions at all, so it's rare
> >> case.
> >
> > I'm sure you know what you are doing:)
> >
> > Best regards,
> > Tobias
>
>
>
> --
> Egor Pugin
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensou
> rce/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers
>
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware op
Re: [cmake-developers] Security in CMake
Hi Egor, On Sat, Aug 20, 2016 at 1:48 PM, Egor Pugin wrote: > Hi, > > I'm working on a package manager based on cmake. Please rethink that approach. Your package manager should support CMake. It should not be based on it. Your design will be safer and more flexible. Cheers, Daniel -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
