Re: [cmake-developers] Setting up environment using ExternalProject_Add
On 11-Aug-15 07:49, James Johnston wrote: I have found it annoyingly difficult to set up an environment for a build when using ExternalProject_Add. I can confirm that there is an additional mess with the environment sometimes and not just about setting it (like vcvarsall.bat for Visual Studio) but with *un*setting them too. In short environment variables for the ExternalProject_Add are not the same as a user's environment. I've found myself some problems with Xcode which add tons of new variables but there are even cases for Makefiles (e.g. stackoverflow.com/questions/25160405/). I will not say it's an annoying thing for me but if there will be some mechanism to control it (better globally or even something from the box) ExternalProject_Add code can be simplified. I'm personally using next patterns: CONFIGURE_COMMAND @MSVC_VCVARSALL@@MSVC_ARCH@ # Set Visual Studio environment COMMAND ... # real configure command (like bootstrap.bat or configure.bat) BUILD_COMMAND . /path/to/clear-all.sh make # run make in a clean environment INSTALL_COMMAND . /path/to/clear-all.sh make install # same for make install include(/path/to/clear-all.cmake) execute_process(COMMAND make @make_params@ ...) # run make in a clean environment Cheers, Ruslo -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Setting up environment using ExternalProject_Add
On 08/12/2015 08:06 PM, James Johnston wrote:
http://www.cmake.org/Bug/view.php?id=15653
I'll explore that option some more, after the feature you linked to is
addressed.
Currently no one is looking at addressing it. If you want to work on
it, take a look at the PERMISSIONS option of the install() command:
http://www.cmake.org/cmake/help/v3.3/command/install.html
There has also been discussion before about permissions options that
respect umask:
http://www.cmake.org/Bug/view.php?id=10126#c19488
http://www.cmake.org/Bug/view.php?id=15163
1. The new step_ENVIRONMENT setting points to a environment-set script
that does not try to do any launching (e.g. you could pass VCVarsAll.bat
directly as the step_ENVIRONMENT). ExternalProject determines which shell
to generate for, and generates the environment-set-and-launch script
itself from scratch (like how I originally proposed with the cmake -E
enhancement).
In other words, ExternalProject(snip CONFIGURE_ENVIRONMENT
$ENV{VS100COMNTOOLS}/../../VC/vcvarsall.bat x86 CMAKE_GENERATOR Ninja
CMAKE_CACHE_ARGS snip) and away the user goes..
I think that is the simplest approach. It should be easier to use
and also allows us to change how it works underneath in the future if
needed because CMake retains control of the final script generation.
Thanks,
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Setting up environment using ExternalProject_Add
-Original Message- From: cmake-developers [mailto:cmake-developers-boun...@cmake.org] On Behalf Of David Cole via cmake-developers Sent: Wednesday, August 12, 2015 15:53 To: Brad King Cc: cmake-developers@cmake.org Subject: Re: [cmake-developers] Setting up environment using ExternalProject_Add One thing to consider before embarking on modifying CMake for required environment purposes is that it might be just as simple to force (or at least strongly encourage) your project developers to use the proper environment for running the whole build. You could, for example, have an initial external project which all others depend on which verifies the proper environment is set when it is called. If not, it fails with an error, and prevents downstream bits from building. snip I think your situation is simpler than mine - i.e. if I'm reading this right, all your ExternalProjects are built for the same Visual C++ version and CPU architecture - such that the user can/should run VCVarsAll before invoking the CMake superbuild? What I'm trying to do: A CMake superbuild that expects NO environment set up, and NO languages configured. It: * Builds Visual C++ 32-bit * Builds Visual C++ 64-bit * Builds some other non-Visual-C++ compilers/architectures All within the same superbuild. This is due to the fact that CMake doesn't support multiple architectures directly (and probably won't for a long time, if ever). So, each ExternalProject_Add needs to have its own unique environment set up. I don't think it makes sense to try to run VCVarsAll.bat before invoking the superbuild - in fact, doing so is detrimental (i.e. increases risk of building for the wrong CPU architecture if a particular VCVarsAll.bat invocation silently malfunctions). Best regards, James Johnston -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Setting up environment using ExternalProject_Add
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
On Behalf Of Brad King
Sent: Wednesday, August 12, 2015 14:10
To: cmake-developers@cmake.org
Subject: Re: [cmake-developers] Setting up environment using
ExternalProject_Add
On 08/11/2015 10:41 AM, James Johnston wrote:
1. CMake creates a temporary shell script / batch file (the shell
chosen based on the platform CMake compiled for - cmd.exe on Windows,
sh on POSIX).
I don't think CMake needs to be the one to generate this. The complexity
of
the file-based or command-line-based interfaces you discussed shows that
it
would be tricky to have CMake generate it.
snip
OK...
enhancing e.g. add_custom_command as I proposed might make it easier,
as this command could be aware of compiled EXE targets.
With file(GENERATE) the project code can easily create scripts that
reference
executable targets and such. One would just need to tackle this feature
request first to get file permissions right:
http://www.cmake.org/Bug/view.php?id=15653
Indeed; that looks like a useful feature to have to begin with, and a
requirement if ExternalProject is to generate its own script...
Therefore I do not think add_custom_command needs any hook for this.
One would simply give the script as the COMMAND.
Right, obviously not needed if no cmake -E option is added/enhanced and if
we want ExternalProject_Add (and anyone else needing custom targets) to make
their own script...
we'd want to add features to ExternalProject that use the new CMake
functionality to finally solve the original problem.
Yes, this part certainly needs an update to ExternalProject. Currently
when
one specifies an individual step_COMMAND like CONFIGURE_COMMAND,
then one could simply give it the environment-set-and-launch script
directly.
What ExternalProject needs is an interface to specify the environment-
setting script for use around otherwise default-generated commands for
each step. For example, a step_ENVIRONMENT setting (which is perhaps
not allowed in combination with step_COMMAND).
This approach is similar to your original alternative (3).
I'll explore that option some more, after the feature you linked to is
addressed. one could simply give it the environment-set-and-launch script
directly ... This script is obviously platform-specific, yet still needs
knowledge of the command that ExternalProject will generate to launch the
sub-CMake - which the user of ExternalProject obviously does not know. I am
not sure that introducing some shell-specific knowledge to ExternalProject
can be avoided in this case, so hopefully it will not be a problem to do
this.
There are two ways I immediately think of - trying to clarify what I think
you are suggesting:
1. The new step_ENVIRONMENT setting points to a environment-set script
that does not try to do any launching (e.g. you could pass VCVarsAll.bat
directly as the step_ENVIRONMENT). ExternalProject determines which shell
to generate for, and generates the environment-set-and-launch script
itself from scratch (like how I originally proposed with the cmake -E
enhancement).
In other words, ExternalProject(snip CONFIGURE_ENVIRONMENT
$ENV{VS100COMNTOOLS}/../../VC/vcvarsall.bat x86 CMAKE_GENERATOR Ninja
CMAKE_CACHE_ARGS snip) and away the user goes..
2. The new step_ENVIRONMENT setting points to an
environment-set-and-launch script that is expected to be configured with
configure_file by ExternalProject. So e.g. the user provides a batch file
that calls VCVarsAll.bat, and then there is a magical
${EXTERNAL_PROJECT_COMMAND} or whatever that the user is expected to provide
at the end of the template batch file that they give to ExternalProject. I
suspect this will simplify ExternalProject a little bit at the cost of more
boiler-plate code that the user has to provide (e.g. can't directly call
VCVarsAll.bat in the common case, must manage their own scripts). And gives
the user opportunity to really mess up ExternalProject operation (e.g.
forgetting to include ${EXTERNAL_PROJECT_COMMAND}, or forgetting to check
ERROR_LEVEL in a shell-specific way after calling EXTERNAL_PROJECT_COMMAND,
etc.). I think this would not be as user friendly, but perhaps slightly
easier for ExternalProject to handle. But at the end, the configure_file
substitution for ${EXTERNAL_PROJECT_COMMAND} is still potentially
shell-specific
In other words, ExternalProject(snip CONFIGURE_ENVIRONMENT
ProjectEnvironment.bat.in CMAKE_GENERATOR Ninja CMAKE_CACHE_ARGS snip)
And ProjectEnvironment.bat.in:
@echo off
call %VS100COMNTOOLS%/../../VC/vcvarsall.bat
rem Users better not forget this next line...
rem ExternalProject would configure_file this.
${EXTERNAL_PROJECT_COMMAND}
rem If a user adds more commands to the end of the batch file, they'd need
this:
if ERRORLEVEL 1 exit /b %ERRORLEVEL%
Best regards,
James Johnston
--
Powered by www.kitware.com
Please keep messages on-topic
Re: [cmake-developers] Setting up environment using ExternalProject_Add
-Original Message-
From: Brad King [mailto:brad.k...@kitware.com]
Sent: Tuesday, August 11, 2015 13:35
To: James Johnston
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] Setting up environment using
ExternalProject_Add
On 08/11/2015 12:49 AM, James Johnston wrote:
CONFIGURE_ENVIRONMENT_COMMAND path to VCVarsAll.bat
[snip]
add_custom_command(snip ENVIRONMENT_COMMAND path to
vcvarsall.bat
COMMAND ${CMAKE_PROGRAM} -GNinja snip)
A problem with both of these approaches is that on UNIX and in Make the
environment does not persist after the command exits.
Exactly right, I'm aware... and also it seems there is no
good/easy/non-hackish way to capture the environment from a child process
after it terminates, either - that I'm aware of. (It would be nice if we
could run a process, then capture its environment after it exits, but I
don't think it can be done.)
Whatever wrapper
tool is used must set the environment and also launch the real command to
be run under that environment. This is what the env tool (and recently
cmake -E env) does.
Right, but the problem with cmake -E env is it assumes you know what
variables you want to set - we don't. Only a shell script / batch file
knows in the case of VCVarsAll.bat...
I think I mentioned this solution in the original e-mail but in more detail,
what I'm proposing with the proposed cmake -E run_commands or whatever we
want to call it would be the following. In fact upon further thought, it
might be better to see about enhancing the cmake -E env command but I am
not sure if this is possible due to the syntax of cmake -E env. Anyhow
the basic idea is this:
1. CMake creates a temporary shell script / batch file (the shell chosen
based on the platform CMake compiled for - cmd.exe on Windows, sh on POSIX).
2. CMake calls one or more environment setup commands. These are pulled
in using call / source. Users would be required to provide scripts for
environment setup in the format of the native shell (e.g. you can't call a
bash script from a Windows .bat file). For example:
@echo off
rem User script provided to CMake, including some arguments to the
script
call dir\VCVarsAll.bat x86
rem An additional environment variable explicitly specified by the user
set MyVar=Value
cmake -GNinja .
Or:
#!/bin/sh
source setMyEnv.sh
cmake -GNinja .
Error handling has been cut for brevity, but we would want the script to
check for ERRORLEVEL / $? after each line above. As you can see, the
problem of environment not being retained has been solved by using call
and source.
3. Then CMake runs the above script as normal and checks the exit code as
normal.
It's basically a close cousin of what cmake -E env does in that we: (1) do
some things to set up the environment, (2) run the specified EXE. Ideally
cmake -E env could be enhanced with the new functionality, but since we now
want to pass an arbitrary list of commands, each with potentially unlimited
parameters of any format, I'm not sure how that should look in practice. It
might be easier to make a new command cmake -E env2 or whatever (need to
find a better name):
* It could be passed a file. The file contents would have a series of
lines, either SET / UNSET name=[value], SOURCE VCVarsAll.bat x86, or
COMMAND cmake.exe snip. Newlines delimit commands.
* It could be passed all the commands directly on the command line. A
recognized -- switch delimits commands. E.g. cmake -E env2 --unset NAME
--set NAME=VALUE --source VCVarsAll.bat x86 --command cmake.exe snip
--source moreSetup.bat --command somethingelse.exe
Downsides with first method: file adds complexity. Downsides with second
method: subject to command line length limits, though probably not an issue
in practice. Also, what if user wants to pass --unset or --set
parameter to their command as an argument? Without a means of escaping,
CMake will think the command is ended and the user is trying to do something
new.
That's the first and most important step. The above functionality would
still be inconvenient to use (e.g. cmake -E will not be aware of any
compiled EXE target output paths). So, enhancing e.g. add_custom_command as
I proposed might make it easier, as this command could be aware of compiled
EXE targets. Finally, we'd want to add features to ExternalProject that use
the new CMake functionality to finally solve the original problem.
The alternative ideas in my e-mail are just a question of who generates
the above temporary script that sources/calls a user script. E.g. in
option 3, we leave C++ portion of CMake alone, and ExternalProject would
then have to auto-generate the above temporary script instead.
Best regards,
James Johnston
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community
Re: [cmake-developers] Setting up environment using ExternalProject_Add
On 08/11/2015 12:49 AM, James Johnston wrote:
CONFIGURE_ENVIRONMENT_COMMAND path to VCVarsAll.bat
[snip]
add_custom_command(snip ENVIRONMENT_COMMAND path to vcvarsall.bat
COMMAND ${CMAKE_PROGRAM} -GNinja snip)
A problem with both of these approaches is that on UNIX and in Make
the environment does not persist after the command exits. Whatever
wrapper tool is used must set the environment and also launch the
real command to be run under that environment. This is what the
env tool (and recently cmake -E env) does. Of course on Windows
we need to run the .bat file to set the environment because we do
not know what variables it will set.
So, in order to do something like this we first need to identify a
proper abstraction to capture both cases.
Thanks,
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] Setting up environment using ExternalProject_Add
Hi,
(Pardon the length; skip to end to see what I am proposing to change. Below
is justification why.)
I have found it annoyingly difficult to set up an environment for a build
when using ExternalProject_Add. This seems to be needed when creating a
superbuild that uses several different compilers. The common workaround
seems to be to use a batch file that kicks off the build, as suggested by
multiple posts I have seen on these mailing lists, StackOverflow, etc. An
example of a nice neat usage that doesn't need an environment set up.
# For this discussion, suppose this is called from a
# superbuild that has no languages set up, and no build
# environment.
ExternalProject_Add(
snip
# Conveniently, VS2010 generator doesn't need VCVarsAll.bat
CMAKE_GENERATOR Visual Studio 10 2010
CMAKE_CACHE_ARGS long list of args)
But let's say I want to use Ninja to build both VS2010 32-bit/64-bit. Now
things get really messy, especially if I want my logic in a reusable piece
of code (effectively acting as a wrapper around ExternalProject_Add):
# ${setEnv} must be path to a batch file that does this.
# 1. Call VCVarsAll for desired VC version.
# 2. Run command passed via arguments.
snip code to configure_file and write ${setEnv}
# ${someScriptFile} has to be made via file(GENERATE).
# Essentially we have to reimplement the CMAKE_CACHE_ARGS option.
# That is made difficult because _ep_command_line_to_initial_cache
# is a private implementation detail of ExternalProject.cmake.
snip code to file(GENERATE) ${someScriptFile}
ExternalProject_Add(
snip
CONFIGURE_COMMAND ${setEnv} ${CMAKE_COMMAND}
-GNinja -C${someScriptFile})
Basically I find myself copy/pasting _ep_command_line_to_initial_cache into
my own project, amongst other annoyances. :(
What would be nice is if I could directly pass VCVarsAll.bat to
ExternalProject_Add and still be able to use CMAKE_GENERATOR and
CMAKE_CACHE_ARGS. It would also be nice if this mechanism did not interfere
with option CONFIGURE_COMMAND in case the external project is not
CMake-based.
Something like this might be perfect:
ExternalProject_Add(
snip
CONFIGURE_ENVIRONMENT_COMMAND path to VCVarsAll.bat
CMAKE_GENERATOR Ninja
CMAKE_CACHE_ARGS long list of args)
# Offer similar capability for other steps like building.
This is made somewhat difficult by the fact that CMake currently provides no
mechanism for a custom command to do this that I know of, and thus nothing
for ExternalProject_Add to easily build off of. Therefore, I propose
several possible options, and I'd like to know which one(s) I should think
about pursuing:
1. add_custom_command (and friends) could be extended with a new keyword
indicating that command is environment preparation for the next command.
For example:
add_custom_command(snip ENVIRONMENT_COMMAND path to vcvarsall.bat
COMMAND ${CMAKE_PROGRAM} -GNinja snip)
In this case, add_custom_command will run the ENVIRONMENT_COMMAND and
COMMAND in the same shell. The ENVIRONMENT_COMMAND would have to be
source'd / call'd as needed, depending on the shell.
Implementation of #1:
a. A new cmake -E run_commands command list file option could be
added that runs each command on each line of the input file in a shared
environment (source'ing and call'ing as needed depending on shell). E.g. by
translating the list of input commands to a temporary batch/shell script and
running it. Behind the scenes, add_custom_command and friends would
assemble things into a cmake -E invocation that can be passed to the
generators - handling EXE targets as needed.
b. OR: Each CMake generator handles this directly without any special
cmake -E option. (This sounds like more work?)
2. Instead of altering add_custom_command, just add the proposed cmake -E
run_commands option as described above. ExternalProject_Add could then be
modified to use this new cmake -E option directly. (Need to think about
if the command is an EXE target...)
3. Instead of altering any C++ CMake code at all, modify
ExternalProject_Add to implement my original workaround outlined at the top
of this e-mail of generating a wrapper script. Downside: now
ExternalProject.cmake has to learn platform-specific knowledge of the shell.
Also need to think about if target is EXE target...
4. Do nothing; this is too complicated and so are not desired changes to
make to CMake. Or: something else?
The one possible wrench in the works would be if you used a Cygwin version
of CMake (i.e. CMake expects /bin/sh shell) and wanted to prepare to use
Visual C++ and run VCVarsAll.bat (i.e. but you need cmd.exe shell). That
wouldn't work without a special option allowing you to explicitly set the
shell. That's a rare enough case it's probably not worth worrying about in
this initial version, I think.
My personal preference would be option 1, and
