Re: [CMake] How would I use parallel make on ExternalProjects?
On 10/8/2010 11:57 AM, kent williams wrote: I think what I'm doing isn't clear to you. I have a CMakeLists.txt that is essentially 'sourceless' -- it defines a bunch of CMake variables, and then builds a bunch of prerequisites and applications added with ExternalProject_add. It doesn't have any add_subdirectory,add_executable, add_library etc of its own. So at build time, the process tree is make # the make of the top level make # build external project 1 . . make # build external project n If I use make -j4, for example, it would build 4 external projects in parallel with make -j1. If any of those projects is on the critical path, and takes significantly longer to build than any other projects, the build process bottlenecks doing the sequential build of that one project. My method -- changing the build command for the external projects to use -j4 for example, will instead do a sequential series of parallel sub-makes, instead of a parallel series of sequential submakes. I've tried it both ways and this is how it works. We have fixed things recently to address this. The way it should be working: make # top level make # build external project 1 . . make # build external project n make -j4 at the top: The make jobserver should take care of the rest and make sure that 4 different things are always building at the same time. This could be .o files from more than one external project if that is how it works out. This is possible because $(MAKE) is used so when the external projects are run gmake will pass the jobserver information down to the next sub-make. The jobserver will keep track of making sure no more than 4 things are running at the same time. -Bill ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
I think what I'm doing isn't clear to you. I have a CMakeLists.txt that is essentially 'sourceless' -- it defines a bunch of CMake variables, and then builds a bunch of prerequisites and applications added with ExternalProject_add. It doesn't have any add_subdirectory,add_executable, add_library etc of its own. So at build time, the process tree is make # the make of the top level make # build external project 1 . . make # build external project n If I use make -j4, for example, it would build 4 external projects in parallel with make -j1. If any of those projects is on the critical path, and takes significantly longer to build than any other projects, the build process bottlenecks doing the sequential build of that one project. My method -- changing the build command for the external projects to use -j4 for example, will instead do a sequential series of parallel sub-makes, instead of a parallel series of sequential submakes. I've tried it both ways and this is how it works. For the gentleman who suggested distcc -- thanks but no thanks for these reasons: 1) it's something else to add to a very long list of prerequisites for our software and 2) it requires a mostly homogenous network of computers to farm compiles out to. We have, by design, a heterogenous network of OS X and different Linux versions, so that we're testing on as many platforms as possible. On Thu, Oct 7, 2010 at 11:29 AM, Bill Hoffman wrote: > ITK will not do a sequential build. The -j N gets passed down to all the > sub projects as well. It will run N build rules at the same time from VTK, > ITK and everything else. Just like it was all one big project... > ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
> There is one problem with that, which is what I'm trying to address: > load balancing. If your project builds ITK, VTK, and sundry other > prerequisite libraries, it will spawn 4 sequential builds. In > practice this means everything except ITK (which we build with > wrapping on) finishes, and then ITK chugs along doing its sequential > build. > > If I do what I'm talking about, I'd sequentially conduct parallel > builds of ITK VTK etc. In which case the big hairy libraries, like > ITK with wrapping, get built in parallel. Have you considered distcc? It's essentially a compiler load balancer and fairly mature by now. I don't think it works on Windows though, so if you need that it might not be what you want. ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
On 10/7/2010 12:13 PM, kent williams wrote: On Thu, Oct 7, 2010 at 11:05 AM, Bill Hoffman wrote: What you want is for make to treat the external projects just like any other library or executable in a build. so, if you run this at the top of the build: make -j4 It should run at most 4 concurrent things at once for the whole thing. By using $(MAKE) in the makefiles make will do that. There is one problem with that, which is what I'm trying to address: load balancing. If your project builds ITK, VTK, and sundry other prerequisite libraries, it will spawn 4 sequential builds. In practice this means everything except ITK (which we build with wrapping on) finishes, and then ITK chugs along doing its sequential build. If I do what I'm talking about, I'd sequentially conduct parallel builds of ITK VTK etc. In which case the big hairy libraries, like ITK with wrapping, get built in parallel. ITK will not do a sequential build. The -j N gets passed down to all the sub projects as well. It will run N build rules at the same time from VTK, ITK and everything else.Just like it was all one big project... -Bill ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
On Thu, Oct 7, 2010 at 11:05 AM, Bill Hoffman wrote:
>
> What you want is for make to treat the external projects just like any other
> library or executable in a build.
>
> so, if you run this at the top of the build:
>
> make -j4
>
> It should run at most 4 concurrent things at once for the whole thing. By
> using $(MAKE) in the makefiles make will do that.
>
There is one problem with that, which is what I'm trying to address:
load balancing. If your project builds ITK, VTK, and sundry other
prerequisite libraries, it will spawn 4 sequential builds. In
practice this means everything except ITK (which we build with
wrapping on) finishes, and then ITK chugs along doing its sequential
build.
If I do what I'm talking about, I'd sequentially conduct parallel
builds of ITK VTK etc. In which case the big hairy libraries, like
ITK with wrapping, get built in parallel.
> That said, CMake will do this by default for make based builds and external
> projects now.
>
> From ExternalProject.cmake:
>
> get_target_property(cmake_generator ${name} _EP_CMAKE_GENERATOR)
> if("${CMAKE_GENERATOR}" MATCHES "Make" AND
> ("${cmake_generator}" STREQUAL "${CMAKE_GENERATOR}" OR
> NOT cmake_generator))
> # The project uses the same Makefile generator. Use recursive make.
> set(cmd "$(MAKE)")
> if(step STREQUAL "INSTALL")
> set(args install)
> endif()
>
>
> -Bill
>
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
On 10/7/2010 11:52 AM, kent williams wrote:
1. Is that $(MAKE) or is it ${MAKE} ? One thing missing from the
CMake documentation -- unless I'm mistaken there's not much
explanation of CMake syntax in the documentation.
No, this is make syntax not CMake syntax.
2. I think it's probably not what one intends to have 'make -j4' (for
example) used every time make is invoked. If you configure a program
that includes several ExternalProjects, then it would spawn 4
concurrent builds of those ExternalProjects, and then each of those
builds would spawn 4 make steps at once, for 16 concurrent processes.
What you want is for make to treat the external projects just like any
other library or executable in a build.
so, if you run this at the top of the build:
make -j4
It should run at most 4 concurrent things at once for the whole thing.
By using $(MAKE) in the makefiles make will do that.
That said, CMake will do this by default for make based builds and
external projects now.
From ExternalProject.cmake:
get_target_property(cmake_generator ${name} _EP_CMAKE_GENERATOR)
if("${CMAKE_GENERATOR}" MATCHES "Make" AND
("${cmake_generator}" STREQUAL "${CMAKE_GENERATOR}" OR
NOT cmake_generator))
# The project uses the same Makefile generator. Use recursive
make.
set(cmd "$(MAKE)")
if(step STREQUAL "INSTALL")
set(args install)
endif()
-Bill
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
On the other hand, with Visual Studio, it's all a big hairy mess.
because the setting for "how many parallel projects to build simultaneously"
is a single top level global setting. So when we spawn sub-VS instances in
VS 2008, for example, each one uses "N" for its parallel setting.
In that case, I typically recommend setting that "N" to be half to a quarter
of your processors available, assuming that there will be between 2 and 4
(on average) projects building simultaneously.
On Thu, Oct 7, 2010 at 11:55 AM, David Cole wrote:
> If you use "$(MAKE)" in a BUILD_COMMAND, then the literal "$(MAKE)" appears
> in the generated makefiles.
>
> That tells the top level make to spawn sub-makes with the job controller
> from the top level make.
>
> Then you do not need to specify any -j flags anywhere except at the top
> level.
>
> And then, the top level job controller makes sure there are only N
> concurrent things happening regardless of the level of sub-makes...
>
> Does that "make" sense?
>
>
> :-)
> David
>
>
> On Thu, Oct 7, 2010 at 11:52 AM, kent williams
> wrote:
>
>> 1. Is that $(MAKE) or is it ${MAKE} ? One thing missing from the
>> CMake documentation -- unless I'm mistaken there's not much
>> explanation of CMake syntax in the documentation.
>>
>> 2. I think it's probably not what one intends to have 'make -j4' (for
>> example) used every time make is invoked. If you configure a program
>> that includes several ExternalProjects, then it would spawn 4
>> concurrent builds of those ExternalProjects, and then each of those
>> builds would spawn 4 make steps at once, for 16 concurrent processes.
>>
>>
>> On Thu, Oct 7, 2010 at 10:36 AM, Bill Hoffman
>> wrote:
>> > On 10/7/2010 11:25 AM, kent williams wrote:
>> >>
>> >> On Wed, Oct 6, 2010 at 5:01 PM, Clifford Yapp
>> wrote:
>> >>>
>> >>> I use $(MAKE) in my BUILD_COMMAND and that seems to do OK, although I
>> >>> don't know if it works universally.
>> >>>
>> >>
>> >> That's an environment variable, as near as I can tell and isn't
>> >> mentioned in the current CMake documentation. So it's probably not the
>> >> best thing to do.
>> >>
>> >> upon reflection, this would be a little safer:
>> >>
>> >> if("${CMAKE_GENERATOR}" STREQUAL "Unix Makefiles")
>> >> set(BUILD_COMMAND_STRING "${CMAKE_MAKE_PROGRAM} -j4")
>> >> else()
>> >> set(BUILD_COMMAND_STRING "$(CMAKE_MAKE_PROGRAM)")
>> >> endif()
>> >
>> > By using $(MAKE), the toplevel -j N option should be passed down. The
>> 2.8.3
>> > RC that is out now has some fixes in this area.
>> >
>> >
>> > -Bill
>> ___
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake
>>
>
>
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
If you use "$(MAKE)" in a BUILD_COMMAND, then the literal "$(MAKE)" appears
in the generated makefiles.
That tells the top level make to spawn sub-makes with the job controller
from the top level make.
Then you do not need to specify any -j flags anywhere except at the top
level.
And then, the top level job controller makes sure there are only N
concurrent things happening regardless of the level of sub-makes...
Does that "make" sense?
:-)
David
On Thu, Oct 7, 2010 at 11:52 AM, kent williams wrote:
> 1. Is that $(MAKE) or is it ${MAKE} ? One thing missing from the
> CMake documentation -- unless I'm mistaken there's not much
> explanation of CMake syntax in the documentation.
>
> 2. I think it's probably not what one intends to have 'make -j4' (for
> example) used every time make is invoked. If you configure a program
> that includes several ExternalProjects, then it would spawn 4
> concurrent builds of those ExternalProjects, and then each of those
> builds would spawn 4 make steps at once, for 16 concurrent processes.
>
>
> On Thu, Oct 7, 2010 at 10:36 AM, Bill Hoffman
> wrote:
> > On 10/7/2010 11:25 AM, kent williams wrote:
> >>
> >> On Wed, Oct 6, 2010 at 5:01 PM, Clifford Yapp
> wrote:
> >>>
> >>> I use $(MAKE) in my BUILD_COMMAND and that seems to do OK, although I
> >>> don't know if it works universally.
> >>>
> >>
> >> That's an environment variable, as near as I can tell and isn't
> >> mentioned in the current CMake documentation. So it's probably not the
> >> best thing to do.
> >>
> >> upon reflection, this would be a little safer:
> >>
> >> if("${CMAKE_GENERATOR}" STREQUAL "Unix Makefiles")
> >> set(BUILD_COMMAND_STRING "${CMAKE_MAKE_PROGRAM} -j4")
> >> else()
> >> set(BUILD_COMMAND_STRING "$(CMAKE_MAKE_PROGRAM)")
> >> endif()
> >
> > By using $(MAKE), the toplevel -j N option should be passed down. The
> 2.8.3
> > RC that is out now has some fixes in this area.
> >
> >
> > -Bill
> ___
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
1. Is that $(MAKE) or is it ${MAKE} ? One thing missing from the
CMake documentation -- unless I'm mistaken there's not much
explanation of CMake syntax in the documentation.
2. I think it's probably not what one intends to have 'make -j4' (for
example) used every time make is invoked. If you configure a program
that includes several ExternalProjects, then it would spawn 4
concurrent builds of those ExternalProjects, and then each of those
builds would spawn 4 make steps at once, for 16 concurrent processes.
On Thu, Oct 7, 2010 at 10:36 AM, Bill Hoffman wrote:
> On 10/7/2010 11:25 AM, kent williams wrote:
>>
>> On Wed, Oct 6, 2010 at 5:01 PM, Clifford Yapp wrote:
>>>
>>> I use $(MAKE) in my BUILD_COMMAND and that seems to do OK, although I
>>> don't know if it works universally.
>>>
>>
>> That's an environment variable, as near as I can tell and isn't
>> mentioned in the current CMake documentation. So it's probably not the
>> best thing to do.
>>
>> upon reflection, this would be a little safer:
>>
>> if("${CMAKE_GENERATOR}" STREQUAL "Unix Makefiles")
>> set(BUILD_COMMAND_STRING "${CMAKE_MAKE_PROGRAM} -j4")
>> else()
>> set(BUILD_COMMAND_STRING "$(CMAKE_MAKE_PROGRAM)")
>> endif()
>
> By using $(MAKE), the toplevel -j N option should be passed down. The 2.8.3
> RC that is out now has some fixes in this area.
>
>
> -Bill
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
On 10/7/2010 11:25 AM, kent williams wrote:
On Wed, Oct 6, 2010 at 5:01 PM, Clifford Yapp wrote:
I use $(MAKE) in my BUILD_COMMAND and that seems to do OK, although I
don't know if it works universally.
That's an environment variable, as near as I can tell and isn't
mentioned in the current CMake documentation. So it's probably not the
best thing to do.
upon reflection, this would be a little safer:
if("${CMAKE_GENERATOR}" STREQUAL "Unix Makefiles")
set(BUILD_COMMAND_STRING "${CMAKE_MAKE_PROGRAM} -j4")
else()
set(BUILD_COMMAND_STRING "$(CMAKE_MAKE_PROGRAM)")
endif()
By using $(MAKE), the toplevel -j N option should be passed down. The
2.8.3 RC that is out now has some fixes in this area.
-Bill
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
On Wed, Oct 6, 2010 at 5:01 PM, Clifford Yapp wrote:
> I use $(MAKE) in my BUILD_COMMAND and that seems to do OK, although I
> don't know if it works universally.
>
That's an environment variable, as near as I can tell and isn't
mentioned in the current CMake documentation. So it's probably not the
best thing to do.
upon reflection, this would be a little safer:
if("${CMAKE_GENERATOR}" STREQUAL "Unix Makefiles")
set(BUILD_COMMAND_STRING "${CMAKE_MAKE_PROGRAM} -j4")
else()
set(BUILD_COMMAND_STRING "$(CMAKE_MAKE_PROGRAM)")
endif()
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] How would I use parallel make on ExternalProjects?
I use $(MAKE) in my BUILD_COMMAND and that seems to do OK, although I don't know if it works universally. ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
[CMake] How would I use parallel make on ExternalProjects?
I'd like to use parallel make to build libraries like ITK and VTK
which are rather time-consuming.
This simple-minded (and non portable) method works: In the
ExternalProject_Add macro, add
BUILD_COMMAND ${CMAKE_MAKE_PROGRAM} -j4
But that only works if CMAKE_MAKE_PROGRAM understands the '-j4' flag.
I can test the generator, i.e.
if("${CMAKE_GENERATOR}" STREQUAL "Unix Makefiles")
set(BUILD_COMMAND_STRING "${CMAKE_PROGRAM} -j4")
endif
and later use in the ExternalProjectAdd
BUILD_COMMAND ${BUILD_COMMAND_STRING}
But if the generator is not "Unix Makefiles" this will expand to the
empty string, with the result of suppressing builds from ever
happening.
The only solution I can imagine is to have 2 ExternalProject_Add
macros inside a conditional
if("${CMAKE_GENERATOR}" STREQUAL "Unix Makefiles")
ExternalProjectAdd(
BUILD_COMMAND ${CMAKE_MAKE_PROGRAM} -j4
)
else()
ExternalProjectAdd(
# let the build command be the default for this generator
)
endif()
Anyone have a better idea?
___
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
