Re: Parallel builds and SUBDIRS
> "Harlan" == Harlan Stenn <[EMAIL PROTECTED]> writes: [ This is a reply to some pretty old email. As is my habit. ] Harlan> I'm working on a project where Somebody decided it would be a Harlan> feature to hack the automake templates to permit subdirs to be Harlan> built in parallel. Ok. Harlan> I'm wondering if there is benefit to allowing SUBDIRS to be Harlan> build in parallel by option (or default), and wondering if Harlan> it's possible, reasonable, and not too difficult to use Harlan> dependency lines in the "parent" Makefile.am to control Harlan> serialization, ie: I've actually considered this from time to time. It isn't an unreasonable idea. It would definitely have to be explicitly requested, though. We've long promised that SUBDIRS will be built in order. Lots of packages depend on this. Harlan> SUBDIRS = include lib bar baz foo Harlan> lib: include Harlan> bar baz foo: lib I suppose what we could do is have a new rule for each subdir: all-recursive-include clean-recursive-include ...: cd include && $(MAKE) ... Then from a specification of subdir interdependencies we could generate the exhaustive list. In Makefile.am: lib-subdir: include-subdir In Makefile.in: all-recursive-lib: all-recursive-include clean-recursive-lib: clean-recursive-include ... Perhaps there's some cleaner way. I don't know, I'm just writing this off the top of my head. Hmm, the clean dependencies would be tricky. Right now we run clean rules in the opposite order as build rules. The gcc top-level Makefile already does something like this. It isn't as robust as what we'd require for automake. One question I have is whether this is actually worth the extra hair. How much parallelism do we really lose with the current implementation? How much does it slow down the build? These questions implicitly assume that the primary benefit of this approach is improved performance. Are there other benefits? Anyway, if the benefit is substantial this would make a good feature-request PR. If there's no benefit, it may be better to drop it. Tom
RE: Parallel builds and SUBDIRS
One more try, this time with the attachment. - Dan -Original Message- From: Dan Kegel To: 'Harlan Stenn '; Dan Kegel Cc: ''[EMAIL PROTECTED] ' ' Sent: 5/17/2002 11:40 PM Subject: RE: Parallel builds and SUBDIRS Yep, I've implemented a build system with that property recently. I promised to post a writeup here, but haven't gotten a round tuit. Maybe it would be useful if I posted part of a project that uses that build system. The basic idea is it's a cross-platform project that needs to generate code for four different architectures: ppc405, ppc750, sh4, and i386. I wanted to do all builds in the same run of Make, but I also wanted to be able to cd anywhere and build everything at that level and below. The trick was to use absolute paths, to put all the interesting logic for a directory in a makefile fragment named make.defs.in, and use autoconf to put a copy of all the makefile fragments in an output subtree for each architecture. Here's most of that build system; it doesn't run in this form, but I could turn it into a working 'hello, world' example fairly easily if there was interest. - Dan -Original Message- From: Harlan Stenn I agree with you, and I agree with you. This project has several hundred subdirs and folks want to be able to go to any subdir and "make". Getting there from here will be Interesting, and will take a while. I'm looking for interim solutions. H -- > The Right Way to do this is to include all the > subdir makefile fragments into one big happy > make; then no special locking is needed. > (A la "recursive make considered harmful".) > > Boy, that's a big change, though. > - Dan > > -Original Message- > From: Harlan Stenn > > First, I'm not saying this is a good idea. > > I'm working on a project where Somebody decided it would be a feature to > hack the automake templates to permit subdirs to be built in parallel. > > They implemented an additional level of locking to prevent, for example, > a > library that is needed by multiple subdirs from being built > simultaneously. > > I'm wondering if there is benefit to allowing SUBDIRS to be build in > parallel by option (or default), and wondering if it's possible, > reasonable, > and not too difficult to use dependency lines in the "parent" > Makefile.am to > control serialization, ie: > > SUBDIRS = include lib bar baz foo > > lib: include > bar baz foo: lib > > so that in a parallel build situation, we'd build include first, then > lib, > then bar, baz and foo in parallel. > > Comments? > > H <> generic.tar.gz Description: Binary data
RE: Parallel builds and SUBDIRS
Yep, I've implemented a build system with that property recently. I promised to post a writeup here, but haven't gotten a round tuit. Maybe it would be useful if I posted part of a project that uses that build system. The basic idea is it's a cross-platform project that needs to generate code for four different architectures: ppc405, ppc750, sh4, and i386. I wanted to do all builds in the same run of Make, but I also wanted to be able to cd anywhere and build everything at that level and below. The trick was to use absolute paths, to put all the interesting logic for a directory in a makefile fragment named make.defs.in, and use autoconf to put a copy of all the makefile fragments in an output subtree for each architecture. Here's most of that build system; it doesn't run in this form, but I could turn it into a working 'hello, world' example fairly easily if there was interest. - Dan -Original Message- From: Harlan Stenn I agree with you, and I agree with you. This project has several hundred subdirs and folks want to be able to go to any subdir and "make". Getting there from here will be Interesting, and will take a while. I'm looking for interim solutions. H -- > The Right Way to do this is to include all the > subdir makefile fragments into one big happy > make; then no special locking is needed. > (A la "recursive make considered harmful".) > > Boy, that's a big change, though. > - Dan > > -Original Message- > From: Harlan Stenn > > First, I'm not saying this is a good idea. > > I'm working on a project where Somebody decided it would be a feature to > hack the automake templates to permit subdirs to be built in parallel. > > They implemented an additional level of locking to prevent, for example, > a > library that is needed by multiple subdirs from being built > simultaneously. > > I'm wondering if there is benefit to allowing SUBDIRS to be build in > parallel by option (or default), and wondering if it's possible, > reasonable, > and not too difficult to use dependency lines in the "parent" > Makefile.am to > control serialization, ie: > > SUBDIRS = include lib bar baz foo > > lib: include > bar baz foo: lib > > so that in a parallel build situation, we'd build include first, then > lib, > then bar, baz and foo in parallel. > > Comments? > > H generic.tar.gz Description: Binary data
Re: Parallel builds and SUBDIRS
I agree with you, and I agree with you. This project has several hundred subdirs and folks want to be able to go to any subdir and "make". Getting there from here will be Interesting, and will take a while. I'm looking for interim solutions. H -- > The Right Way to do this is to include all the > subdir makefile fragments into one big happy > make; then no special locking is needed. > (A la "recursive make considered harmful".) > > Boy, that's a big change, though. > - Dan > > -Original Message- > From: Harlan Stenn > To: [EMAIL PROTECTED] > Sent: 5/17/2002 5:41 PM > Subject: Parallel builds and SUBDIRS > > First, I'm not saying this is a good idea. > > I'm working on a project where Somebody decided it would be a feature to > hack the automake templates to permit subdirs to be built in parallel. > > They implemented an additional level of locking to prevent, for example, > a > library that is needed by multiple subdirs from being built > simultaneously. > > I'm wondering if there is benefit to allowing SUBDIRS to be build in > parallel by option (or default), and wondering if it's possible, > reasonable, > and not too difficult to use dependency lines in the "parent" > Makefile.am to > control serialization, ie: > > SUBDIRS = include lib bar baz foo > > lib: include > bar baz foo: lib > > so that in a parallel build situation, we'd build include first, then > lib, > then bar, baz and foo in parallel. > > Comments? > > H
RE: Parallel builds and SUBDIRS
The Right Way to do this is to include all the subdir makefile fragments into one big happy make; then no special locking is needed. (A la "recursive make considered harmful".) Boy, that's a big change, though. - Dan -Original Message- From: Harlan Stenn To: [EMAIL PROTECTED] Sent: 5/17/2002 5:41 PM Subject: Parallel builds and SUBDIRS First, I'm not saying this is a good idea. I'm working on a project where Somebody decided it would be a feature to hack the automake templates to permit subdirs to be built in parallel. They implemented an additional level of locking to prevent, for example, a library that is needed by multiple subdirs from being built simultaneously. I'm wondering if there is benefit to allowing SUBDIRS to be build in parallel by option (or default), and wondering if it's possible, reasonable, and not too difficult to use dependency lines in the "parent" Makefile.am to control serialization, ie: SUBDIRS = include lib bar baz foo lib: include bar baz foo: lib so that in a parallel build situation, we'd build include first, then lib, then bar, baz and foo in parallel. Comments? H
Parallel builds and SUBDIRS
First, I'm not saying this is a good idea. I'm working on a project where Somebody decided it would be a feature to hack the automake templates to permit subdirs to be built in parallel. They implemented an additional level of locking to prevent, for example, a library that is needed by multiple subdirs from being built simultaneously. I'm wondering if there is benefit to allowing SUBDIRS to be build in parallel by option (or default), and wondering if it's possible, reasonable, and not too difficult to use dependency lines in the "parent" Makefile.am to control serialization, ie: SUBDIRS = include lib bar baz foo lib: include bar baz foo: lib so that in a parallel build situation, we'd build include first, then lib, then bar, baz and foo in parallel. Comments? H
