Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-04-24 Thread Justin T. Gibbs
Why wouldn’t we fix these with explicit subdir dependencies (e.g: subdira: 
subdirb)?  I’m pretty sure I had this working with a hacked up bsd.subdir.mk at 
a previous job, but my memory is hazy.

—
Justin
  
On Apr 21, 2014, at 7:35 AM, Warner Losh i...@bsdimp.com wrote:

 (sorry for the top post)
 
 This looks good to my eye. I’d be tempted to toss in a comment about the
 __wait=.WAIT construct is due to the primitive nature of bmake’s parser
 so it runs afoul of the .for/.if construction rules and this is needed to 
 expand
 the for variable. It tripped me up when I looked at it, until I recalled a 
 comment
 from similar code in NetBSD.
 
 Warner
 
 On Apr 21, 2014, at 7:15 AM, Ian Lepore i...@freebsd.org wrote:
 
 On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote:
 On 27 Mar 2014, at 19:12, Jilles Tjoelker jil...@stack.nl wrote:
 On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote:
 On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
 Author: dim
 Date: Wed Mar 26 22:30:38 2014
 New Revision: 263778
 URL: http://svnweb.freebsd.org/changeset/base/263778
 
 Log:
 Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
 all the SUBDIR entries in parallel, instead of serially.  Apply this
 option to a selected number of Makefiles, which can greatly speed up the
 build on multi-core machines, when using make -j.
 
 This can be extended to more Makefiles later on, whenever they are
 verified to work correctly with parallel building.
 
 Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are
 there any known bad dependencies this introduces?
 
 I'm paranoid about build systems ;) It is easy to add dependencies
 across directories and as long as directories are built in sequence,
 nothing goes wrong.
 
 In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as well,
 but this caused mysterious failures with some kernels such as mips
 ADM5120.
 
 There are a bunch of other parts that don't really like parallel builds
 at the moment.  For example, gnu/usr.bin/binutils needs its libraries
 (libbfd.a, etc) built first, before it can link the programs.  Similar
 for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being able
 to build the rest of gcc.
 
 Most of these cases can hopefully be solved by adding .WAIT targets at
 strategic points in the SUBDIR lists, but this also needs a bit of extra
 logic in bsd.subdir.mk.
 
 -Dimitry
 
 
 It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL for
 $work, so I came up with the attached, does this look okay to commit?
 
 -- Ian
 
 diff -r 67802e319fc6 share/mk/bsd.subdir.mk
 --- a/share/mk/bsd.subdir.mk Sun Apr 20 21:01:07 2014 -0600
 +++ b/share/mk/bsd.subdir.mk Mon Apr 21 06:59:37 2014 -0600
 @@ -4,10 +4,10 @@
 # The include file bsd.subdir.mk contains the default targets
 # for building subdirectories.
 #
 -# For all of the directories listed in the variable SUBDIRS, the
 +# For all of the directories listed in the variable SUBDIR, the
 # specified directory will be visited and the target made. There is
 # also a default target which allows the command make subdir where
 -# subdir is any directory listed in the variable SUBDIRS.
 +# subdir is any directory listed in the variable SUBDIR.
 #
 #
 # +++ variables +++
 @@ -42,7 +42,7 @@ distribute:
 
 _SUBDIR: .USE
 .if defined(SUBDIR)  !empty(SUBDIR)  !defined(NO_SUBDIR)
 -@${_+_}for entry in ${SUBDIR}; do \
 +@${_+_}for entry in ${SUBDIR:N.WAIT}; do \
  if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \
  ${ECHODIR} === ${DIRPRFX}$${entry}.${MACHINE_ARCH} 
 (${.TARGET:realinstall=install}); \
  edir=$${entry}.${MACHINE_ARCH}; \
 @@ -57,7 +57,7 @@ distribute:
  done
 .endif
 
 -${SUBDIR}: .PHONY
 +${SUBDIR:N.WAIT}: .PHONY
  ${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \
  cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \
  else \
 @@ -65,13 +65,18 @@ distribute:
  fi; \
  ${MAKE} all
 
 +__wait=.WAIT
 .for __target in all all-man checkdpadd clean cleandepend cleandir \
depend distribute lint maninstall manlint \
obj objlink realinstall regress tags \
${SUBDIR_TARGETS}
 .ifdef SUBDIR_PARALLEL
 +__subdir_targets=
 .for __dir in ${SUBDIR}
 -${__target}: ${__target}_subdir_${__dir}
 +.if ${__wait} == ${__dir}
 +__subdir_targets+= .WAIT
 +.else
 +__subdir_targets+= ${__target}_subdir_${__dir}
 ${__target}_subdir_${__dir}: .MAKE
  @${_+_}set -e; \
  if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \
 @@ -85,7 +90,9 @@ distribute:
  fi; \
  ${MAKE} ${__target:realinstall=install} \
  DIRPRFX=${DIRPRFX}$$edir/
 +.endif
 .endfor
 +${__target}: ${__subdir_targets}
 .else
 ${__target}: _SUBDIR
 .endif
 
 
 

___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail 

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-04-24 Thread Ian Lepore
I couldn't think of a way to do that, because the directory itself isn't
the dependency.  That is, whether one directory has a newer/older
timestamp than the other isn't a function of the build process.  The
fact that someone else mentioned inserting .WAIT into the list is
something I took as a confirmation that the regular dependency mechanism
wouldn't work (so maybe I didn't think about it hard enough).

-- Ian

On Thu, 2014-04-24 at 11:17 -0600, Justin T. Gibbs wrote:
 Why wouldn’t we fix these with explicit subdir dependencies (e.g: subdira: 
 subdirb)?  I’m pretty sure I had this working with a hacked up bsd.subdir.mk 
 at a previous job, but my memory is hazy.
 
 —
 Justin
   
 On Apr 21, 2014, at 7:35 AM, Warner Losh i...@bsdimp.com wrote:
 
  (sorry for the top post)
  
  This looks good to my eye. I’d be tempted to toss in a comment about the
  __wait=.WAIT construct is due to the primitive nature of bmake’s parser
  so it runs afoul of the .for/.if construction rules and this is needed to 
  expand
  the for variable. It tripped me up when I looked at it, until I recalled a 
  comment
  from similar code in NetBSD.
  
  Warner
  
  On Apr 21, 2014, at 7:15 AM, Ian Lepore i...@freebsd.org wrote:
  
  On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote:
  On 27 Mar 2014, at 19:12, Jilles Tjoelker jil...@stack.nl wrote:
  On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote:
  On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
  Author: dim
  Date: Wed Mar 26 22:30:38 2014
  New Revision: 263778
  URL: http://svnweb.freebsd.org/changeset/base/263778
  
  Log:
  Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
  all the SUBDIR entries in parallel, instead of serially.  Apply this
  option to a selected number of Makefiles, which can greatly speed up 
  the
  build on multi-core machines, when using make -j.
  
  This can be extended to more Makefiles later on, whenever they are
  verified to work correctly with parallel building.
  
  Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are
  there any known bad dependencies this introduces?
  
  I'm paranoid about build systems ;) It is easy to add dependencies
  across directories and as long as directories are built in sequence,
  nothing goes wrong.
  
  In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as well,
  but this caused mysterious failures with some kernels such as mips
  ADM5120.
  
  There are a bunch of other parts that don't really like parallel builds
  at the moment.  For example, gnu/usr.bin/binutils needs its libraries
  (libbfd.a, etc) built first, before it can link the programs.  Similar
  for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being able
  to build the rest of gcc.
  
  Most of these cases can hopefully be solved by adding .WAIT targets at
  strategic points in the SUBDIR lists, but this also needs a bit of extra
  logic in bsd.subdir.mk.
  
  -Dimitry
  
  
  It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL for
  $work, so I came up with the attached, does this look okay to commit?
  
  -- Ian
  
  diff -r 67802e319fc6 share/mk/bsd.subdir.mk
  --- a/share/mk/bsd.subdir.mk   Sun Apr 20 21:01:07 2014 -0600
  +++ b/share/mk/bsd.subdir.mk   Mon Apr 21 06:59:37 2014 -0600
  @@ -4,10 +4,10 @@
  # The include file bsd.subdir.mk contains the default targets
  # for building subdirectories.
  #
  -# For all of the directories listed in the variable SUBDIRS, the
  +# For all of the directories listed in the variable SUBDIR, the
  # specified directory will be visited and the target made. There is
  # also a default target which allows the command make subdir where
  -# subdir is any directory listed in the variable SUBDIRS.
  +# subdir is any directory listed in the variable SUBDIR.
  #
  #
  # +++ variables +++
  @@ -42,7 +42,7 @@ distribute:
  
  _SUBDIR: .USE
  .if defined(SUBDIR)  !empty(SUBDIR)  !defined(NO_SUBDIR)
  -  @${_+_}for entry in ${SUBDIR}; do \
  +  @${_+_}for entry in ${SUBDIR:N.WAIT}; do \
 if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \
 ${ECHODIR} === ${DIRPRFX}$${entry}.${MACHINE_ARCH} 
  (${.TARGET:realinstall=install}); \
 edir=$${entry}.${MACHINE_ARCH}; \
  @@ -57,7 +57,7 @@ distribute:
 done
  .endif
  
  -${SUBDIR}: .PHONY
  +${SUBDIR:N.WAIT}: .PHONY
 ${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \
 cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \
 else \
  @@ -65,13 +65,18 @@ distribute:
 fi; \
 ${MAKE} all
  
  +__wait=.WAIT
  .for __target in all all-man checkdpadd clean cleandepend cleandir \
 depend distribute lint maninstall manlint \
 obj objlink realinstall regress tags \
 ${SUBDIR_TARGETS}
  .ifdef SUBDIR_PARALLEL
  +__subdir_targets=
  .for __dir in ${SUBDIR}
  -${__target}: ${__target}_subdir_${__dir}
  +.if ${__wait} == ${__dir}
  +__subdir_targets+= .WAIT
  +.else
  

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-04-24 Thread Jilles Tjoelker
On Thu, Apr 24, 2014 at 12:14:06PM -0600, Ian Lepore wrote:
 On Thu, 2014-04-24 at 11:17 -0600, Justin T. Gibbs wrote:
  Why wouldn’t we fix these with explicit subdir dependencies (e.g:
  subdira: subdirb)?  I’m pretty sure I had this working with a hacked
  up bsd.subdir.mk at a previous job, but my memory is hazy.

 I couldn't think of a way to do that, because the directory itself isn't
 the dependency.  That is, whether one directory has a newer/older
 timestamp than the other isn't a function of the build process.  The
 fact that someone else mentioned inserting .WAIT into the list is
 something I took as a confirmation that the regular dependency mechanism
 wouldn't work (so maybe I didn't think about it hard enough).

You can add dependencies between the targets generated by bsd.subdir.mk:
${target}_subdir_${subdir}. This will probably be more verbose in the
individual Makefiles than .WAIT, though.

-- 
Jilles Tjoelker
___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-04-24 Thread Warner Losh
The dependency would have to be on the artificial targets that
are created, and could easily be done. .WAIT is good for things
like libc where everything is dependent on it, but poor for other
dependencies which would be expressed as individual
dependencies.

Warner

On Apr 24, 2014, at 12:14 PM, Ian Lepore i...@freebsd.org wrote:

 I couldn't think of a way to do that, because the directory itself isn't
 the dependency.  That is, whether one directory has a newer/older
 timestamp than the other isn't a function of the build process.  The
 fact that someone else mentioned inserting .WAIT into the list is
 something I took as a confirmation that the regular dependency mechanism
 wouldn't work (so maybe I didn't think about it hard enough).
 
 -- Ian
 
 On Thu, 2014-04-24 at 11:17 -0600, Justin T. Gibbs wrote:
 Why wouldn’t we fix these with explicit subdir dependencies (e.g: subdira: 
 subdirb)?  I’m pretty sure I had this working with a hacked up bsd.subdir.mk 
 at a previous job, but my memory is hazy.
 
 —
 Justin
 
 On Apr 21, 2014, at 7:35 AM, Warner Losh i...@bsdimp.com wrote:
 
 (sorry for the top post)
 
 This looks good to my eye. I’d be tempted to toss in a comment about the
 __wait=.WAIT construct is due to the primitive nature of bmake’s parser
 so it runs afoul of the .for/.if construction rules and this is needed to 
 expand
 the for variable. It tripped me up when I looked at it, until I recalled a 
 comment
 from similar code in NetBSD.
 
 Warner
 
 On Apr 21, 2014, at 7:15 AM, Ian Lepore i...@freebsd.org wrote:
 
 On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote:
 On 27 Mar 2014, at 19:12, Jilles Tjoelker jil...@stack.nl wrote:
 On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote:
 On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
 Author: dim
 Date: Wed Mar 26 22:30:38 2014
 New Revision: 263778
 URL: http://svnweb.freebsd.org/changeset/base/263778
 
 Log:
 Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
 all the SUBDIR entries in parallel, instead of serially.  Apply this
 option to a selected number of Makefiles, which can greatly speed up 
 the
 build on multi-core machines, when using make -j.
 
 This can be extended to more Makefiles later on, whenever they are
 verified to work correctly with parallel building.
 
 Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are
 there any known bad dependencies this introduces?
 
 I'm paranoid about build systems ;) It is easy to add dependencies
 across directories and as long as directories are built in sequence,
 nothing goes wrong.
 
 In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as well,
 but this caused mysterious failures with some kernels such as mips
 ADM5120.
 
 There are a bunch of other parts that don't really like parallel builds
 at the moment.  For example, gnu/usr.bin/binutils needs its libraries
 (libbfd.a, etc) built first, before it can link the programs.  Similar
 for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being able
 to build the rest of gcc.
 
 Most of these cases can hopefully be solved by adding .WAIT targets at
 strategic points in the SUBDIR lists, but this also needs a bit of extra
 logic in bsd.subdir.mk.
 
 -Dimitry
 
 
 It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL for
 $work, so I came up with the attached, does this look okay to commit?
 
 -- Ian
 
 diff -r 67802e319fc6 share/mk/bsd.subdir.mk
 --- a/share/mk/bsd.subdir.mk   Sun Apr 20 21:01:07 2014 -0600
 +++ b/share/mk/bsd.subdir.mk   Mon Apr 21 06:59:37 2014 -0600
 @@ -4,10 +4,10 @@
 # The include file bsd.subdir.mk contains the default targets
 # for building subdirectories.
 #
 -# For all of the directories listed in the variable SUBDIRS, the
 +# For all of the directories listed in the variable SUBDIR, the
 # specified directory will be visited and the target made. There is
 # also a default target which allows the command make subdir where
 -# subdir is any directory listed in the variable SUBDIRS.
 +# subdir is any directory listed in the variable SUBDIR.
 #
 #
 # +++ variables +++
 @@ -42,7 +42,7 @@ distribute:
 
 _SUBDIR: .USE
 .if defined(SUBDIR)  !empty(SUBDIR)  !defined(NO_SUBDIR)
 -  @${_+_}for entry in ${SUBDIR}; do \
 +  @${_+_}for entry in ${SUBDIR:N.WAIT}; do \
if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \
${ECHODIR} === ${DIRPRFX}$${entry}.${MACHINE_ARCH} 
 (${.TARGET:realinstall=install}); \
edir=$${entry}.${MACHINE_ARCH}; \
 @@ -57,7 +57,7 @@ distribute:
done
 .endif
 
 -${SUBDIR}: .PHONY
 +${SUBDIR:N.WAIT}: .PHONY
${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \
cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \
else \
 @@ -65,13 +65,18 @@ distribute:
fi; \
${MAKE} all
 
 +__wait=.WAIT
 .for __target in all all-man checkdpadd clean cleandepend cleandir \
   depend distribute lint maninstall manlint \
   obj objlink 

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-04-24 Thread Ian Lepore
On Thu, 2014-04-24 at 20:19 +0200, Jilles Tjoelker wrote:
 On Thu, Apr 24, 2014 at 12:14:06PM -0600, Ian Lepore wrote:
  On Thu, 2014-04-24 at 11:17 -0600, Justin T. Gibbs wrote:
   Why wouldn’t we fix these with explicit subdir dependencies (e.g:
   subdira: subdirb)?  I’m pretty sure I had this working with a hacked
   up bsd.subdir.mk at a previous job, but my memory is hazy.
 
  I couldn't think of a way to do that, because the directory itself isn't
  the dependency.  That is, whether one directory has a newer/older
  timestamp than the other isn't a function of the build process.  The
  fact that someone else mentioned inserting .WAIT into the list is
  something I took as a confirmation that the regular dependency mechanism
  wouldn't work (so maybe I didn't think about it hard enough).
 
 You can add dependencies between the targets generated by bsd.subdir.mk:
 ${target}_subdir_${subdir}. This will probably be more verbose in the
 individual Makefiles than .WAIT, though.
 

What I was hoping is that we would eventually sprinkle 3 or 4 or 5
strategic .WAITs throughout the source tree and it would all be good.
Maybe that's too optimistic.

-- Ian


___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org


Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-04-21 Thread Ian Lepore
On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote:
 On 27 Mar 2014, at 19:12, Jilles Tjoelker jil...@stack.nl wrote:
  On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote:
  On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
  Author: dim
  Date: Wed Mar 26 22:30:38 2014
  New Revision: 263778
  URL: http://svnweb.freebsd.org/changeset/base/263778
  
  Log:
  Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
  all the SUBDIR entries in parallel, instead of serially.  Apply this
  option to a selected number of Makefiles, which can greatly speed up the
  build on multi-core machines, when using make -j.
  
  This can be extended to more Makefiles later on, whenever they are
  verified to work correctly with parallel building.
  
  Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are
  there any known bad dependencies this introduces?
  
  I'm paranoid about build systems ;) It is easy to add dependencies
  across directories and as long as directories are built in sequence,
  nothing goes wrong.
  
  In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as well,
  but this caused mysterious failures with some kernels such as mips
  ADM5120.
 
 There are a bunch of other parts that don't really like parallel builds
 at the moment.  For example, gnu/usr.bin/binutils needs its libraries
 (libbfd.a, etc) built first, before it can link the programs.  Similar
 for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being able
 to build the rest of gcc.
 
 Most of these cases can hopefully be solved by adding .WAIT targets at
 strategic points in the SUBDIR lists, but this also needs a bit of extra
 logic in bsd.subdir.mk.
 
 -Dimitry
  

It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL for
$work, so I came up with the attached, does this look okay to commit?

-- Ian

diff -r 67802e319fc6 share/mk/bsd.subdir.mk
--- a/share/mk/bsd.subdir.mk	Sun Apr 20 21:01:07 2014 -0600
+++ b/share/mk/bsd.subdir.mk	Mon Apr 21 06:59:37 2014 -0600
@@ -4,10 +4,10 @@
 # The include file bsd.subdir.mk contains the default targets
 # for building subdirectories.
 #
-# For all of the directories listed in the variable SUBDIRS, the
+# For all of the directories listed in the variable SUBDIR, the
 # specified directory will be visited and the target made. There is
 # also a default target which allows the command make subdir where
-# subdir is any directory listed in the variable SUBDIRS.
+# subdir is any directory listed in the variable SUBDIR.
 #
 #
 # +++ variables +++
@@ -42,7 +42,7 @@ distribute:
 
 _SUBDIR: .USE
 .if defined(SUBDIR)  !empty(SUBDIR)  !defined(NO_SUBDIR)
-	@${_+_}for entry in ${SUBDIR}; do \
+	@${_+_}for entry in ${SUBDIR:N.WAIT}; do \
 		if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \
 			${ECHODIR} === ${DIRPRFX}$${entry}.${MACHINE_ARCH} (${.TARGET:realinstall=install}); \
 			edir=$${entry}.${MACHINE_ARCH}; \
@@ -57,7 +57,7 @@ distribute:
 	done
 .endif
 
-${SUBDIR}: .PHONY
+${SUBDIR:N.WAIT}: .PHONY
 	${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \
 		cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \
 	else \
@@ -65,13 +65,18 @@ distribute:
 	fi; \
 	${MAKE} all
 
+__wait=.WAIT
 .for __target in all all-man checkdpadd clean cleandepend cleandir \
 depend distribute lint maninstall manlint \
 obj objlink realinstall regress tags \
 ${SUBDIR_TARGETS}
 .ifdef SUBDIR_PARALLEL
+__subdir_targets=
 .for __dir in ${SUBDIR}
-${__target}: ${__target}_subdir_${__dir}
+.if ${__wait} == ${__dir}
+__subdir_targets+= .WAIT
+.else
+__subdir_targets+= ${__target}_subdir_${__dir}
 ${__target}_subdir_${__dir}: .MAKE
 	@${_+_}set -e; \
 		if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \
@@ -85,7 +90,9 @@ distribute:
 		fi; \
 		${MAKE} ${__target:realinstall=install} \
 		DIRPRFX=${DIRPRFX}$$edir/
+.endif
 .endfor
+${__target}: ${__subdir_targets}
 .else
 ${__target}: _SUBDIR
 .endif
___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-04-21 Thread Warner Losh
(sorry for the top post)

This looks good to my eye. I’d be tempted to toss in a comment about the
__wait=.WAIT construct is due to the primitive nature of bmake’s parser
so it runs afoul of the .for/.if construction rules and this is needed to expand
the for variable. It tripped me up when I looked at it, until I recalled a 
comment
from similar code in NetBSD.

Warner

On Apr 21, 2014, at 7:15 AM, Ian Lepore i...@freebsd.org wrote:

 On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote:
 On 27 Mar 2014, at 19:12, Jilles Tjoelker jil...@stack.nl wrote:
 On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote:
 On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
 Author: dim
 Date: Wed Mar 26 22:30:38 2014
 New Revision: 263778
 URL: http://svnweb.freebsd.org/changeset/base/263778
 
 Log:
 Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
 all the SUBDIR entries in parallel, instead of serially.  Apply this
 option to a selected number of Makefiles, which can greatly speed up the
 build on multi-core machines, when using make -j.
 
 This can be extended to more Makefiles later on, whenever they are
 verified to work correctly with parallel building.
 
 Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are
 there any known bad dependencies this introduces?
 
 I'm paranoid about build systems ;) It is easy to add dependencies
 across directories and as long as directories are built in sequence,
 nothing goes wrong.
 
 In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as well,
 but this caused mysterious failures with some kernels such as mips
 ADM5120.
 
 There are a bunch of other parts that don't really like parallel builds
 at the moment.  For example, gnu/usr.bin/binutils needs its libraries
 (libbfd.a, etc) built first, before it can link the programs.  Similar
 for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being able
 to build the rest of gcc.
 
 Most of these cases can hopefully be solved by adding .WAIT targets at
 strategic points in the SUBDIR lists, but this also needs a bit of extra
 logic in bsd.subdir.mk.
 
 -Dimitry
 
 
 It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL for
 $work, so I came up with the attached, does this look okay to commit?
 
 -- Ian
 
 diff -r 67802e319fc6 share/mk/bsd.subdir.mk
 --- a/share/mk/bsd.subdir.mk  Sun Apr 20 21:01:07 2014 -0600
 +++ b/share/mk/bsd.subdir.mk  Mon Apr 21 06:59:37 2014 -0600
 @@ -4,10 +4,10 @@
 # The include file bsd.subdir.mk contains the default targets
 # for building subdirectories.
 #
 -# For all of the directories listed in the variable SUBDIRS, the
 +# For all of the directories listed in the variable SUBDIR, the
 # specified directory will be visited and the target made. There is
 # also a default target which allows the command make subdir where
 -# subdir is any directory listed in the variable SUBDIRS.
 +# subdir is any directory listed in the variable SUBDIR.
 #
 #
 # +++ variables +++
 @@ -42,7 +42,7 @@ distribute:
 
 _SUBDIR: .USE
 .if defined(SUBDIR)  !empty(SUBDIR)  !defined(NO_SUBDIR)
 - @${_+_}for entry in ${SUBDIR}; do \
 + @${_+_}for entry in ${SUBDIR:N.WAIT}; do \
   if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \
   ${ECHODIR} === ${DIRPRFX}$${entry}.${MACHINE_ARCH} 
 (${.TARGET:realinstall=install}); \
   edir=$${entry}.${MACHINE_ARCH}; \
 @@ -57,7 +57,7 @@ distribute:
   done
 .endif
 
 -${SUBDIR}: .PHONY
 +${SUBDIR:N.WAIT}: .PHONY
   ${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \
   cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \
   else \
 @@ -65,13 +65,18 @@ distribute:
   fi; \
   ${MAKE} all
 
 +__wait=.WAIT
 .for __target in all all-man checkdpadd clean cleandepend cleandir \
 depend distribute lint maninstall manlint \
 obj objlink realinstall regress tags \
 ${SUBDIR_TARGETS}
 .ifdef SUBDIR_PARALLEL
 +__subdir_targets=
 .for __dir in ${SUBDIR}
 -${__target}: ${__target}_subdir_${__dir}
 +.if ${__wait} == ${__dir}
 +__subdir_targets+= .WAIT
 +.else
 +__subdir_targets+= ${__target}_subdir_${__dir}
 ${__target}_subdir_${__dir}: .MAKE
   @${_+_}set -e; \
   if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \
 @@ -85,7 +90,9 @@ distribute:
   fi; \
   ${MAKE} ${__target:realinstall=install} \
   DIRPRFX=${DIRPRFX}$$edir/
 +.endif
 .endfor
 +${__target}: ${__subdir_targets}
 .else
 ${__target}: _SUBDIR
 .endif

___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-27 Thread John Baldwin
On Wednesday, March 26, 2014 6:30:39 pm Dimitry Andric wrote:
 Author: dim
 Date: Wed Mar 26 22:30:38 2014
 New Revision: 263778
 URL: http://svnweb.freebsd.org/changeset/base/263778
 
 Log:
   Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
   all the SUBDIR entries in parallel, instead of serially.  Apply this
   option to a selected number of Makefiles, which can greatly speed up the
   build on multi-core machines, when using make -j.
   
   This can be extended to more Makefiles later on, whenever they are
   verified to work correctly with parallel building.
   
   I tested this on a 24-core machine, with make -j48 buildworld (N = 6):
   
   beforestddev   afterstddev
   ===   ==   ===  ==
   real time1741.1 16.5 959.8 2.7
   user time   12468.7 16.4   14393.016.8
   sys  time1825.0 54.82110.622.8
   
   (user+sys)/real 8.2   17.1
   
   E.g. the build was approximately 45% faster in real time.  On machines
   with less cores, or with lower -j settings, the speedup will not be as
   impressive.  But at least you can now almost max out a machine with
   buildworld!
   
   Submitted by:   jilles
   MFC after:  2 weeks

Nice!

-- 
John Baldwin
___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org


Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-27 Thread David Chisnall
On 26 Mar 2014, at 22:30, Dimitry Andric d...@freebsd.org wrote:

  Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
  all the SUBDIR entries in parallel, instead of serially.  Apply this
  option to a selected number of Makefiles, which can greatly speed up the
  build on multi-core machines, when using make -j.

THANK YOU!  That's really excellent.  We can probably parallelise pretty much 
all of usr.lib and usr.bin as well, but going from using 8 cores to 17 is a 
very nice improvement.  This should help tinderbox / Jenkins build a LOT!

David

___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org


Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-27 Thread Bryan Drewery

On 2014-03-27 10:39, David Chisnall wrote:

On 26 Mar 2014, at 22:30, Dimitry Andric d...@freebsd.org wrote:

 Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to 
process

 all the SUBDIR entries in parallel, instead of serially.  Apply this
 option to a selected number of Makefiles, which can greatly speed up 
the

 build on multi-core machines, when using make -j.


THANK YOU!  That's really excellent.  We can probably parallelise
pretty much all of usr.lib and usr.bin as well, but going from using 8
cores to 17 is a very nice improvement.  This should help tinderbox /
Jenkins build a LOT!

David


I'll do some wider tests on the portmgr boxes. We have some 32 cpu 
systems.


--
Regards,
Bryan Drewery
___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org


Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-27 Thread Warner Losh
Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are there any 
known bad dependencies this introduces?

Warner

On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:

 Author: dim
 Date: Wed Mar 26 22:30:38 2014
 New Revision: 263778
 URL: http://svnweb.freebsd.org/changeset/base/263778
 
 Log:
  Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
  all the SUBDIR entries in parallel, instead of serially.  Apply this
  option to a selected number of Makefiles, which can greatly speed up the
  build on multi-core machines, when using make -j.
 
  This can be extended to more Makefiles later on, whenever they are
  verified to work correctly with parallel building.
 
  I tested this on a 24-core machine, with make -j48 buildworld (N = 6):
 
  beforestddev   afterstddev
  ===   ==   ===  ==
  real time1741.1 16.5 959.8 2.7
  user time   12468.7 16.4   14393.016.8
  sys  time1825.0 54.82110.622.8
 
  (user+sys)/real 8.2   17.1
 
  E.g. the build was approximately 45% faster in real time.  On machines
  with less cores, or with lower -j settings, the speedup will not be as
  impressive.  But at least you can now almost max out a machine with
  buildworld!
 
  Submitted by:jilles
  MFC after:   2 weeks
 
 Modified:
  head/bin/Makefile
  head/lib/Makefile
  head/lib/clang/Makefile
  head/sbin/Makefile
  head/share/mk/bsd.subdir.mk
  head/usr.bin/Makefile
  head/usr.sbin/Makefile
 
 Modified: head/bin/Makefile
 ==
 --- head/bin/Makefile Wed Mar 26 20:43:40 2014(r263777)
 +++ head/bin/Makefile Wed Mar 26 22:30:38 2014(r263778)
 @@ -60,4 +60,6 @@ SUBDIR+=tests
 
 SUBDIR:=  ${SUBDIR:O}
 
 +SUBDIR_PARALLEL=
 +
 .include bsd.subdir.mk
 
 Modified: head/lib/Makefile
 ==
 --- head/lib/Makefile Wed Mar 26 20:43:40 2014(r263777)
 +++ head/lib/Makefile Wed Mar 26 22:30:38 2014(r263778)
 @@ -276,4 +276,8 @@ afterinstall:
   ${INSTALL_SYMLINK} ../include ${DESTDIR}/usr/lib/include
 .endif
 
 +.if !make(install)
 +SUBDIR_PARALLEL=
 +.endif
 +
 .include bsd.subdir.mk
 
 Modified: head/lib/clang/Makefile
 ==
 --- head/lib/clang/Makefile   Wed Mar 26 20:43:40 2014(r263777)
 +++ head/lib/clang/Makefile   Wed Mar 26 22:30:38 2014(r263778)
 @@ -147,4 +147,6 @@ SUBDIR+=liblldb \
 
 SUBDIR+= include
 
 +SUBDIR_PARALLEL=
 +
 .include bsd.subdir.mk
 
 Modified: head/sbin/Makefile
 ==
 --- head/sbin/MakefileWed Mar 26 20:43:40 2014(r263777)
 +++ head/sbin/MakefileWed Mar 26 22:30:38 2014(r263778)
 @@ -126,4 +126,6 @@ SUBDIR+=  tests
 
 SUBDIR:=  ${SUBDIR:O}
 
 +SUBDIR_PARALLEL=
 +
 .include bsd.subdir.mk
 
 Modified: head/share/mk/bsd.subdir.mk
 ==
 --- head/share/mk/bsd.subdir.mk   Wed Mar 26 20:43:40 2014
 (r263777)
 +++ head/share/mk/bsd.subdir.mk   Wed Mar 26 22:30:38 2014
 (r263778)
 @@ -71,7 +71,26 @@ ${SUBDIR}: .PHONY .MAKE
 .for __target in all all-man checkdpadd clean cleandepend cleandir \
 cleanilinks depend distribute lint maninstall manlint obj objlink \
 realinstall regress tags ${SUBDIR_TARGETS}
 +.ifdef SUBDIR_PARALLEL
 +.for __dir in ${SUBDIR}
 +${__target}: ${__target}_subdir_${__dir}
 +${__target}_subdir_${__dir}: .MAKE
 + @${_+_}set -e; \
 + if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \
 + ${ECHODIR} === ${DIRPRFX}${__dir}.${MACHINE_ARCH} 
 (${__target:realinstall=install}); \
 + edir=${__dir}.${MACHINE_ARCH}; \
 + cd ${.CURDIR}/$${edir}; \
 + else \
 + ${ECHODIR} === ${DIRPRFX}${__dir} 
 (${__target:realinstall=install}); \
 + edir=${__dir}; \
 + cd ${.CURDIR}/$${edir}; \
 + fi; \
 + ${MAKE} ${__target:realinstall=install} \
 + DIRPRFX=${DIRPRFX}$$edir/
 +.endfor
 +.else
 ${__target}: _SUBDIR
 +.endif
 .endfor
 
 .for __target in files includes
 
 Modified: head/usr.bin/Makefile
 ==
 --- head/usr.bin/Makefile Wed Mar 26 20:43:40 2014(r263777)
 +++ head/usr.bin/Makefile Wed Mar 26 22:30:38 2014(r263778)
 @@ -379,4 +379,6 @@ SUBDIR+=  svn
 
 SUBDIR:=  ${SUBDIR:O}
 
 +SUBDIR_PARALLEL=
 +
 .include bsd.subdir.mk
 
 Modified: head/usr.sbin/Makefile
 

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-27 Thread Ian Lepore
It's also a bit odd that the way of opting in makes it appear as if
you're opting out.

-- Ian

On Thu, 2014-03-27 at 11:05 -0600, Warner Losh wrote:
 Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are there 
 any known bad dependencies this introduces?
 
 Warner
 
 On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
 
  Author: dim
  Date: Wed Mar 26 22:30:38 2014
  New Revision: 263778
  URL: http://svnweb.freebsd.org/changeset/base/263778
  
  Log:
   Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
   all the SUBDIR entries in parallel, instead of serially.  Apply this
   option to a selected number of Makefiles, which can greatly speed up the
   build on multi-core machines, when using make -j.
  
   This can be extended to more Makefiles later on, whenever they are
   verified to work correctly with parallel building.
  
   I tested this on a 24-core machine, with make -j48 buildworld (N = 6):
  
   beforestddev   afterstddev
   ===   ==   ===  ==
   real time1741.1 16.5 959.8 2.7
   user time   12468.7 16.4   14393.016.8
   sys  time1825.0 54.82110.622.8
  
   (user+sys)/real 8.2   17.1
  
   E.g. the build was approximately 45% faster in real time.  On machines
   with less cores, or with lower -j settings, the speedup will not be as
   impressive.  But at least you can now almost max out a machine with
   buildworld!
  
   Submitted by:  jilles
   MFC after: 2 weeks
  
  Modified:
   head/bin/Makefile
   head/lib/Makefile
   head/lib/clang/Makefile
   head/sbin/Makefile
   head/share/mk/bsd.subdir.mk
   head/usr.bin/Makefile
   head/usr.sbin/Makefile
  
  Modified: head/bin/Makefile
  ==
  --- head/bin/Makefile   Wed Mar 26 20:43:40 2014(r263777)
  +++ head/bin/Makefile   Wed Mar 26 22:30:38 2014(r263778)
  @@ -60,4 +60,6 @@ SUBDIR+=  tests
  
  SUBDIR:=${SUBDIR:O}
  
  +SUBDIR_PARALLEL=
  +
  .include bsd.subdir.mk
  
  Modified: head/lib/Makefile
  ==
  --- head/lib/Makefile   Wed Mar 26 20:43:40 2014(r263777)
  +++ head/lib/Makefile   Wed Mar 26 22:30:38 2014(r263778)
  @@ -276,4 +276,8 @@ afterinstall:
  ${INSTALL_SYMLINK} ../include ${DESTDIR}/usr/lib/include
  .endif
  
  +.if !make(install)
  +SUBDIR_PARALLEL=
  +.endif
  +
  .include bsd.subdir.mk
  
  Modified: head/lib/clang/Makefile
  ==
  --- head/lib/clang/Makefile Wed Mar 26 20:43:40 2014(r263777)
  +++ head/lib/clang/Makefile Wed Mar 26 22:30:38 2014(r263778)
  @@ -147,4 +147,6 @@ SUBDIR+=liblldb \
  
  SUBDIR+= include
  
  +SUBDIR_PARALLEL=
  +
  .include bsd.subdir.mk
  
  Modified: head/sbin/Makefile
  ==
  --- head/sbin/Makefile  Wed Mar 26 20:43:40 2014(r263777)
  +++ head/sbin/Makefile  Wed Mar 26 22:30:38 2014(r263778)
  @@ -126,4 +126,6 @@ SUBDIR+=tests
  
  SUBDIR:=${SUBDIR:O}
  
  +SUBDIR_PARALLEL=
  +
  .include bsd.subdir.mk
  
  Modified: head/share/mk/bsd.subdir.mk
  ==
  --- head/share/mk/bsd.subdir.mk Wed Mar 26 20:43:40 2014
  (r263777)
  +++ head/share/mk/bsd.subdir.mk Wed Mar 26 22:30:38 2014
  (r263778)
  @@ -71,7 +71,26 @@ ${SUBDIR}: .PHONY .MAKE
  .for __target in all all-man checkdpadd clean cleandepend cleandir \
  cleanilinks depend distribute lint maninstall manlint obj objlink \
  realinstall regress tags ${SUBDIR_TARGETS}
  +.ifdef SUBDIR_PARALLEL
  +.for __dir in ${SUBDIR}
  +${__target}: ${__target}_subdir_${__dir}
  +${__target}_subdir_${__dir}: .MAKE
  +   @${_+_}set -e; \
  +   if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \
  +   ${ECHODIR} === ${DIRPRFX}${__dir}.${MACHINE_ARCH} 
  (${__target:realinstall=install}); \
  +   edir=${__dir}.${MACHINE_ARCH}; \
  +   cd ${.CURDIR}/$${edir}; \
  +   else \
  +   ${ECHODIR} === ${DIRPRFX}${__dir} 
  (${__target:realinstall=install}); \
  +   edir=${__dir}; \
  +   cd ${.CURDIR}/$${edir}; \
  +   fi; \
  +   ${MAKE} ${__target:realinstall=install} \
  +   DIRPRFX=${DIRPRFX}$$edir/
  +.endfor
  +.else
  ${__target}: _SUBDIR
  +.endif
  .endfor
  
  .for __target in files includes
  
  Modified: head/usr.bin/Makefile
  ==
  --- head/usr.bin/Makefile   Wed Mar 26 20:43:40 2014(r263777)
  +++ head/usr.bin/Makefile   

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-27 Thread Jilles Tjoelker
On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote:
 On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
  Author: dim
  Date: Wed Mar 26 22:30:38 2014
  New Revision: 263778
  URL: http://svnweb.freebsd.org/changeset/base/263778

  Log:
   Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
   all the SUBDIR entries in parallel, instead of serially.  Apply this
   option to a selected number of Makefiles, which can greatly speed up the
   build on multi-core machines, when using make -j.

   This can be extended to more Makefiles later on, whenever they are
   verified to work correctly with parallel building.

 Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are
 there any known bad dependencies this introduces?

I'm paranoid about build systems ;) It is easy to add dependencies
across directories and as long as directories are built in sequence,
nothing goes wrong.

In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as well,
but this caused mysterious failures with some kernels such as mips
ADM5120.

-- 
Jilles Tjoelker
___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org

Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-27 Thread Dimitry Andric
On 27 Mar 2014, at 19:12, Jilles Tjoelker jil...@stack.nl wrote:
 On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote:
 On Mar 26, 2014, at 4:30 PM, Dimitry Andric d...@freebsd.org wrote:
 Author: dim
 Date: Wed Mar 26 22:30:38 2014
 New Revision: 263778
 URL: http://svnweb.freebsd.org/changeset/base/263778
 
 Log:
 Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
 all the SUBDIR entries in parallel, instead of serially.  Apply this
 option to a selected number of Makefiles, which can greatly speed up the
 build on multi-core machines, when using make -j.
 
 This can be extended to more Makefiles later on, whenever they are
 verified to work correctly with parallel building.
 
 Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are
 there any known bad dependencies this introduces?
 
 I'm paranoid about build systems ;) It is easy to add dependencies
 across directories and as long as directories are built in sequence,
 nothing goes wrong.
 
 In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as well,
 but this caused mysterious failures with some kernels such as mips
 ADM5120.

There are a bunch of other parts that don't really like parallel builds
at the moment.  For example, gnu/usr.bin/binutils needs its libraries
(libbfd.a, etc) built first, before it can link the programs.  Similar
for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being able
to build the rest of gcc.

Most of these cases can hopefully be solved by adding .WAIT targets at
strategic points in the SUBDIR lists, but this also needs a bit of extra
logic in bsd.subdir.mk.

-Dimitry
 


signature.asc
Description: Message signed with OpenPGP using GPGMail


svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

2014-03-26 Thread Dimitry Andric
Author: dim
Date: Wed Mar 26 22:30:38 2014
New Revision: 263778
URL: http://svnweb.freebsd.org/changeset/base/263778

Log:
  Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
  all the SUBDIR entries in parallel, instead of serially.  Apply this
  option to a selected number of Makefiles, which can greatly speed up the
  build on multi-core machines, when using make -j.
  
  This can be extended to more Makefiles later on, whenever they are
  verified to work correctly with parallel building.
  
  I tested this on a 24-core machine, with make -j48 buildworld (N = 6):
  
  beforestddev   afterstddev
  ===   ==   ===  ==
  real time1741.1 16.5 959.8 2.7
  user time   12468.7 16.4   14393.016.8
  sys  time1825.0 54.82110.622.8
  
  (user+sys)/real 8.2   17.1
  
  E.g. the build was approximately 45% faster in real time.  On machines
  with less cores, or with lower -j settings, the speedup will not be as
  impressive.  But at least you can now almost max out a machine with
  buildworld!
  
  Submitted by: jilles
  MFC after:2 weeks

Modified:
  head/bin/Makefile
  head/lib/Makefile
  head/lib/clang/Makefile
  head/sbin/Makefile
  head/share/mk/bsd.subdir.mk
  head/usr.bin/Makefile
  head/usr.sbin/Makefile

Modified: head/bin/Makefile
==
--- head/bin/Makefile   Wed Mar 26 20:43:40 2014(r263777)
+++ head/bin/Makefile   Wed Mar 26 22:30:38 2014(r263778)
@@ -60,4 +60,6 @@ SUBDIR+=  tests
 
 SUBDIR:=   ${SUBDIR:O}
 
+SUBDIR_PARALLEL=
+
 .include bsd.subdir.mk

Modified: head/lib/Makefile
==
--- head/lib/Makefile   Wed Mar 26 20:43:40 2014(r263777)
+++ head/lib/Makefile   Wed Mar 26 22:30:38 2014(r263778)
@@ -276,4 +276,8 @@ afterinstall:
${INSTALL_SYMLINK} ../include ${DESTDIR}/usr/lib/include
 .endif
 
+.if !make(install)
+SUBDIR_PARALLEL=
+.endif
+
 .include bsd.subdir.mk

Modified: head/lib/clang/Makefile
==
--- head/lib/clang/Makefile Wed Mar 26 20:43:40 2014(r263777)
+++ head/lib/clang/Makefile Wed Mar 26 22:30:38 2014(r263778)
@@ -147,4 +147,6 @@ SUBDIR+=liblldb \
 
 SUBDIR+= include
 
+SUBDIR_PARALLEL=
+
 .include bsd.subdir.mk

Modified: head/sbin/Makefile
==
--- head/sbin/Makefile  Wed Mar 26 20:43:40 2014(r263777)
+++ head/sbin/Makefile  Wed Mar 26 22:30:38 2014(r263778)
@@ -126,4 +126,6 @@ SUBDIR+=tests
 
 SUBDIR:=   ${SUBDIR:O}
 
+SUBDIR_PARALLEL=
+
 .include bsd.subdir.mk

Modified: head/share/mk/bsd.subdir.mk
==
--- head/share/mk/bsd.subdir.mk Wed Mar 26 20:43:40 2014(r263777)
+++ head/share/mk/bsd.subdir.mk Wed Mar 26 22:30:38 2014(r263778)
@@ -71,7 +71,26 @@ ${SUBDIR}: .PHONY .MAKE
 .for __target in all all-man checkdpadd clean cleandepend cleandir \
 cleanilinks depend distribute lint maninstall manlint obj objlink \
 realinstall regress tags ${SUBDIR_TARGETS}
+.ifdef SUBDIR_PARALLEL
+.for __dir in ${SUBDIR}
+${__target}: ${__target}_subdir_${__dir}
+${__target}_subdir_${__dir}: .MAKE
+   @${_+_}set -e; \
+   if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \
+   ${ECHODIR} === ${DIRPRFX}${__dir}.${MACHINE_ARCH} 
(${__target:realinstall=install}); \
+   edir=${__dir}.${MACHINE_ARCH}; \
+   cd ${.CURDIR}/$${edir}; \
+   else \
+   ${ECHODIR} === ${DIRPRFX}${__dir} 
(${__target:realinstall=install}); \
+   edir=${__dir}; \
+   cd ${.CURDIR}/$${edir}; \
+   fi; \
+   ${MAKE} ${__target:realinstall=install} \
+   DIRPRFX=${DIRPRFX}$$edir/
+.endfor
+.else
 ${__target}: _SUBDIR
+.endif
 .endfor
 
 .for __target in files includes

Modified: head/usr.bin/Makefile
==
--- head/usr.bin/Makefile   Wed Mar 26 20:43:40 2014(r263777)
+++ head/usr.bin/Makefile   Wed Mar 26 22:30:38 2014(r263778)
@@ -379,4 +379,6 @@ SUBDIR+=svn
 
 SUBDIR:=   ${SUBDIR:O}
 
+SUBDIR_PARALLEL=
+
 .include bsd.subdir.mk

Modified: head/usr.sbin/Makefile
==
--- head/usr.sbin/Makefile  Wed Mar 26 20:43:40 2014(r263777)
+++ head/usr.sbin/Makefile  Wed Mar 26 22:30:38 2014(r263778)
@@ -320,4 +320,6 @@ SUBDIR+=wpa
 
 SUBDIR:=   ${SUBDIR:O}
 
+SUBDIR_PARALLEL=
+
 .include