INDEX build optimizations - please review
Please review and test the following patches that optimize port INDEX builds (and as a side-effect, other recursive tree traversals). I am particularly interested in a comparison between old and new indexes built locally: the only diff should be in audio/festvox-hvs [1]. The patches remove most of the extraneous command executions required for each invocation of bsd.port.mk (e.g. via 'make describe' in a port), by replacing external command invocations with shell or make builtins, as well as caching of constants. Another important optimization is to use /rescue/sh instead of /bin/sh for index builds. The former is statically linked and this is much faster to execute. One further change I didn't do would be to move the WWW specification from pkg-descr into a Makefile variable (this could be done mechanically). The new 'make describe' target runs entirely using shell builtins apart from the need to sed pkg-descr to extract the WWW [2] (previously 'describe' was a combination of several shell executions and a perl script, invoked for every port). With these patches an index build on an 8 core system drops from 796.486u 974.564s 5:25.14 544.7%28+193k 37252+719io 27pf+0w to 642.846u 164.520s 2:31.29 533.6%67+297k 0+721io 0pf+0w (or with statically linked sed: 637.805u 142.335s 2:27.32 529.5%71+304k 0+720io 0pf+0w ) The new version requires 69301 forks and 68614 execs (would be ~19000 fewer of each without WWW in pkg-descr), compared to 252383 forks and 226875 execs with the current version. The resulting index is identical except for one port [1] but generating it is more than twice as fast (and uses 6 times less system CPU). I am only getting ~530% CPU utilization because of contention in the scheduler, so there is scope for going as low as 100 seconds if this went away. Further improvements are no doubt possible but would require profiling the work done within make(1) to see where it is spending its time (variable setting, conditional evaluation, loop invocation, regexp processing, etc). ** * NOTE TO PORT DEVELOPERS ** Variable assignments with != are bad! Try as hard as you can to avoid using them -- especially in Mk/*! Every time something processes your makefile it will spawn a command, even if it is not relevant for the operation being performed. If you need to run shell commands, try to isolate them within a makefile target. You can avoid code duplication by assigning the *shell commands* (not their output) to a variable and inserting it into your code block. e.g. instead of -- VARIABLE!= do some shell stuff; do some other stuff target: echo ${VARIABLE} -- do this (or similar): -- VARIABLE_CMDS= do some shell stuff; do some other stuff target: echo $(${VARIABLE_CMDS}) -- This defers the command execution to the point where the target runs, so in the case when the target is *not* run, then you avoid wasting one or more process executions. Kris [1] This patch exposed a bug: for some reason no dependencies were previously recorded for audio/festvox-hvs! Probably because it only has RUN_DEPENDS and the 'make describe' perl script was broken. [2] Actually I am not happy with this but couldn't think of a way to do it better. Having to fork the subshell costs about 60 seconds of system time and 10 of wall time. Index: Makefile === RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Makefile,v retrieving revision 1.103 diff -u -r1.103 Makefile --- Makefile27 Sep 2007 05:36:26 - 1.103 +++ Makefile10 Jun 2008 09:56:17 - @@ -103,6 +103,7 @@ tmpdir=`/usr/bin/mktemp -d -t index` || exit 1; \ trap "rm -rf $${tmpdir}; exit 1" 1 2 3 5 10 13 15; \ ( cd ${.CURDIR} && make -j${INDEX_JOBS} INDEX_TMPDIR=$${tmpdir} BUILDING_INDEX=1 \ + __MAKE_SHELL=/rescue/sh \ ECHO_MSG="${INDEX_ECHO_MSG}" describe ) || \ (rm -rf $${tmpdir} ; \ if [ "${INDEX_QUIET}" = "" ]; then \ Index: Mk/bsd.java.mk === RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.java.mk,v retrieving revision 1.82 diff -u -r1.82 bsd.java.mk --- Mk/bsd.java.mk 28 Oct 2007 15:09:43 - 1.82 +++ Mk/bsd.java.mk 10 Jun 2008 09:56:17 - @@ -249,34 +249,30 @@ . endfor # Error checking: JAVA_VERSION +.if !defined(_JAVA_VERSION_LIST_REGEXP) _JAVA_VERSION_LIST_REGEXP!=${ECHO_CMD} "${_JAVA_VERSION_LIST}" | ${SED} "s/ /\\\|/g" -_ERROR_CHECKING_JAVA_VERSION!= ${ECHO_CMD} "${JAVA_VERSION}" | ${TR} " " "\n" \ - | ${GREP} -v "${_JAVA_VERSION_LIST_REGEXP}"
Re: INDEX build optimizations - please review
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Kris Kennaway wrote: | Another important optimization is to use /rescue/sh instead of /bin/sh | for index builds. The former is statically linked and this is much | faster to execute. True, but the for is not even guaranteed to exist (WITHOUT_RESCUE in src.conf or similar). I would opt for checking, and use it only if it exists | Kris - -- Pietro Cerutti [EMAIL PROTECTED] PGP Public Key: http://gahr.ch/pgp -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (FreeBSD) iEYEAREKAAYFAkhOsRUACgkQwMJqmJVx945PpgCeO6ylFV7Lnsizo1p0v9h1jJX1 1RYAn2jwtKuwiy4aXAAsST2CidupEKmO =hsgo -END PGP SIGNATURE- ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Pietro Cerutti wrote: | Kris Kennaway wrote: | | | Another important optimization is to use /rescue/sh instead of /bin/sh | | for index builds. The former is statically linked and this is much | | faster to execute. | | True, but the for is not even guaranteed to exist (WITHOUT_RESCUE in | src.conf or similar). "the for"??? err... "the file" I meant.. | | I would opt for checking, and use it only if it exists | | | Kris | - -- Pietro Cerutti [EMAIL PROTECTED] PGP Public Key: http://gahr.ch/pgp -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (FreeBSD) iEYEAREKAAYFAkhOsgMACgkQwMJqmJVx946UagCgoLs7no6NgcLPLSn0lzOHyB0M 4HcAn359kBKeDCevr4IM5cOFfEHh6p6q =81w7 -END PGP SIGNATURE- ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
On Tue, 10 Jun 2008 11:45:32 -0500, Kris Kennaway <[EMAIL PROTECTED]> wrote: Another important optimization is to use /rescue/sh instead of /bin/sh for index builds. The former is statically linked and this is much faster to execute. I don't have INDEX in all of my systems, so I am only take a peek. What about users that have WITHOUT_RESCUE defined or/and something different? IMO, I think your patch should have a check on if /rescue/sh exists when our src provides optional of WITHOUT_RESCUE and is harmless to kill the /rescue. -- .if exists(/rescue/sh) __MAKE_SHELL=/rescue/sh .else __MAKE_SHELL=/bin/sh .endif -- Thanks for hint on 'NOTE TO PORT DEVELOPERS' part as I didn't know about VARIABLE!= vs $(${VARIABLE_CMDS}). /me copies it in my note. -- As for the audio/festvox-hvs issue, probably because of no executables or files in RUN_DEPENDS: -- RUN_DEPENDS=${PORTSDIR}/audio/festival+OGI \ ${PORTSDIR}/audio/festlex-poslex \ ${PORTSDIR}/audio/festlex-ogi \ ${PORTSDIR}/audio/festogi-spanish -- Cheers, Mezz -- [EMAIL PROTECTED] - [EMAIL PROTECTED] FreeBSD GNOME Team http://www.FreeBSD.org/gnome/ - [EMAIL PROTECTED] ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Pietro Cerutti wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Pietro Cerutti wrote: | Kris Kennaway wrote: | | | Another important optimization is to use /rescue/sh instead of /bin/sh | | for index builds. The former is statically linked and this is much | | faster to execute. | | True, but the for is not even guaranteed to exist (WITHOUT_RESCUE in | src.conf or similar). "the for"??? err... "the file" I meant.. Yeah, I forgot to add a check for that. Kris ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Jeremy Messenger wrote: On Tue, 10 Jun 2008 11:45:32 -0500, Kris Kennaway <[EMAIL PROTECTED]> wrote: Another important optimization is to use /rescue/sh instead of /bin/sh for index builds. The former is statically linked and this is much faster to execute. I don't have INDEX in all of my systems, so I am only take a peek. What about users that have WITHOUT_RESCUE defined or/and something different? IMO, I think your patch should have a check on if /rescue/sh exists when our src provides optional of WITHOUT_RESCUE and is harmless to kill the /rescue. -- .if exists(/rescue/sh) __MAKE_SHELL=/rescue/sh .else __MAKE_SHELL=/bin/sh .endif -- Yeah, I will add this in the next version. Thanks for hint on 'NOTE TO PORT DEVELOPERS' part as I didn't know about VARIABLE!= vs $(${VARIABLE_CMDS}). /me copies it in my note. What would be good is if someone can add it to the porter's handbook. As for the audio/festvox-hvs issue, probably because of no executables or files in RUN_DEPENDS: -- RUN_DEPENDS=${PORTSDIR}/audio/festival+OGI \ ${PORTSDIR}/audio/festlex-poslex \ ${PORTSDIR}/audio/festlex-ogi \ ${PORTSDIR}/audio/festogi-spanish -- Ah yes, good catch. This looks like a port bug, so I don't think I need to modify my patches. Kris ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
On Tue, Jun 10, 2008 at 06:45:32PM +0200, Kris Kennaway wrote: > Please review and test the following patches that optimize port INDEX > builds (and as a side-effect, other recursive tree traversals). I am > particularly interested in a comparison between old and new indexes > built locally: the only diff should be in audio/festvox-hvs [1]. Work on RELENG_6 only after this patch (maybe good candidate for MFC): Index: usr.bin/make/var.c @@ -1232,6 +1232,41 @@ } /** + * Remove adjacent duplicate words. + * + * Results: + * A string containing the resulting words. + */ +static char * +VarUniq(const char *str) +{ + ArgArrayaa; + Buffer *buf; /* Buffer for new string */ + int i, j; + + buf = Buf_Init(0); + brk_string(&aa, str, FALSE); + + if (aa.argc > 2) { + for (j = 1, i = 2; i < aa.argc; i++) { + if (strcmp(aa.argv[i], aa.argv[j]) != 0 && (++j != i)) + aa.argv[j] = aa.argv[i]; + } + aa.argc = j + 1; + } + + for (i = 1; i < aa.argc; i++) { + Buf_AddBytes(buf, strlen(aa.argv[i]), (Byte *)aa.argv[i]); + if (i != aa.argc - 1) + Buf_AddByte(buf, ' '); + } + Buf_AddByte(buf, '\0'); + + ArgArray_Done(&aa); + return (Buf_Peel(buf)); +} + +/** * Pass through the tstr looking for 1) escaped delimiters, * '$'s and backslashes (place the escaped character in * uninterpreted) and 2) unescaped $'s that aren't before @@ -1681,6 +1716,7 @@ * the invocation. * :U Converts variable to upper-case. * :L Converts variable to lower-case. + * :u ("uniq") Remove adjacent duplicate words. * * XXXHB update this comment or remove it and point to the man page. */ @@ -1793,6 +1829,10 @@ newStr = VarModify(value, VarRoot, NULL); vp->ptr++; break; + case 'u': + newStr = VarUniq(value); + vp->ptr++; + break; default: newStr = sysVvarsub(vp, startc, v, value); break; -- Adios ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Alex Kozlov wrote: On Tue, Jun 10, 2008 at 06:45:32PM +0200, Kris Kennaway wrote: Please review and test the following patches that optimize port INDEX builds (and as a side-effect, other recursive tree traversals). I am particularly interested in a comparison between old and new indexes built locally: the only diff should be in audio/festvox-hvs [1]. Work on RELENG_6 only after this patch (maybe good candidate for MFC): Crap! I need :u! Without it the duplicate removal will have to be pushed into the perl script that collates the output, which means that Colin's portsnap script may be affected. The patch definitely should be MFCed though. Who committed it originally to HEAD? Kris ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
On Tue, Jun 10, 2008 at 11:52:01PM +0200, Kris Kennaway wrote: > Alex Kozlov wrote: > > On Tue, Jun 10, 2008 at 06:45:32PM +0200, Kris Kennaway wrote: > >> Please review and test the following patches that optimize port INDEX > >> builds (and as a side-effect, other recursive tree traversals). I am > >> particularly interested in a comparison between old and new indexes > >> built locally: the only diff should be in audio/festvox-hvs [1]. > > Work on RELENG_6 only after this patch (maybe good candidate for MFC): > > Crap! I need :u! Without it the duplicate removal will have to be > pushed into the perl script that collates the output, which means that > Colin's portsnap script may be affected. > > The patch definitely should be MFCed though. Who committed it > originally to HEAD? var.c Revision 1.161 Sat Apr 8 06:59:54 2006 UTC (2 years, 2 months ago) by fjoe -- Adios ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Alex Kozlov wrote: On Tue, Jun 10, 2008 at 11:52:01PM +0200, Kris Kennaway wrote: Alex Kozlov wrote: On Tue, Jun 10, 2008 at 06:45:32PM +0200, Kris Kennaway wrote: Please review and test the following patches that optimize port INDEX builds (and as a side-effect, other recursive tree traversals). I am particularly interested in a comparison between old and new indexes built locally: the only diff should be in audio/festvox-hvs [1]. Work on RELENG_6 only after this patch (maybe good candidate for MFC): Crap! I need :u! Without it the duplicate removal will have to be pushed into the perl script that collates the output, which means that Colin's portsnap script may be affected. The patch definitely should be MFCed though. Who committed it originally to HEAD? var.c Revision 1.161 Sat Apr 8 06:59:54 2006 UTC (2 years, 2 months ago) by fjoe Thanks, I have emailed Max. What I think I will do is add an .if ${OSVERSION} > 6xx check around it, and keep the old perl script as a fallback. In a couple of years when 6.3 is no longer supported we can GC the old code. Kris ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Kris Kennaway wrote: The new 'make describe' target runs entirely using shell builtins apart from the need to sed pkg-descr to extract the WWW [2] [2] Actually I am not happy with this but couldn't think of a way to do it better. Having to fork the subshell costs about 60 seconds of system time and 10 of wall time. Here's one way to do it. This is quick and dirty and I haven't benchmarked it, but I imagine it would be faster. while read one two discard; do case "$one" in WWW:) echo one: $one two: $two case "$two" in http://*) echo WWW= $two ;; *) echo WWW= http://$two ;; esac break ;; esac done < pkg-descr I did test this briefly and it pulls out the right values for variables with and without http://. hth, Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
On Tue, Jun 10, 2008 at 04:49:18PM -0700, Doug Barton wrote: > Kris Kennaway wrote: > > > The new 'make describe' target runs entirely using shell > > builtins apart from the need to sed pkg-descr to extract the WWW [2] > > > [2] Actually I am not happy with this but couldn't think of a way to do > > it better. Having to fork the subshell costs about 60 seconds of system > > time and 10 of wall time. > > Here's one way to do it. This is quick and dirty and I haven't > benchmarked it, but I imagine it would be faster. > > while read one two discard; do > case "$one" in > WWW:) echo one: $one two: $two > case "$two" in > http://*) echo WWW= $two ;; > *) echo WWW= http://$two ;; > esac > break > ;; > esac > done < pkg-descr > > I did test this briefly and it pulls out the right values for > variables with and without http://. Good idea. I also do something like that [1]. But before we can use this method, we have to fix ports with bad pkg-descr: Don't have \n in WWW: line databases/powerarchitect graphics/picturebook Have CRLF line terminators: devel/p5-Tie-Restore www/knowledgekit www/squishdot www/znavigator www/zope-cmfforum Have http:/ instead of http:// security/shimmer Don't have protocol prefix(optional): net/pdb sysutils/iograph [1] (Incremental to kris@ patch.): Index: Mk/bsd.port.mk @@ -5290,16 +5290,18 @@ _RUN_DEPENDS=${RUN_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} ${_LIB_DEPENDS} .if exists(${DESCR}) _DESCR=${DESCR} +_WWW=while read line; do set -- $${line}; case $$1 in WWW:) case $$2 \ +in http://*|https://*|ftp://*) ${ECHO_CMD} $$2 ;; *) \ +${ECHO_CMD} "http://$$2"; ;; esac; break ;; esac; done < ${DESCR} .else _DESCR=/dev/null +_WWW= .endif describe: - @${ECHO_CMD} -n "${PKGNAME}|${.CURDIR}|${PREFIX}|"; \ - ${ECHO_CMD} -n ${COMMENT:Q}; \ - ${ECHO_CMD} -n "|${_DESCR}|${MAINTAINER}|${CATEGORIES}|${_EXTRACT_DEPENDS}|${_PATCH_DEPENDS}|${_FETCH_DEPENDS}|${_BUILD_DEPENDS:O:u}|${_RUN_DEPENDS:O:u}|"; \ - set "" $$(sed -E -e '/^WWW:[[:blank:]]+/!d' -e 's,^WWW:[[:blank:]]+([[:print:]]+).*$$,\1,' ${_DESCR}); \ - echo $$2 + @${ECHO_CMD} "${PKGNAME}|${.CURDIR}|${PREFIX}|"${COMMENT:Q}"|${_DESCR}|${MAINTAINER}"\ + "|${CATEGORIES}|${_EXTRACT_DEPENDS}|${_PATCH_DEPENDS}|${_FETCH_DEPENDS}"\ + "|${_BUILD_DEPENDS:O:u}|${_RUN_DEPENDS:O:u}|"$$( ${_WWW}) .endif # | (read site && ${ECHO_CMD} $${site}) || ${ECHO_CMD} # sed -E -e '/^WWW:/!d' -e 's,^WWW:.*(http[:print:]*),\1,' ${_DESCR} | (read site && ${ECHO_CMD} $${site}) -- Adios ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Alex Kozlov wrote: Good idea. I also do something like that [1]. But before we can use this method, we have to fix ports with bad pkg-descr: Yeah, in a brief glance at Kris' sed routines I would think at least some of this work would have to be done no matter what (and it should be done anyway), so thank you for your thorough examination of this. :) Since you were kind enough to do the legwork, I'll be glad to commit the fixes if no other committer picks up the ball. Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
On Wed, Jun 11, 2008 at 10:22:18AM -0700, Doug Barton wrote: > Alex Kozlov wrote: > > > Good idea. I also do something like that [1]. But before we can use this > > method, we have to fix ports with bad pkg-descr: > > Yeah, in a brief glance at Kris' sed routines I would think at least > some of this work would have to be done no matter what (and it should be > done anyway), so thank you for your thorough examination of this. :) > > Since you were kind enough to do the legwork, I'll be glad to commit the > fixes if no other committer picks up the ball. Was glad to help. p.s. Played with more sophisticated parser and found some more broken pkg-descr, can You please fix it too? Thanks. WWW: is not the first word in a line: chinese/pine4 devel/ptmalloc emulators/its emulators/klh10 japanese/Wnn6 -- Adios ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Pietro Cerutti wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Kris Kennaway wrote: | ** | * NOTE TO PORT DEVELOPERS | ** | | Variable assignments with != are bad! Try as hard as you can to avoid | using them -- especially in Mk/*! Every time something processes your | makefile it will spawn a command, even if it is not relevant for the | operation being performed. If you need to run shell commands, try to | isolate them within a makefile target. You can avoid code duplication | by assigning the *shell commands* (not their output) to a variable and | inserting it into your code block. | | e.g. instead of | | -- | VARIABLE!=do some shell stuff; do some other stuff | | target: | echo ${VARIABLE} | -- | | do this (or similar): | | -- | VARIABLE_CMDS=do some shell stuff; do some other stuff | | target: | echo $(${VARIABLE_CMDS}) | -- | | This defers the command execution to the point where the target runs, so | in the case when the target is *not* run, then you avoid wasting one or | more process executions. Yes, in theory. Any clue why this doesn't work? SCHED_NAME= sysctl -n kern.sched.name all: ~echo $(${SCHED_NAME}) Try $$(${SCHED_NAME}) (escape the $ you don't want make to process) Kris ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Kris Kennaway wrote: | Pietro Cerutti wrote: |> -BEGIN PGP SIGNED MESSAGE- |> Hash: SHA512 |> |> Kris Kennaway wrote: |> |> | ** |> | * NOTE TO PORT DEVELOPERS |> | ** |> | |> | Variable assignments with != are bad! Try as hard as you can to avoid |> | using them -- especially in Mk/*! Every time something processes your |> | makefile it will spawn a command, even if it is not relevant for the |> | operation being performed. If you need to run shell commands, try to |> | isolate them within a makefile target. You can avoid code duplication |> | by assigning the *shell commands* (not their output) to a variable and |> | inserting it into your code block. |> | |> | e.g. instead of |> | |> | -- |> | VARIABLE!=do some shell stuff; do some other stuff |> | |> | target: |> | echo ${VARIABLE} |> | -- |> | |> | do this (or similar): |> | |> | -- |> | VARIABLE_CMDS=do some shell stuff; do some other stuff |> | |> | target: |> | echo $(${VARIABLE_CMDS}) |> | -- |> | |> | This defers the command execution to the point where the target |> runs, so |> | in the case when the target is *not* run, then you avoid wasting one or |> | more process executions. |> |> Yes, in theory. |> |> Any clue why this doesn't work? |> |> SCHED_NAME= sysctl -n kern.sched.name |> |> all: |> ~echo $(${SCHED_NAME}) | | Try $$(${SCHED_NAME}) (escape the $ you don't want make to process) mh yep... :) tnx! | | Kris - -- Pietro Cerutti [EMAIL PROTECTED] PGP Public Key: http://gahr.ch/pgp -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (FreeBSD) iEYEAREKAAYFAkhaNa4ACgkQwMJqmJVx944OVgCggOX/Pa/LkLltaQM/Orl+NESL SXAAn0tIPuesfM1DNU8Fnf89wHufU3+F =/McK -END PGP SIGNATURE- ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Kris Kennaway wrote: | ** | * NOTE TO PORT DEVELOPERS | ** | | Variable assignments with != are bad! Try as hard as you can to avoid | using them -- especially in Mk/*! Every time something processes your | makefile it will spawn a command, even if it is not relevant for the | operation being performed. If you need to run shell commands, try to | isolate them within a makefile target. You can avoid code duplication | by assigning the *shell commands* (not their output) to a variable and | inserting it into your code block. | | e.g. instead of | | -- | VARIABLE!=do some shell stuff; do some other stuff | | target: | echo ${VARIABLE} | -- | | do this (or similar): | | -- | VARIABLE_CMDS=do some shell stuff; do some other stuff | | target: | echo $(${VARIABLE_CMDS}) | -- | | This defers the command execution to the point where the target runs, so | in the case when the target is *not* run, then you avoid wasting one or | more process executions. Yes, in theory. Any clue why this doesn't work? SCHED_NAME= sysctl -n kern.sched.name all: ~echo $(${SCHED_NAME}) | | Kris | - -- Pietro Cerutti [EMAIL PROTECTED] PGP Public Key: http://gahr.ch/pgp -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (FreeBSD) iEYEAREKAAYFAkhaMxIACgkQwMJqmJVx945CEgCePoNe6vevue/Hfww4DpM0vJrM r/gAn0aAJ+Y68FqLSi7axSdOHuo5EiY4 =6LX2 -END PGP SIGNATURE- ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Doug Barton wrote: Kris Kennaway wrote: The new 'make describe' target runs entirely using shell builtins apart from the need to sed pkg-descr to extract the WWW [2] [2] Actually I am not happy with this but couldn't think of a way to do it better. Having to fork the subshell costs about 60 seconds of system time and 10 of wall time. Here's one way to do it. This is quick and dirty and I haven't benchmarked it, but I imagine it would be faster. while read one two discard; do case "$one" in WWW:)echo one: $one two: $two case "$two" in http://*) echo WWW= $two ;; *) echo WWW= http://$two ;; esac break ;; esac done < pkg-descr I did test this briefly and it pulls out the right values for variables with and without http://. hth, Doug Unfortunately it doesn't DTRT with files terminated with DOS-style CRLF (e.g. devel/p5-Tie-Restore, others). Kris ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: INDEX build optimizations - please review
Kris Kennaway wrote: Unfortunately it doesn't DTRT with files terminated with DOS-style CRLF (e.g. devel/p5-Tie-Restore, others). First, I certainly would not have any problem with a policy that says files in ports shouldn't have CRLF line endings. Second, I am sure we could probably do something about that if we needed to. And Third, I fixed p5-Tie-Restore, along with a whole bunch of others, thanks to Alex Kozlov pointing me in the right direction. :) I really don't care if you use this solution or not, although I would be interested in benchmark results if you chose to give it a try. Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"