FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Given an already built, installed and booted system version, I've noted a big difference for META_MODE in 2 rebuild contexts (no source updates involved): *) Prior buildworld buildkernel, installkernel, installworld, boot presumed before (A) and before (B) below. A) make . . . buildworld buildkernel make . . . buildworld buildkernel (the 2nd buildworld buildkernel in (A) builds far less than the first) (that means that the first built more than I would have guessed) vs. B) make . . . buildworld buildkernel make . . . installworld make . . . buildworld buildkernel (the 2nd buildworld buildkernel in (B) builds far more than it did in (A)) (so, more like the 1st buildworld buildkernel in (A) and (B), given the specified prior context) So I used make -dM for the commented buildworld buildkernel lines, logging the build output and later diff'ing them. Result that I noticed? Lots of lines uniquely from (B)'s case, ending with one of: file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cap_mkdb' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cat' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cp' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchgen' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchide' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/dd' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/egrep' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/env' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/file2c' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gencat' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/grep' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gzip' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/jot' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/lex' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/ln' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/m4' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/mv' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/patch' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sed' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sh' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/touch' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/truncate' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uudecode' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uuencode' is newer than the target... file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/xargs' is newer than the target... The lines with these lead to more files being updated and so causing more indirect rebuild activity (that cascades). Many/most/all(?) of these seem to me to be unlikely to actually need to contribute to what needs to be rebuilt (just based on being newer). So the option to ignore (some of?) them could be useful in making META_MODE builds quicker. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On 2021-Jan-22, at 01:45, Mark Millard wrote: > Given an already built, installed and booted system version, I've > noted a big difference for META_MODE in 2 rebuild contexts (no > source updates involved): > > *) Prior buildworld buildkernel, installkernel, installworld, boot > presumed before (A) and before (B) below. > > A) make . . . buildworld buildkernel > make . . . buildworld buildkernel > (the 2nd buildworld buildkernel in (A) builds far less than the first) > (that means that the first built more than I would have guessed) > > vs. > > B) make . . . buildworld buildkernel > make . . . installworld > make . . . buildworld buildkernel > (the 2nd buildworld buildkernel in (B) builds far more than it did in (A)) > (so, more like the 1st buildworld buildkernel in (A) and (B), given >the specified prior context) > > So I used make -dM for the commented buildworld buildkernel lines, logging > the build output and later diff'ing them. > > Result that I noticed? Lots of lines uniquely from (B)'s case, ending with > one of: > > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cap_mkdb' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cat' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cp' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchgen' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchide' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/dd' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/egrep' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/env' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/file2c' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gencat' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/grep' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gzip' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/jot' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/lex' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/ln' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/m4' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/mv' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/patch' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sed' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sh' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/touch' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/truncate' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uudecode' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uuencode' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/xargs' > is newer than the target... > > The lines with these lead to more files being updated and so causing more > indirect rebuild activity (that cascades). > > Many/most/all(?) of these seem to me to be unlikely to actually need to > contribute to what needs to be rebuilt (just based on being newer). So > the option to ignore (some of?) them could be useful in making META_MODE > builds quicker. The following from one of the .meta files makes the point tha
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On 2021-Jan-23, at 21:37, Mark Millard wrote: > On 2021-Jan-22, at 01:45, Mark Millard wrote: > >> Given an already built, installed and booted system version, I've >> noted a big difference for META_MODE in 2 rebuild contexts (no >> source updates involved): >> >> *) Prior buildworld buildkernel, installkernel, installworld, boot >> presumed before (A) and before (B) below. >> >> A) make . . . buildworld buildkernel >> make . . . buildworld buildkernel >> (the 2nd buildworld buildkernel in (A) builds far less than the first) >> (that means that the first built more than I would have guessed) >> >> vs. >> >> B) make . . . buildworld buildkernel >> make . . . installworld >> make . . . buildworld buildkernel >> (the 2nd buildworld buildkernel in (B) builds far more than it did in (A)) >> (so, more like the 1st buildworld buildkernel in (A) and (B), given >> the specified prior context) >> >> So I used make -dM for the commented buildworld buildkernel lines, logging >> the build output and later diff'ing them. >> >> Result that I noticed? Lots of lines uniquely from (B)'s case, ending with >> one of: >> >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cap_mkdb' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cat' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cp' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchgen' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchide' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/dd' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/egrep' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/env' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/file2c' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gencat' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/grep' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gzip' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/jot' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/lex' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/ln' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/m4' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/mv' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/patch' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sed' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sh' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/touch' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/truncate' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uudecode' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uuencode' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/xargs' >> is newer than the target... >> >> The lines with these lead to more files being updated and so causing more >> indirect rebuild activity (that cascades). >> >> Many/most/all(?) of these seem to me to be unlikely to actually need to >> contribute to what needs to be rebuilt (just based on being newer). So >
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard via freebsd-current wrote: > >> Given an already built, installed and booted system version, I've > >> noted a big difference for META_MODE in 2 rebuild contexts (no > >> source updates involved): > >> > >> *) Prior buildworld buildkernel, installkernel, installworld, boot > >> presumed before (A) and before (B) below. Yes installworld is going to cause problems unless you tell meta mode to ignore everything outside of the src/obj tree. But even that isn't enough as you note below. > >> So I used make -dM for the commented buildworld buildkernel lines, logging > >> the build output and later diff'ing them. > >> > >> Result that I noticed? Lots of lines uniquely from (B)'s case, ending with > >> one of: > >> > >> file > >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk' > >> is newer than the target... Yes. That would be expected. I think Bryan added some knobs to mitigate some of this. grep -n META.*IGNORE share/mk/*mk might be instructive. > >> Many/most/all(?) of these seem to me to be unlikely to actually need to > >> contribute to what needs to be rebuilt (just based on being newer). So > >> the option to ignore (some of?) them could be useful in making META_MODE > >> builds quicker. Yes on all counts. That's why bmake provides a number of knobs to ignore such things. They are listed in the man page in increasing order of expense. One of the risks of the sort of introspection meta mode does, is delving too deep (like the dwarfs ;-) The .MAKE.META.IGNORE_* and .MAKE.META.BAILIWICK variables help to contain the potential insanity. > > The following from one of the .meta files makes the point that rm use > > in the example is unlikely to be important to needing to rebuild, > > despite it actually causing a file rebuild. Nor is the specific echo > > command listed relevant. Only the "ar" command is: > > > > # Meta data file > > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/libc++.a.meta > > CMD @echo building static c++ library > > CMD @rm -f libc++.a > > CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o > > -- filemon acquired metadata -- > > # filemon version 5 > > # Target pid 22471 > > # Start 1611359217.214996 > > V 5 > > E 22961 /bin/sh > > R 22961 /etc/libmap.conf > > R 22961 /var/run/ld-elf.so.hints > > R 22961 /lib/libedit.so.7 > > R 22961 /lib/libc.so.7 > > R 22961 /lib/libncursesw.so.9 > > R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE > > F 22961 22962 > > E 22962 > > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm > > The timestamp on . . ./tmp/legacy/usr/sbin/rm is not > > actually relevant to if libc++.a needs to be rebuilt. True. If there is nothing under .../tmp/legacy that should be counted you can just: .MAKE.META_IGNORE_PATHS += that path which would be the cheapest option. If however there are things that should be considered you'd have to use a much more specific list of .MAKE.META_IGNORE_PATHS or perhaps something like: .MAKE.META.IGNORE_PATTERNS += */rm would ignore an rm binary regardless of where found. worst case you might need something like: # realpath .MAKE.META.IGNORE_FILTER += tA # ignore anything here .MAKE.META.IGNORE_FILTER += N*/tmp/legacy/usr/bin/* Unlike .MAKE.META.IGNORE_PATTERNS which is a list of patterns to match the goal with .MAKE.META.IGNORE_FILTER is to reduced to an empty string paths to be ignored. > > Of course, the structure also point out the judgment > > is specific to understanding the sequence of CMD's > > listed above. Only a hack of ignoring, not recording, > > or commenting out the filemon lines ending in > > /tmp/legacy/usr/sbin/rm would seem to avoid the @rm > > handling issue. Such might well have its own risks. No hacking needed, there are a range of knobs to help. > Just for reference for more about the sequencing involved: > > Looks like in my example various . . ./tmp/legacy/. . ./*bin/ > actually are links to files in: > > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bin/ > > and the later after-install buildworld "Rebuilding the > temporary build tree" step leads to the updated dates for > files in that area, updated via the code that reports: > > Linking host tools into > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bin > > So the prior installworld leaves updated dates and the > linking to those installed files makes > . . ./tmp/legacy/. . ./*bin/* have the newer dates show > up for the legacy paths as well. > > In turn the dependency tracking via META_MODE uses the new > dates on . . ./tmp/legacy/. . ./*bin/* files to cause > rebuilds of more materials than if installworld had not > been done. > > It is not obvious if Bryan D. would find the effort to avoid > this worthwhile for the contexts that drive FreeBSD's build > environment choices. For people that use installworld and also want META_MODE it would seem some addi
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
[This will be a question relative to the old material below.] On Jan 27, 2021, at 17:33, Simon J. Gerraty wrote: > Mark Millard via freebsd-current wrote: Given an already built, installed and booted system version, I've noted a big difference for META_MODE in 2 rebuild contexts (no source updates involved): *) Prior buildworld buildkernel, installkernel, installworld, boot presumed before (A) and before (B) below. > > Yes installworld is going to cause problems unless you tell meta mode to > ignore everything outside of the src/obj tree. > But even that isn't enough as you note below. > So I used make -dM for the commented buildworld buildkernel lines, logging the build output and later diff'ing them. Result that I noticed? Lots of lines uniquely from (B)'s case, ending with one of: file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk' is newer than the target... > > Yes. That would be expected. > I think Bryan added some knobs to mitigate some of this. > > grep -n META.*IGNORE share/mk/*mk > > might be instructive. > Many/most/all(?) of these seem to me to be unlikely to actually need to contribute to what needs to be rebuilt (just based on being newer). So the option to ignore (some of?) them could be useful in making META_MODE builds quicker. > > Yes on all counts. That's why bmake provides a number of knobs to > ignore such things. > They are listed in the man page in increasing order of expense. > > One of the risks of the sort of introspection meta mode does, is delving > too deep (like the dwarfs ;-) > > The .MAKE.META.IGNORE_* and .MAKE.META.BAILIWICK variables help to > contain the potential insanity. > >>> The following from one of the .meta files makes the point that rm use >>> in the example is unlikely to be important to needing to rebuild, >>> despite it actually causing a file rebuild. Nor is the specific echo >>> command listed relevant. Only the "ar" command is: >>> >>> # Meta data file >>> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/libc++.a.meta >>> CMD @echo building static c++ library >>> CMD @rm -f libc++.a >>> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o > >>> -- filemon acquired metadata -- >>> # filemon version 5 >>> # Target pid 22471 >>> # Start 1611359217.214996 >>> V 5 >>> E 22961 /bin/sh >>> R 22961 /etc/libmap.conf >>> R 22961 /var/run/ld-elf.so.hints >>> R 22961 /lib/libedit.so.7 >>> R 22961 /lib/libc.so.7 >>> R 22961 /lib/libncursesw.so.9 >>> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE >>> F 22961 22962 >>> E 22962 >>> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm > >>> The timestamp on . . ./tmp/legacy/usr/sbin/rm is not >>> actually relevant to if libc++.a needs to be rebuilt. > > True. > If there is nothing under .../tmp/legacy that should be counted you can > just: > > .MAKE.META_IGNORE_PATHS += that path > > which would be the cheapest option. > > If however there are things that should be considered you'd have to > use a much more specific list of .MAKE.META_IGNORE_PATHS or > perhaps something like: > > .MAKE.META.IGNORE_PATTERNS += */rm > > would ignore an rm binary regardless of where found. > > worst case you might need something like: > > # realpath > .MAKE.META.IGNORE_FILTER += tA > # ignore anything here > .MAKE.META.IGNORE_FILTER += N*/tmp/legacy/usr/bin/* > > Unlike .MAKE.META.IGNORE_PATTERNS which is a list of patterns to match > the goal with .MAKE.META.IGNORE_FILTER is to reduced to an empty string > paths to be ignored. > >>> Of course, the structure also point out the judgment >>> is specific to understanding the sequence of CMD's >>> listed above. Only a hack of ignoring, not recording, >>> or commenting out the filemon lines ending in >>> /tmp/legacy/usr/sbin/rm would seem to avoid the @rm >>> handling issue. Such might well have its own risks. > > No hacking needed, there are a range of knobs to help. > >> Just for reference for more about the sequencing involved: >> >> Looks like in my example various . . ./tmp/legacy/. . ./*bin/ >> actually are links to files in: >> >> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bin/ >> >> and the later after-install buildworld "Rebuilding the >> temporary build tree" step leads to the updated dates for >> files in that area, updated via the code that reports: >> >> Linking host tools into >> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bin >> >> So the prior installworld leaves updated dates and the >> linking to those installed files makes >> . . ./tmp/legacy/. . ./*bin/* have the newer dates show >> up for the legacy paths as well. >> >> In turn the dependency tracking via META_MODE uses the new >> dates on . . ./tmp/legacy/. . ./*bin/* files to cause >> rebuilds of more materials than if installwo
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
[Added a question about a possible typo in the old original message.]
On Feb 21, 2023, at 13:31, Mark Millard wrote:
> [This will be a question relative to the old material below.]
>
> On Jan 27, 2021, at 17:33, Simon J. Gerraty wrote:
>
>> Mark Millard via freebsd-current wrote:
> Given an already built, installed and booted system version, I've
> noted a big difference for META_MODE in 2 rebuild contexts (no
> source updates involved):
>
> *) Prior buildworld buildkernel, installkernel, installworld, boot
> presumed before (A) and before (B) below.
>>
>> Yes installworld is going to cause problems unless you tell meta mode to
>> ignore everything outside of the src/obj tree.
>> But even that isn't enough as you note below.
>>
> So I used make -dM for the commented buildworld buildkernel lines, logging
> the build output and later diff'ing them.
>
> Result that I noticed? Lots of lines uniquely from (B)'s case, ending with
> one of:
>
> file
> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk'
> is newer than the target...
>>
>> Yes. That would be expected.
>> I think Bryan added some knobs to mitigate some of this.
>>
>> grep -n META.*IGNORE share/mk/*mk
>>
>> might be instructive.
>>
> Many/most/all(?) of these seem to me to be unlikely to actually need to
> contribute to what needs to be rebuilt (just based on being newer). So
> the option to ignore (some of?) them could be useful in making META_MODE
> builds quicker.
>>
>> Yes on all counts. That's why bmake provides a number of knobs to
>> ignore such things.
>> They are listed in the man page in increasing order of expense.
>>
>> One of the risks of the sort of introspection meta mode does, is delving
>> too deep (like the dwarfs ;-)
>>
>> The .MAKE.META.IGNORE_* and .MAKE.META.BAILIWICK variables help to
>> contain the potential insanity.
>>
The following from one of the .meta files makes the point that rm use
in the example is unlikely to be important to needing to rebuild,
despite it actually causing a file rebuild. Nor is the specific echo
command listed relevant. Only the "ar" command is:
# Meta data file
/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/libc++.a.meta
CMD @echo building static c++ library
CMD @rm -f libc++.a
CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o
>>
-- filemon acquired metadata --
# filemon version 5
# Target pid 22471
# Start 1611359217.214996
V 5
E 22961 /bin/sh
R 22961 /etc/libmap.conf
R 22961 /var/run/ld-elf.so.hints
R 22961 /lib/libedit.so.7
R 22961 /lib/libc.so.7
R 22961 /lib/libncursesw.so.9
R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE
F 22961 22962
E 22962
/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm
>>
The timestamp on . . ./tmp/legacy/usr/sbin/rm is not
actually relevant to if libc++.a needs to be rebuilt.
>>
>> True.
>> If there is nothing under .../tmp/legacy that should be counted you can
>> just:
>>
>> .MAKE.META_IGNORE_PATHS += that path
Was that supposed to be ("." vs. "_"):
.MAKE.META.IGNORE_PATHS += that path
The later ones listed have ".".
>> which would be the cheapest option.
>>
>> If however there are things that should be considered you'd have to
>> use a much more specific list of .MAKE.META_IGNORE_PATHS or
(Same question applies to the above.)
>> perhaps something like:
>>
>> .MAKE.META.IGNORE_PATTERNS += */rm
>>
>> would ignore an rm binary regardless of where found.
>>
>> worst case you might need something like:
>>
>> # realpath
>> .MAKE.META.IGNORE_FILTER += tA
>> # ignore anything here
>> .MAKE.META.IGNORE_FILTER += N*/tmp/legacy/usr/bin/*
>>
>> Unlike .MAKE.META.IGNORE_PATTERNS which is a list of patterns to match
>> the goal with .MAKE.META.IGNORE_FILTER is to reduced to an empty string
>> paths to be ignored.
>>
Of course, the structure also point out the judgment
is specific to understanding the sequence of CMD's
listed above. Only a hack of ignoring, not recording,
or commenting out the filemon lines ending in
/tmp/legacy/usr/sbin/rm would seem to avoid the @rm
handling issue. Such might well have its own risks.
>>
>> No hacking needed, there are a range of knobs to help.
>>
>>> Just for reference for more about the sequencing involved:
>>>
>>> Looks like in my example various . . ./tmp/legacy/. . ./*bin/
>>> actually are links to files in:
>>>
>>> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bin/
>>>
>>> and the later after-install buildworld "Rebuilding the
>>> temporary build tree" step leads to the updated dates for
>>> files in that area, updated via the code that reports:
>>>
>>> Linking host tools into
>>> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-s
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
> The timestamp on . . ./tmp/legacy/usr/sbin/rm is not
> actually relevant to if libc++.a needs to be rebuilt.
> >>
> >> True.
> >> If there is nothing under .../tmp/legacy that should be counted you can
> >> just:
> >>
> >> .MAKE.META_IGNORE_PATHS += that path
>
> Was that supposed to be ("." vs. "_"):
>
> .MAKE.META.IGNORE_PATHS += that path
Yes, sorry.
strings `which bmake` | grep META.IGNORE
.MAKE.META.IGNORE_PATHS
.MAKE.META.IGNORE_PATTERNS
${.MAKE.META.IGNORE_PATHS:O:u:tA}
.MAKE.META.IGNORE_FILTER
${.MAKE.META.IGNORE_PATTERNS:@m@${.p.:M$m}@}
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 22, 2023, at 15:26, Simon J. Gerraty wrote:
>> The timestamp on . . ./tmp/legacy/usr/sbin/rm is not
>> actually relevant to if libc++.a needs to be rebuilt.
True.
If there is nothing under .../tmp/legacy that should be counted you can
just:
.MAKE.META_IGNORE_PATHS += that path
>>
>> Was that supposed to be ("." vs. "_"):
>>
>> .MAKE.META.IGNORE_PATHS += that path
>
> Yes, sorry.
Thanks for the information.
> strings `which bmake` | grep META.IGNORE
> .MAKE.META.IGNORE_PATHS
> .MAKE.META.IGNORE_PATTERNS
> ${.MAKE.META.IGNORE_PATHS:O:u:tA}
The -dM output's "is newer than the target" lines
show the path from before the above transformation.
(The :tA results possibly could use another
sort/uniq sequence for the realpath results?)
I've been pondering things because, so far, my
attempts to experiment with this has failed to make
the -dM output lines for the paths go away and it
still does the related build activity. I've been
trying the likes of:
.for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep env
file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch realpath rm
sed sh touch truncate uudecode uuencode
xargs
.MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
.endfor
.for ignore_other_tool in ctfconvert objcopy nm
.MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/usr/bin/${ignore_other_tool}
.endfor
in what I use for make.conf via:
__MAKE_CONF=/usr/home/root/src.configs/make.conf
It is using paths that match the -dM output lines ( sbin
use despite sbin -> ../bin being a symbolic link).
Note: WORLDTMP is not defined that early, thus the ${OBJTOP}/tmp
use.
-V.MAKE.META.IGNORE_PATHS is showing the paths I would
expect, matching the -dM lines.
So I'm still pondering what might be going on.
> .MAKE.META.IGNORE_FILTER
> ${.MAKE.META.IGNORE_PATTERNS:@m@${.p.:M$m}@}
===
Mark Millard
marklmi at yahoo.com
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote: > Since some of the paths reported ended up being links > (symbolic, as I remember), what are the principles for > which form of paths should be the basis for paths in > the likes of: > > .MAKE.META_IGNORE_PATHS .MAKE.META.IGNORE_PATHS > .MAKE.META.IGNORE_PATTERNS > .MAKE.META.IGNORE_FILTER > > Target of link? Path to the link itself, not the > target? Both? These all refer to paths that might appear in a meta file. They are listed in order of expense. .MAKE.META.IGNORE_PATHS is a list of path prefixes , eg if /tmp is in the list then /tmp* will be ignored .MAKE.META.IGNORE_PATTERNS might include */tmp/* which means any path that patches that will be ignored. .MAKE.META.IGNORE_FILTER is the most expensive but also the most flexible. Eg you could have tA:N/something/*, then anything that has a realpath starting with /something/ will be ignored.
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> Thanks for the information.
>
> > strings `which bmake` | grep META.IGNORE
> > .MAKE.META.IGNORE_PATHS
> > .MAKE.META.IGNORE_PATTERNS
> > ${.MAKE.META.IGNORE_PATHS:O:u:tA}
>
> The -dM output's "is newer than the target" lines
> show the path from before the above transformation.
> (The :tA results possibly could use another
> sort/uniq sequence for the realpath results?)
That indicates the above IGNOREs are not working.
> I've been pondering things because, so far, my
> attempts to experiment with this has failed to make
> the -dM output lines for the paths go away and it
> still does the related build activity. I've been
> trying the likes of:
>
> .for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep
> env file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch
> realpath rm sed sh touch truncate uudecode uuencode
> xargs
Is there anything under ${OBJTOP}/tmp that you don't want to ignore?
Otherwise you could simply use
.MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
You might need ${OBJTOP:tA}/tmp/
or both.
> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
> .endfor
> .for ignore_other_tool in ctfconvert objcopy nm
> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/usr/bin/${ignore_other_tool}
> .endfor
>
> in what I use for make.conf via:
>
> __MAKE_CONF=/usr/home/root/src.configs/make.conf
>
> It is using paths that match the -dM output lines ( sbin
> use despite sbin -> ../bin being a symbolic link).
>
> Note: WORLDTMP is not defined that early, thus the ${OBJTOP}/tmp
> use.
>
> -V.MAKE.META.IGNORE_PATHS is showing the paths I would
> expect, matching the -dM lines.
Do you have example?
I really need to add some unit-tests for these...
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 22, 2023, at 17:18, Simon J. Gerraty wrote:
> Mark Millard wrote:
>
>> Thanks for the information.
>>
>>> strings `which bmake` | grep META.IGNORE
>>> .MAKE.META.IGNORE_PATHS
>>> .MAKE.META.IGNORE_PATTERNS
>>> ${.MAKE.META.IGNORE_PATHS:O:u:tA}
>>
>> The -dM output's "is newer than the target" lines
>> show the path from before the above transformation.
>> (The :tA results possibly could use another
>> sort/uniq sequence for the realpath results?)
>
> That indicates the above IGNOREs are not working.
>
>> I've been pondering things because, so far, my
>> attempts to experiment with this has failed to make
>> the -dM output lines for the paths go away and it
>> still does the related build activity. I've been
>> trying the likes of:
>>
>> .for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep
>> env file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch
>> realpath rm sed sh touch truncate uudecode uuencode
>> xargs
>
> Is there anything under ${OBJTOP}/tmp that you don't want to ignore?
More than just _bootstrap_tools_links entries end up in
${WORLDTMP}/legacy/bin/ (so in ${WORLDTMP}/legacy/sbin/
via the symbolic link pointing to ${WORLDTMP}/legacy/bin/ ).
So: yes.
Also, OBJTOP is not constant over all the parts of
buildworld buildkernel . Having the late-substitution
form of notation ${OBJTOP} might not be appropriate
for the content of .MAKE.META.IGNORE_PATHS .
I'm trying to figure out if there is a stable way of
getting a path that would not suffer variability
via late substitution.
> Otherwise you could simply use
>
> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
(Ignoring the variability of OBJTOP issue . . .)
I do not expect that would work: ignoring things
it likely should not.
Also, I'd rather grow a smaller set of ignores
gradually to make it easier to detect if an
addition starts causing a problem and can be
backed out. Starting with everything ignored
would make things much harder to figure out
when ignoring creates a problem.
> You might need ${OBJTOP:tA}/tmp/
> or both.
>
>> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
>> .endfor
>> .for ignore_other_tool in ctfconvert objcopy nm
>> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/usr/bin/${ignore_other_tool}
>> .endfor
>>
>> in what I use for make.conf via:
>>
>> __MAKE_CONF=/usr/home/root/src.configs/make.conf
>>
>> It is using paths that match the -dM output lines ( sbin
>> use despite sbin -> ../bin being a symbolic link).
>>
>> Note: WORLDTMP is not defined that early, thus the ${OBJTOP}/tmp
>> use.
>>
>> -V.MAKE.META.IGNORE_PATHS is showing the paths I would
>> expect, matching the -dM lines.
>
> Do you have example?
> I really need to add some unit-tests for these...
You may want to wait while I see if I can come up with
a better example context to show things. I only just
noticed the late-substitution potential issue and
started looking for a way to avoid it, for example.
(Either a value that does not vary or a form of
causing up-front substitutions in my make.conf .)
===
Mark Millard
marklmi at yahoo.com
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 22, 2023, at 19:47, Mark Millard wrote:
> On Feb 22, 2023, at 17:18, Simon J. Gerraty wrote:
>
>> Mark Millard wrote:
>>
>>> Thanks for the information.
>>>
strings `which bmake` | grep META.IGNORE
.MAKE.META.IGNORE_PATHS
.MAKE.META.IGNORE_PATTERNS
${.MAKE.META.IGNORE_PATHS:O:u:tA}
>>>
>>> The -dM output's "is newer than the target" lines
>>> show the path from before the above transformation.
>>> (The :tA results possibly could use another
>>> sort/uniq sequence for the realpath results?)
>>
>> That indicates the above IGNOREs are not working.
>>
>>> I've been pondering things because, so far, my
>>> attempts to experiment with this has failed to make
>>> the -dM output lines for the paths go away and it
>>> still does the related build activity. I've been
>>> trying the likes of:
>>>
>>> .for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep
>>> env file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch
>>> realpath rm sed sh touch truncate uudecode uuencode
>>> xargs
>>
>> Is there anything under ${OBJTOP}/tmp that you don't want to ignore?
>
> More than just _bootstrap_tools_links entries end up in
> ${WORLDTMP}/legacy/bin/ (so in ${WORLDTMP}/legacy/sbin/
> via the symbolic link pointing to ${WORLDTMP}/legacy/bin/ ).
> So: yes.
>
> Also, OBJTOP is not constant over all the parts of
> buildworld buildkernel . Having the late-substitution
> form of notation ${OBJTOP} might not be appropriate
> for the content of .MAKE.META.IGNORE_PATHS .
>
> I'm trying to figure out if there is a stable way of
> getting a path that would not suffer variability
> via late substitution.
>
>> Otherwise you could simply use
>>
>> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
>
> (Ignoring the variability of OBJTOP issue . . .)
>
> I do not expect that would work: ignoring things
> it likely should not.
>
> Also, I'd rather grow a smaller set of ignores
> gradually to make it easier to detect if an
> addition starts causing a problem and can be
> backed out. Starting with everything ignored
> would make things much harder to figure out
> when ignoring creates a problem.
>
>> You might need ${OBJTOP:tA}/tmp/
>> or both.
>>
>>> .MAKE.META.IGNORE_PATHS+=
>>> ${OBJTOP}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
>>> .endfor
>>> .for ignore_other_tool in ctfconvert objcopy nm
>>> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/usr/bin/${ignore_other_tool}
>>> .endfor
>>>
>>> in what I use for make.conf via:
>>>
>>> __MAKE_CONF=/usr/home/root/src.configs/make.conf
>>>
>>> It is using paths that match the -dM output lines ( sbin
>>> use despite sbin -> ../bin being a symbolic link).
>>>
>>> Note: WORLDTMP is not defined that early, thus the ${OBJTOP}/tmp
>>> use.
>>>
>>> -V.MAKE.META.IGNORE_PATHS is showing the paths I would
>>> expect, matching the -dM lines.
>>
>> Do you have example?
>> I really need to add some unit-tests for these...
>
> You may want to wait while I see if I can come up with
> a better example context to show things. I only just
> noticed the late-substitution potential issue and
> started looking for a way to avoid it, for example.
> (Either a value that does not vary or a form of
> causing up-front substitutions in my make.conf .)
>
I got some useful information about one type of context
that is odd and creates a far mount of "is newer than
the target" notices.
This is an example of something that always "rebuilds"
via a . . ./sbin/realpath "is newer":
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine
It turns out that the .meta file is for a symbolic link:
# ls -Tld
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine
lrwxr-xr-x 1 root wheel 31 Feb 22 20:19:27 2023
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine
-> /usr/main-src/sys/amd64/include
. . ./sys/modules/*/machine being a symbolic link to a directory is normal.
It turns out that . . ./sbin/ln "is newer" examples are
tied to a similar issue:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h.meta:
12: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h
But . . .
# ls -Tld
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h
lrwxr-xr-x 1 root wheel 9 Feb 22 20:18:24 2023
/usr/obj/BUILDs/main-amd64-nodbg-clan
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Simon J. Gerraty wrote:
> > > strings `which bmake` | grep META.IGNORE
> > > .MAKE.META.IGNORE_PATHS
> > > .MAKE.META.IGNORE_PATTERNS
> > > ${.MAKE.META.IGNORE_PATHS:O:u:tA}
> >
> > The -dM output's "is newer than the target" lines
> > show the path from before the above transformation.
> > (The :tA results possibly could use another
> > sort/uniq sequence for the realpath results?)
>
> That indicates the above IGNOREs are not working.
FWIW I just committed some unit-tests for these.
All working as expected.
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> >
> > Is there anything under ${OBJTOP}/tmp that you don't want to ignore?
>
> More than just _bootstrap_tools_links entries end up in
> ${WORLDTMP}/legacy/bin/ (so in ${WORLDTMP}/legacy/sbin/
> via the symbolic link pointing to ${WORLDTMP}/legacy/bin/ ).
> So: yes.
>
> Also, OBJTOP is not constant over all the parts of
> buildworld buildkernel . Having the late-substitution
> form of notation ${OBJTOP} might not be appropriate
> for the content of .MAKE.META.IGNORE_PATHS .
Fwiw .MAKE.META.IGNORE_PATHS is evaluated when meta_init() is
called after all the makefiles have been read - and had a
chance to influence .MAKE.MODE, so I'm not sure how varaiablity of
OBJTOP would matter?
> > .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
>
> (Ignoring the variability of OBJTOP issue . . .)
>
> I do not expect that would work: ignoring things
> it likely should not.
Sure, but it may be useful as an experiment to ensure things are
behaving as expected.
> Also, I'd rather grow a smaller set of ignores
> gradually to make it easier to detect if an
> addition starts causing a problem and can be
> backed out. Starting with everything ignored
> would make things much harder to figure out
> when ignoring creates a problem.
Yes.
>
> > You might need ${OBJTOP:tA}/tmp/
> > or both.
I found it necessary in the unit tests to add :tA to both TMPDIR
and .OBJDIR to get sane result on one test platform.
> >> It is using paths that match the -dM output lines ( sbin
> >> use despite sbin -> ../bin being a symbolic link).
use :tA if you want to ensure consistent results.
> > I really need to add some unit-tests for these...
Done - not yet imported to FreeBSD though
> You may want to wait while I see if I can come up with
> a better example context to show things. I only just
> noticed the late-substitution potential issue and
> started looking for a way to avoid it, for example.
> (Either a value that does not vary or a form of
> causing up-front substitutions in my make.conf .)
Ok
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote: > It appears that for symbolic links being the target, META_MODE does > not respect .MAKE.META.IGNORE_PATHS . The handling of links is rather complicated ;-) Among other things, the src needs to be treated as for a Read and the target as for Write Note that .MAKE.META.IGNORE_PATHS is only applied to absolute paths .MAKE.META.IGNORE_PATTERNS and .MAKE.META.IGNORE_FILTER can be used to apply more generally. --sjg
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 22, 2023, at 22:43, Simon J. Gerraty wrote:
> Mark Millard wrote:
>> It appears that for symbolic links being the target, META_MODE does
>> not respect .MAKE.META.IGNORE_PATHS .
>
> The handling of links is rather complicated ;-)
> Among other things, the src needs to be treated as for a Read and the
> target as for Write
>
> Note that .MAKE.META.IGNORE_PATHS is only applied to absolute paths
> .MAKE.META.IGNORE_PATTERNS and .MAKE.META.IGNORE_FILTER can be used to
> apply more generally.
All paths that I'm adding are absolute paths.
For reference (my additions are after the normal
ones, I use -dV here to show what is based on
other substitutions vs. not):
# ~/sys-build-scripts.amd64-host/make-main-amd64-nodbg-clang.amd64-host.sh -dV
-V.MAKE.META.IGNORE_PATHS buildworld buildkernel
Script started, output file is
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-22:23:06:28
${__MAKE_SHELL} /bin /lib /rescue /sbin /usr/bin /usr/lib /usr/sbin
/usr/share /usr/include /usr/local/etc/libmap.d
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/awk
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/cap_mkdb
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/cat
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/cp
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/crunchgen
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/crunchide
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/dd
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/egrep
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/env
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/file2c
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/gencat
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/grep
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/gzip
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/jot
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/lex
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/lb
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/m4
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/mkcsmapper
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/mktemp
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/mv
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/patch
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/rm
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/sed
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/sh
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/touch
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/truncate
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/uudecode
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/uuencode
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/xargs
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/bin/ctfconvert
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/bin/objcopy
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/bin/nm
Script done, output file is
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-22:23:06:28
The contribution to the list is currently generated via
use of:
.if ${.MAKE.LEVEL} == 0
.for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep env
file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch realpath rm
sed sh touch truncate uudecode uuencode xargs
.MAKE.META.IGNORE_PATHS+=
${MAKEOBJDIR}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
.endfor
.for ignore_other_tool in ctfconvert objcopy nm
.MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIR}/tmp/usr/bin/${ignore_other_tool}
.endfor
.MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
.endif
The := use is how I avoided needing to worry about late binding
substitutions for my
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 22, 2023, at 22:23, Simon J. Gerraty wrote:
> Mark Millard wrote:
>>>
>>> Is there anything under ${OBJTOP}/tmp that you don't want to ignore?
>>
>> More than just _bootstrap_tools_links entries end up in
>> ${WORLDTMP}/legacy/bin/ (so in ${WORLDTMP}/legacy/sbin/
>> via the symbolic link pointing to ${WORLDTMP}/legacy/bin/ ).
>> So: yes.
>>
>> Also, OBJTOP is not constant over all the parts of
>> buildworld buildkernel . Having the late-substitution
>> form of notation ${OBJTOP} might not be appropriate
>> for the content of .MAKE.META.IGNORE_PATHS .
>
> Fwiw .MAKE.META.IGNORE_PATHS is evaluated when meta_init() is
> called after all the makefiles have been read - and had a
> chance to influence .MAKE.MODE, so I'm not sure how varaiablity of
> OBJTOP would matter?
To my knowledge, there is no place to find documentation
about when/how-often the .MAKE.META.IGNORE_PATHS original
text is reevaluated other than the above statement or
source code inspection. (Others have a similar status.)
-dV -V.MAKE.META.IGNORE_PATHS use does list ${__MAKE_SHELL}
but lists /bin/sh without the -dV . (So -V use does not
give a direct clue at what you report.)
I got past the issue using := before reading the above.
(I'm also using MAKEOBJDIR instead of OBJTOP currently.)
>>> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
>>
>> (Ignoring the variability of OBJTOP issue . . .)
>>
>> I do not expect that would work: ignoring things
>> it likely should not.
>
> Sure, but it may be useful as an experiment to ensure things are
> behaving as expected.
As a test:
.if ${.MAKE.LEVEL} == 0
.MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIR:tA}/tmp/
.MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
.endif
leads to:
# ~/sys-build-scripts.amd64-host/make-main-amd64-nodbg-clang.amd64-host.sh -dV
-V.MAKE.META.IGNORE_PATHS buildworld buildkernel
Script started, output file is
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-22:23:29:01
${__MAKE_SHELL} /bin /lib /rescue /sbin /usr/bin /usr/lib /usr/sbin
/usr/share /usr/include /usr/local/etc/libmap.d
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/
Script done, output file is
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-22:23:29:01
Note: I'm currently avoiding -jN for 1> Also, I'd rather grow a smaller set of ignores
>> gradually to make it easier to detect if an
>> addition starts causing a problem and can be
>> backed out. Starting with everything ignored
>> would make things much harder to figure out
>> when ignoring creates a problem.
>
> Yes.
>
>>
>>> You might need ${OBJTOP:tA}/tmp/
>>> or both.
>
> I found it necessary in the unit tests to add :tA to both TMPDIR
> and .OBJDIR to get sane result on one test platform.
>
It is using paths that match the -dM output lines ( sbin
use despite sbin -> ../bin being a symbolic link).
>
> use :tA if you want to ensure consistent results.
So, for each:
.MAKE.META.IGNORE_PATHS+=
${MAKEOBJDIR}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
I need to form an overall :tA on the path? Something
like:
.if ${.MAKE.LEVEL} == 0
.for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep env
file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch realpath rm
sed sh touch truncate uudecode uuencode
xargs
IGNORELEGACY_${ignore_legacy_tool}=
${MAKEOBJDIR}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
.MAKE.META.IGNORE_PATHS+= ${IGNORELEGACY_${ignore_legacy_tool}:tA}
.endfor
.for ignore_other_tool in ctfconvert objcopy nm
IGNOREOTHER_${ignore_other_tool}= ${MAKEOBJDIR}/tmp/usr/bin/${ignore_other_tool}
.MAKE.META.IGNORE_PATHS+= ${IGNOREOTHER_${ignore_other_tool}:tA}
.endfor
.MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
.endif
Such seems to make no difference to the text reported via
-dV -V.MAKE.META.IGNORE_PATHS in my context.
>>> I really need to add some unit-tests for these...
>
> Done - not yet imported to FreeBSD though
>
>> You may want to wait while I see if I can come up with
>> a better example context to show things. I only just
>> noticed the late-substitution potential issue and
>> started looking for a way to avoid it, for example.
>> (Either a value that does not vary or a form of
>> causing up-front substitutions in my make.conf .)
>
> Ok
===
Mark Millard
marklmi at yahoo.com
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> The contribution to the list is currently generated via
> use of:
>
> .if ${.MAKE.LEVEL} == 0
Why is this guarded by .MAKE.LEVEL 0?
.MAKE.META.IGNORE_* should be largely irrelevant at level 0
which in the DIRDEPS_BUILD is reserved for orchestration.
Even in the legacy build, level 0 would be just the top-level makefiles
and anything dealing with meta files would be expected to be level 1 or
greater.
> .for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep
> env file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch
> realpath rm sed sh touch truncate uudecode uuencode xargs
> .MAKE.META.IGNORE_PATHS+=
> ${MAKEOBJDIR}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
> .endfor
> .for ignore_other_tool in ctfconvert objcopy nm
> .MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIR}/tmp/usr/bin/${ignore_other_tool}
> .endfor
> .MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
> .endif
>
> The := use is how I avoided needing to worry about late binding
> substitutions for my additions (independent of the internals of
> make's specific .MAKE.META.IGNORE_PATHS handling).
Depending on value of MAKEOBJDIR the above may not work at all.
The default is
share/mk/src.sys.obj.mk:_default_makeobjdir=
$${.CURDIR:S,^$${SRCTOP},$${OBJTOP},}
In which case the above would not be correct.
> For reference:
>
> # more
> ~/sys-build-scripts.amd64-host/make-main-amd64-nodbg-clang.amd64-host.sh
> kldload -n filemon && \
> cd /usr/main-src/ && \
> mkdir -p /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/ && \
> script
> /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-$(date
> +%Y-%m-%d:%H:%M:%S) \
> env __MAKE_CONF="/usr/home/root/src.configs/make.conf" SRCCONF="/dev/null"
> SRC_ENV_CONF="/usr/home/root/src.configs/src.conf.amd64-nodbg-clang.amd64-host"
> \
> WITH_META_MODE=yes \
> MAKEOBJDIRPREFIX="/usr/obj/BUILDs/main-amd64-nodbg-clang" \
> make $*
>
>
> ===
> Mark Millard
> marklmi at yahoo.com
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> >> Also, OBJTOP is not constant over all the parts of
> >> buildworld buildkernel . Having the late-substitution
> >> form of notation ${OBJTOP} might not be appropriate
> >> for the content of .MAKE.META.IGNORE_PATHS .
> >
> > Fwiw .MAKE.META.IGNORE_PATHS is evaluated when meta_init() is
> > called after all the makefiles have been read - and had a
> > chance to influence .MAKE.MODE, so I'm not sure how varaiablity of
> > OBJTOP would matter?
>
> To my knowledge, there is no place to find documentation
> about when/how-often the .MAKE.META.IGNORE_PATHS original
> text is reevaluated other than the above statement or
> source code inspection. (Others have a similar status.)
True. That should be fixed.
.MAKE.META.IGNORE_PATHS is only evaluated once by make
after all makefiles have been read. It is use to populate a list.
By contrast .MAKE.META.IGNORE_PATTERNS and .MAKE.META.IGNORE_FILTER
are evaluated for every line of filemon output - but again all that
happens after all makefiles have been read.
> -dV -V.MAKE.META.IGNORE_PATHS use does list ${__MAKE_SHELL}
> but lists /bin/sh without the -dV . (So -V use does not
> give a direct clue at what you report.)
One of the local changes to bmake in FreeBSD is that
without -dV -V shows the fully resolved value.
> I got past the issue using := before reading the above.
> (I'm also using MAKEOBJDIR instead of OBJTOP currently.)
Per my last response, I'd be pretty sure MAKEOBJDIR is incorrect.
>
> >>> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
> >>
> >> (Ignoring the variability of OBJTOP issue . . .)
> >>
> >> I do not expect that would work: ignoring things
> >> it likely should not.
> >
> > Sure, but it may be useful as an experiment to ensure things are
> > behaving as expected.
>
> As a test:
>
> .if ${.MAKE.LEVEL} == 0
> .MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIR:tA}/tmp/
> .MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
> .endif
Lose the .if ${.MAKE.LEVEL} == 0
it is almost certainly keeping things from working as expected.
> I still get things like:
>
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/zlib/x86.meta:
> 23: file
> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
> is newer than the target...
> Building
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/zlib/x86
Because that will not be level 0 and so .MAKE.META.IGNORE_PATHS is not
set.
>
> and:
>
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/xl/opt_platform.h.meta:
> 12: file
> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
> is newer than the target...
> Building
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/xl/opt_platform.h
>
> for both of a pair of back-to-back runs of buildworld buildkernel.
>
> FYI: The file system is zfs with mounts that look
> like:
>
> zoptb /zoptb
> zoptb/BUILDs/usr/obj/BUILDs
> . . .
> zoptb/BUILDs/main-amd64-nodbg-clang /usr/obj/BUILDs/main-amd64-nodbg-clang
> . . .
> zoptb/ROOT/main-amd64 /
> . . .
> zoptb/tmp /tmp
> . . .
>
> # bectl list
> BE Active Mountpoint Space Created
> 13S-amd64 - - 4.97G 2021-08-20 16:57
> 13_0R-amd64- - 4.30G 2021-08-20 16:56
> 13_1R-amd64- - 4.12G 2022-03-10 12:38
> main-amd64 NR / 7.42G 2023-02-19 15:37
> old-main-amd64 - - 2.25G 2023-02-09 19:07
>
> (I use zfs in order to use bectl on a couple of
> systems, not for redundancy.)
>
>
> >> Also, I'd rather grow a smaller set of ignores
> >> gradually to make it easier to detect if an
> >> addition starts causing a problem and can be
> >> backed out. Starting with everything ignored
> >> would make things much harder to figure out
> >> when ignoring creates a problem.
> >
> > Yes.
> >
> >>
> >>> You might need ${OBJTOP:tA}/tmp/
> >>> or both.
> >
> > I found it necessary in the unit tests to add :tA to both TMPDIR
> > and .OBJDIR to get sane result on one test platform.
> >
> It is using paths that match the -dM output lines ( sbin
> use despite sbin -> ../bin being a symbolic link).
> >
> > use :tA if you want to ensure consistent results.
>
> So, for each:
>
> .MAKE.META.IGNORE_PATHS+=
> ${MAKEOBJDIR}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
>
> I need to form an overall :tA on the path? Something
> like:
>
> .if ${.MAKE.LEVEL} == 0
I think you need to first get rid of that level 0 check before
worrying about anything else.
> .for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep
> env file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch
> realpat
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 23, 2023, at 12:15, Simon J. Gerraty wrote:
> Mark Millard wrote:
>> The contribution to the list is currently generated via
>> use of:
>>
>> .if ${.MAKE.LEVEL} == 0
>
> Why is this guarded by .MAKE.LEVEL 0?
> .MAKE.META.IGNORE_* should be largely irrelevant at level 0
> which in the DIRDEPS_BUILD is reserved for orchestration.
> Even in the legacy build, level 0 would be just the top-level makefiles
> and anything dealing with meta files would be expected to be level 1 or
> greater.
I never found a stable notation that provides
a reference to whichever of my
/usr/obj/BUILDs/*/usr/main-src/*.*/ the
build is for. (I've been showing just
main-amd64-nodbg-clang amd64.amd64 examples but
there are many more.) MAKEOBJDIR, OBJTOP, etc.
all looked to vary. But here the only place
relevant is that one absolute path, no
alternatives should be used as these other
things move around for where they reference.
MAKEOBJDIRPREFIX is another one that looked to
vary, despite the environment assignment that
I use to get things started.
Note, too the := use. .MAKE.META.IGNORE_PATHS
ends up with no references to MAKEOBJDIR
or the like for my additions, just absolute
paths. (I've also done experiments with :tA
in use in the := assignment.)
-V.MAKE.META.IGNORE_PATHS showed the correct
result with my content. I may have inferred too
much from that for my := use. (May be -V should
not be based on .MAKE.LEVEL zero content as it
can be misleading vs. what will actually be
used?)
So I was hoping for a "assigned once and
inherited" status relative to submakes for
.MAKE.META.IGNORE_PATHS .
>> .for ignore_legacy_tool in awk cap_mkdb cat cp crunchgen crunchide dd egrep
>> env file2c gencat grep gzip jot lex lb ln m4 mkcsmapper mktemp mv patch
>> realpath rm sed sh touch truncate uudecode uuencode xargs
>> .MAKE.META.IGNORE_PATHS+=
>> ${MAKEOBJDIR}/tmp/legacy/usr/sbin/${ignore_legacy_tool}
>> .endfor
>> .for ignore_other_tool in ctfconvert objcopy nm
>> .MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIR}/tmp/usr/bin/${ignore_other_tool}
>> .endfor
>> .MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
>> .endif
>>
>> The := use is how I avoided needing to worry about late binding
>> substitutions for my additions (independent of the internals of
>> make's specific .MAKE.META.IGNORE_PATHS handling).
>
> Depending on value of MAKEOBJDIR the above may not work at all.
> The default is
>
> share/mk/src.sys.obj.mk:_default_makeobjdir=
> $${.CURDIR:S,^$${SRCTOP},$${OBJTOP},}
>
> In which case the above would not be correct.
And I've not found any notation that is always correct
but adjusts to what /usr/obj/BUILDs/*/usr/main-src/*.*/
I happen to be building for. The example that I've
been showing is main-amd64-nodbg-clang with amd64.amd64
but there are other *-*-*dbg-* and *.* that I build for.
(I have found multiple notations that work for
-V.MAKE.META.IGNORE_PATHS .)
I'm wondering if I need to invent a new, personal name
that will not clash with official names and just use
reference to to my name, adjusting my build scripts
to provide the definition.
So: I'm still searching for approriate notation, at least
for the tmp/legacy/usr/ related paths. (The tmp/usr/bin/
experiment is more questionable it is appropriate overall.)
Until I know a valid notational technique, expect to
see experiments involved in what I do. Going the other
way: if I'm to test something for you, let me know the
context you want used instead of whatever my experiment
status happens to be.
>> For reference:
>>
>> # more
>> ~/sys-build-scripts.amd64-host/make-main-amd64-nodbg-clang.amd64-host.sh
>> kldload -n filemon && \
>> cd /usr/main-src/ && \
>> mkdir -p /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/ && \
>> script
>> /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-$(date
>> +%Y-%m-%d:%H:%M:%S) \
>> env __MAKE_CONF="/usr/home/root/src.configs/make.conf" SRCCONF="/dev/null"
>> SRC_ENV_CONF="/usr/home/root/src.configs/src.conf.amd64-nodbg-clang.amd64-host"
>> \
>> WITH_META_MODE=yes \
>> MAKEOBJDIRPREFIX="/usr/obj/BUILDs/main-amd64-nodbg-clang" \
>> make $*
===
Mark Millard
marklmi at yahoo.com
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
[Note for "how many separate bmake instances are in that log?":
I do not know how to tell how many submakes are run total. It
was with -j32 on the threadripper 1950X, if that is was you
were after.]
On Feb 23, 2023, at 12:25, Simon J. Gerraty wrote:
> Mark Millard wrote:
. . .
>
>> I got past the issue using := before reading the above.
>> (I'm also using MAKEOBJDIR instead of OBJTOP currently.)
>
> Per my last response, I'd be pretty sure MAKEOBJDIR is incorrect.
Which still leaves me experimenting to find a correct
reference. Do you know notation will always lead to the
same absolute path with the proper /usr/obj/BUILDs/*/usr/main-src/*.*/
prefix ? (I've been showing just the main-amd64-nodbg-clang
and amd64.amd64 combination but there are many more.)
> .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
(Ignoring the variability of OBJTOP issue . . .)
I do not expect that would work: ignoring things
it likely should not.
>>>
>>> Sure, but it may be useful as an experiment to ensure things are
>>> behaving as expected.
>>
>> As a test:
>>
>> .if ${.MAKE.LEVEL} == 0
>> .MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIR:tA}/tmp/
>> .MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
>> .endif
>
> Lose the .if ${.MAKE.LEVEL} == 0
> it is almost certainly keeping things from working as expected.
What do you want tested instead of MAKEOBJDIR ?
I'm taking a guess (no .MAKE.LEVEL use):
.MAKE.META.IGNORE_PATHS+= ${OBJTOP:tA}/tmp/
.MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS:tA}
>> I still get things like:
>>
>> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/zlib/x86.meta:
>> 23: file
>> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
>> is newer than the target...
>> Building
>> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/zlib/x86
>
> Because that will not be level 0 and so .MAKE.META.IGNORE_PATHS is not
> set.
I tried the above and I still get (picking to look for
tmp/legacy/usr/sbin/ln examples):
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h.meta:
22: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h
--
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/asmc/opt_acpi.h.meta:
22: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/asmc/opt_acpi.h
--
. . .
and (looking for realpath examples):
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/accf_dns/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/accf_dns/machine
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/accf_data/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/accf_http/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/accf_http/machine
--
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/ae/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/ae/machine
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/acl_posix1e/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/acl_posix1e/machine
--
. . .
>>
>> and:
>>
>> /usr/obj/BUILDs/
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> So I was hoping for a "assigned once and
> inherited" status relative to submakes for
> .MAKE.META.IGNORE_PATHS .
No, .MAKE.* are specifically named to ensure they cannot come from the
environment. So bounding them in a check for level 0 pretty much
ensures they have no effect.
> > In which case the above would not be correct.
>
> And I've not found any notation that is always correct
I think your fundamental issue is above.
These variables apply only per make instance.
They are not inherited, they are not global.
As I mentioned previously, there is no variablity of OBJTOP within the
context of a single make instance - at least not once it starts running
targets.
> but adjusts to what /usr/obj/BUILDs/*/usr/main-src/*.*/
> I happen to be building for. The example that I've
> been showing is main-amd64-nodbg-clang with amd64.amd64
> but there are other *-*-*dbg-* and *.* that I build for.
Sorry I guess I'm not following what you are saying as I cannot see how
any of that is relevant.
>
> (I have found multiple notations that work for
> -V.MAKE.META.IGNORE_PATHS .)
Sorry not sure what that means. Note that when you do
make -V BLAH
you are seeing the value of BLAH as level 0 sees it - which in your
setup looks right. But because of your conditional on level 0
the effect is not what you want once the build gets going.
> I'm wondering if I need to invent a new, personal name
> that will not clash with official names and just use
> reference to to my name, adjusting my build scripts
> to provide the definition.
Sorry, a name for what?
>
> So: I'm still searching for approriate notation, at least
> for the tmp/legacy/usr/ related paths. (The tmp/usr/bin/
> experiment is more questionable it is appropriate overall.)
Again not really following. If there are paths under tmp/legacy/usr/
that should be ignored by meta.c - why would that not be so for
everyone?
> Until I know a valid notational technique, expect to
> see experiments involved in what I do. Going the other
> way: if I'm to test something for you, let me know the
> context you want used instead of whatever my experiment
> status happens to be.
>
> >> For reference:
> >>
> >> # more
> >> ~/sys-build-scripts.amd64-host/make-main-amd64-nodbg-clang.amd64-host.sh
> >> kldload -n filemon && \
> >> cd /usr/main-src/ && \
> >> mkdir -p /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/ && \
> >> script
> >> /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-$(date
> >> +%Y-%m-%d:%H:%M:%S) \
> >> env __MAKE_CONF="/usr/home/root/src.configs/make.conf" SRCCONF="/dev/null"
> >> SRC_ENV_CONF="/usr/home/root/src.configs/src.conf.amd64-nodbg-clang.amd64-host"
> >> \
> >> WITH_META_MODE=yes \
> >> MAKEOBJDIRPREFIX="/usr/obj/BUILDs/main-amd64-nodbg-clang" \
> >> make $*
This is why we introduced variables like SRCTOP and OBJTOP behind which you
can hide all the above. It should not matter what or how OBJTOP gets its
value it should be useful.
.MAKE.META.IGNORE_PATHS += ${OBJTOP}/tmp/legacy/usr
should result in nothing under ${OBJTOP}/tmp/legacy/usr causing a target
to be out of date - just because it is newer.
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> [External Email. Be cautious of content]
>
>
> [Note for "how many separate bmake instances are in that log?":
> I do not know how to tell how many submakes are run total. It
> was with -j32 on the threadripper 1950X, if that is was you
> were after.]
You can get a clue from the number of directories being built - each
will have its own instance of bmake.
> > Per my last response, I'd be pretty sure MAKEOBJDIR is incorrect.
>
> Which still leaves me experimenting to find a correct
> reference. Do you know notation will always lead to the
> same absolute path with the proper /usr/obj/BUILDs/*/usr/main-src/*.*/
> prefix ? (I've been showing just the main-amd64-nodbg-clang
> and amd64.amd64 combination but there are many more.)
I would question why you care. What is wrong with ${OBJTOP} ?
Hmm, this does not look promising actually:
grep '^OBJTOP' share/mk/*mk
share/mk/bsd.obj.mk:OBJTOP?= ${MAKEOBJDIR}
share/mk/bsd.obj.mk:OBJTOP?= ${.OBJDIR:S,${.CURDIR},,}${SRCTOP}
share/mk/local.meta.sys.mk:OBJTOP:= ${OBJROOT}${TARGET_OBJ_SPEC}
share/mk/local.meta.sys.mk:OBJTOP := ${HOST_OBJTOP}
share/mk/local.sys.mk:OBJTOP?= ${.OBJDIR:S,${.CURDIR},,}${SRCTOP}
share/mk/src.sys.obj.mk:OBJTOP:=
${OBJROOT}${TARGET}.${TARGET_ARCH}
share/mk/src.sys.obj.mk:OBJTOP:=${OBJROOT:H}
share/mk/src.sys.obj.mk:OBJTOP:=
${OBJROOT}${MACHINE}.${MACHINE_ARCH}
share/mk/src.sys.obj.mk:OBJTOP:=${OBJROOT:H}
share/mk/src.sys.obj.mk:OBJTOP:=${MAKEOBJDIRPREFIX}${SRCTOP}
share/mk/src.sys.obj.mk:OBJTOP= ${SRCTOP}
Without full context, apart from local.meta.sys.mk some of the above
looks highly dubious
In some respects the BSD build provides way too much flexibility.
About 20 years ago at Juniper we found it untenable to continue allowing
users 4 different options for the relationship between their srcs and
objects. So we simply mandated:
SRCTOP= ${SB}/src
OBJROOT= ${SB}/obj/
OBJTOP= ${OBJROOT}/${MACHINE}/
MAKEOBJDIR= ${.CURDIR:S,${SRCTOP},${OBJTOP},}
and life was better, but in a project like FreeBSD you cannot imposed
your will that way ;-)
> > .MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/
>
> (Ignoring the variability of OBJTOP issue . . .)
>
> I do not expect that would work: ignoring things
> it likely should not.
> >>>
> >>> Sure, but it may be useful as an experiment to ensure things are
> >>> behaving as expected.
> >>
> >> As a test:
> >>
> >> .if ${.MAKE.LEVEL} == 0
> >> .MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIR:tA}/tmp/
> >> .MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
> >> .endif
> >
> > Lose the .if ${.MAKE.LEVEL} == 0
> > it is almost certainly keeping things from working as expected.
>
> What do you want tested instead of MAKEOBJDIR ?
I would expect you want ${OBJTOP} (assuming a sane value for OBJTOP ;-)
> I'm taking a guess (no .MAKE.LEVEL use):
Correct.
>
> .MAKE.META.IGNORE_PATHS+= ${OBJTOP:tA}/tmp/
> .MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS:tA}
You do not want to apply :tA to everything.
.MAKE.META.IGNORE_PATHS+= ${OBJTOP}/tmp/ ${OBJTOP:tA}/tmp/
should be valid.
>
>
> >> I still get things like:
> >>
> >> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/zlib/x86.meta:
> >> 23: file
> >> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
> >> is newer than the target...
and is OBJTOP /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64
?
> >> Building
> >> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/zlib/x86
> >
> > Because that will not be level 0 and so .MAKE.META.IGNORE_PATHS is not
> > set.
>
> I tried the above and I still get (picking to look for
> tmp/legacy/usr/sbin/ln examples):
>
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h.meta:
> 22: file
> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
> is newer than the target...
Its a bit messy with kernels since there isn't a src directory
associated with each (in FreeBSD anyway) but you could add the following
to one of the makefiles that will be in play:
.info ${.CURDIR .OBJDIR OBJTOP
.MAKE.META.IGNORE_PATHS:L:@v@${.newline}$v=${$v}@}
to help you see what is going on.
It should then be easy to see whether OBJTOP is covered by
.MAKE.META.IGNORE_PATHS
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 23, 2023, at 14:03, Simon J. Gerraty wrote:
Simplifying context . . .
. . .
> As I mentioned previously, there is no variablity of OBJTOP within the
> context of a single make instance - at least not once it starts running
> targets.
>
>> . . .
>
> .MAKE.META.IGNORE_PATHS += ${OBJTOP}/tmp/legacy/usr
I'll use that definition line for the below.
> should result in nothing under ${OBJTOP}/tmp/legacy/usr causing a target
> to be out of date - just because it is newer.
I'll ignore there that that is skipping too much
and just show what happens for the 2nd buildkernel
of 2 in a row when I use that exact line for both
make runs.
First counts of the "is newer than" lines, counting
separate program names separately:
# cat
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:16:15:18
| grep "is newer than the target" | sed -
e "s@^.*: file '@file '@" | sort | uniq -c | sort -rn | more
2553 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
1001 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
Thousands of rebuilt things based on:
. . ./tmp/legacy/usr/sbin/realpath
. . ./tmp/legacy/usr/sbin/ln
It appears that buildkernel does not use an OBJTOP definition
that references:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64
in my context.
For reference of Build lines paired with a few of those "is newer
than" lines:
First realpath:
# grep -A1 "tmp/legacy/usr/sbin/realpath\>"
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:16:15:18
| more
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine
--
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/x86.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/x86
--
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/i386.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/i386
--
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aacraid/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aacraid/machine
--
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aacraid/x86.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aacraid/x86
--
. . .
Then for ln:
# grep -A1 "tmp/legacy/usr/sbin/ln\>"
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:16:15:18
| more
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h.meta:
12: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_cam.h.meta:
12: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_cam.h
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_aac.h.meta:
12: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> Simplifying context . . .
> . . .
> > As I mentioned previously, there is no variablity of OBJTOP within the
> > context of a single make instance - at least not once it starts running
> > targets.
> >
> >> . . .
> >
> > .MAKE.META.IGNORE_PATHS += ${OBJTOP}/tmp/legacy/usr
>
> I'll use that definition line for the below.
Ok.
> > should result in nothing under ${OBJTOP}/tmp/legacy/usr causing a target
> > to be out of date - just because it is newer.
>
> I'll ignore there that that is skipping too much
> and just show what happens for the 2nd buildkernel
> of 2 in a row when I use that exact line for both
> make runs.
>
> First counts of the "is newer than" lines, counting
> separate program names separately:
>
> # cat
> /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:16:15:18
> | grep "is newer than the target" | sed -
> e "s@^.*: file '@file '@" | sort | uniq -c | sort -rn | more
> 2553 file
> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
> is newer than the target...
> 1001 file
> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
> is newer than the target...
>
It is of course critical to know what OBJDIR is at this point.
also can you show me the line in the meta file that is matching.
If you add the .info line I suggested in kern.mk or kmod.mk
you should get some useful info.
> Thousands of rebuilt things based on:
>
> . . ./tmp/legacy/usr/sbin/realpath
> . . ./tmp/legacy/usr/sbin/ln
>
> It appears that buildkernel does not use an OBJTOP definition
> that references:
It is possible it does not have a "normal" value for it.
Kernel builds start in the objdir so .CURDIR is actually under
OBJTOP, so unless the makefiles have appropriate logic for that case,
and depending on how OBJTOP is derrived, you may have a bogus value.
The fix in that case would be the makefile setting OBJTOP.
>
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64
>
> in my context.
>
> For reference of Build lines paired with a few of those "is newer
> than" lines:
>
>
> I've still no clue of a notation that avoids this for
> my choice to use personal MAKEOBJDIRPREFIX paths:
That should work, looking at share/mk/src.sys.obj.mk
though looks like you might get
OBJROOT?= ${_default_makeobjdirprefix}${SRCTOP}/
which may not be valid for a kernel build.
At least ${.OBJDIR} will not have that ${OBJROOT} as a prefix
and some of the settings for OBJTOP look just wrong to me:
OBJTOP:=${OBJROOT:H}
is the opposite of what the relationship b/w OBJROOT and OBJTOP are in
DIRDEPS_BUILD but it looks like
OBJTOP:=${OBJROOT}${MACHINE}.${MACHINE_ARCH}
is more likely to be used, but given the above default for OBJROOT
that is unlikely to work for a kernel build.
Looks like share/mk/src.sys.obj.mk grok's SB (the setup I/we use) so you
can take control by setting SB and SB_OBJROOT to anything you like and
it should be used for OBJROOT and presumably hence OBJTOP even for a
kernel build. Though I normally use MAKEOBJDIR (similar to the way it
is set in _default_makeobjdir), I don't know how well that works with
legacy targets though - there's a lot of baked in assumptions about
using MAKEOBJDIRPREFIX.
Again that .info line I gave you would provide some useful clues.
--sjg
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
[I'm back at this again after some time on other subjects.]
On Feb 23, 2023, at 17:42, Simon J. Gerraty wrote:
> Mark Millard wrote:
>
>> Simplifying context . . .
>> . . .
>>> As I mentioned previously, there is no variablity of OBJTOP within the
>>> context of a single make instance - at least not once it starts running
>>> targets.
>>>
. . .
>>>
>>> .MAKE.META.IGNORE_PATHS += ${OBJTOP}/tmp/legacy/usr
>>
>> I'll use that definition line for the below.
>
> Ok.
>
>>> should result in nothing under ${OBJTOP}/tmp/legacy/usr causing a target
>>> to be out of date - just because it is newer.
>>
>> I'll ignore there that that is skipping too much
>> and just show what happens for the 2nd buildkernel
>> of 2 in a row when I use that exact line for both
>> make runs.
>>
>> First counts of the "is newer than" lines, counting
>> separate program names separately:
>>
>> # cat
>> /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:16:15:18
>> | grep "is newer than the target" | sed -
>> e "s@^.*: file '@file '@" | sort | uniq -c | sort -rn | more
>> 2553 file
>> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
>> is newer than the target...
>> 1001 file
>> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
>> is newer than the target...
>>
>
> It is of course critical to know what OBJDIR is at this point.
> also can you show me the line in the meta file that is matching.
So, using specific "is newer than" notice examples instead of
the above counts . . .
First some output log lines around a few sbin/realpath "is newer than"
related Building lines, with the .info lines in place now (I've
got both kmod.mk and kern.mk with the .info line, likely producing
redundant output but I did not know up front for sure):
make[4]: "/usr/main-src/sys/conf/kmod.mk" line 72:
.CURDIR=/usr/main-src/sys/modules/aac
.OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
.MAKE.META.IGNORE_PATHS=/bin/sh /bin /lib /rescue /sbin /usr/bin /usr/lib
/usr/sbin /usr/share /usr/include /usr/local/etc/libmap.d
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.am
d64/sys/GENERIC-NODBG/modules/usr/main-src/tmp/legacy/usr
make[4]: "/usr/main-src/sys/conf/kern.mk" line 3:
.CURDIR=/usr/main-src/sys/modules/aac
.OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
.MAKE.META.IGNORE_PATHS=/bin/sh /bin /lib /rescue /sbin /usr/bin /usr/lib
/usr/sbin /usr/share /usr/include /usr/local/etc/libmap.d
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.am
d64/sys/GENERIC-NODBG/modules/usr/main-src/tmp/legacy/usr
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine
machine -> /usr/main-src/sys/amd64/include
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/x86.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/x86
x86 -> /usr/main-src/sys/x86/include
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/i386.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/i386
i386 -> /usr/main-src/sys/i386/include
Some lines just following the above are some sbin/ln "is newer than"
related Building lines:
Skipping meta for beforedepend: .PHONY
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h.meta:
12: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sy
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
>
> First some output log lines around a few sbin/realpath "is newer than"
> related Building lines, with the .info lines in place now (I've
> got both kmod.mk and kern.mk with the .info line, likely producing
> redundant output but I did not know up front for sure):
>
> make[4]: "/usr/main-src/sys/conf/kmod.mk" line 72:
> .CURDIR=/usr/main-src/sys/modules/aac
> .OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
> OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
So as you can see that OBJTOP not does provide a match for where
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath
is
you really want it fixed at
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64
which is difficult given the way it is defined in
src.sys.obj.mk
Perhaps you want to be using
.MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIRPREFIX}/tmp/legacy/usr
or is that ${MAKEOBJDIRPREFIX}/${TARGET}.${TARGET_ARCH}/tmp/legacy/usr
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 23, 2023, at 19:54, Simon J. Gerraty wrote:
>
>>
>> First some output log lines around a few sbin/realpath "is newer than"
>> related Building lines, with the .info lines in place now (I've
>> got both kmod.mk and kern.mk with the .info line, likely producing
>> redundant output but I did not know up front for sure):
>>
>> make[4]: "/usr/main-src/sys/conf/kmod.mk" line 72:
>> .CURDIR=/usr/main-src/sys/modules/aac
>> .OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
>> OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
>
> So as you can see that OBJTOP not does provide a match for where
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath
> is
>
> you really want it fixed at
>
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64
>
> which is difficult given the way it is defined in
> src.sys.obj.mk
Yep, but possibly understated.
> Perhaps you want to be using
>
> .MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIRPREFIX}/tmp/legacy/usr
> or is that ${MAKEOBJDIRPREFIX}/${TARGET}.${TARGET_ARCH}/tmp/legacy/usr
>
I had started with trying to use MAKEOBJDIRPREFIX but it
appeared to end up with an empty expansion in something
I'd looked at, making the addition to
.MAKE.META.IGNORE_PATHS ineffective.
But with the .info lines in place, I should probably
recheck an example with ${MAKEOBJDIRPREFIX} in it.
(Expecting .MAKE.META.IGNORE_PATHS to not work but
showing what happens for the MAKEOBJDIRPREFIX use.)
This turns out to be different for "modules" vs.
"pure kernel". I start with a "modules" example
below.
So for .MAKE.META.IGNORE_PATHS += ${MAKEOBJDIRPREFIX}/tmp/legacy/usr
. . .
make[4]: "/usr/main-src/sys/conf/kmod.mk" line 72:
.CURDIR=/usr/main-src/sys/modules/aac
.OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
.MAKE.META.IGNORE_PATHS=/bin/sh /bin /lib /rescue /sbin /usr/bin /usr/lib
/usr/sbin /usr/share /usr/include /usr/local/etc/libmap.d
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/tmp/legacy/usr
make[4]: "/usr/main-src/sys/conf/kern.mk" line 3:
.CURDIR=/usr/main-src/sys/modules/aac
.OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
.MAKE.META.IGNORE_PATHS=/bin/sh /bin /lib /rescue /sbin /usr/bin /usr/lib
/usr/sbin /usr/share /usr/include /usr/local/etc/libmap.d
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/tmp/legacy/usr
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine.meta:
23: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath'
is newer than the target...
Building
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine
So, it looks like it ended up with MAKEOBJDIRPREFIX being:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules
But looking outside the "modules" part of the kernel build
log ends up showing:
.MAKE.META.IGNORE_PATHS=/bin/sh /bin /lib /rescue /sbin /usr/bin /usr/lib
/usr/sbin /usr/share /usr/include /usr/local/etc/libmap.d /tmp/legacy/usr
So MAKEOBJDIRPREFIX expanded to empty, resulting in:
/tmp/legacy/usr .
So: more can-not-get-there-from-here. I've still no
clue of a notation that will work. (It would need to
apply to buildworld as well, making for more
constraints than just working for buildkernel.)
As for:
${MAKEOBJDIRPREFIX}/${TARGET}.${TARGET_ARCH}/tmp/legacy/usr
my memory is that ${TARGET}.${TARGET_ARCH} are supposed to
only be use in the top level makefiles. But, just seeing
what results . . .
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/amd64.amd64/tmp/legacy/usr
Still not right for "modules".
I'll note that outside modules, it is:
/amd64.amd64/tmp/legacy/usr
and still not right (MAKEOBJDIRPREFIX expanded
to empty).
I still do not know notation to make .MAKE.META.IGNORE_PATHS
effective for the specific list of tmp/legacy/usr/sbin/*
examples in question.
Effectively, it appears that the coverage of
.MAKE.META.IGNORE_PATHS is just incomplete (via the
notational constraints it is used within).
===
Mark Millard
marklmi at yahoo.com
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
Mark Millard wrote:
> > Perhaps you want to be using
> >
> > .MAKE.META.IGNORE_PATHS+= ${MAKEOBJDIRPREFIX}/tmp/legacy/usr
> > or is that ${MAKEOBJDIRPREFIX}/${TARGET}.${TARGET_ARCH}/tmp/legacy/usr
> >
>
> I had started with trying to use MAKEOBJDIRPREFIX but it
> appeared to end up with an empty expansion in something
> I'd looked at, making the addition to
> .MAKE.META.IGNORE_PATHS ineffective.
>
> But with the .info lines in place, I should probably
> recheck an example with ${MAKEOBJDIRPREFIX} in it.
> (Expecting .MAKE.META.IGNORE_PATHS to not work but
> showing what happens for the MAKEOBJDIRPREFIX use.)
> This turns out to be different for "modules" vs.
> "pure kernel". I start with a "modules" example
> below.
Yes, if the value of MAKEOBJDIRPREFIX isn't consistent that's going to
cause problems (I'd call it a bug). If so don't use MAKEOBJDIRPREFIX
directly, set some other variable and export that.
Hmm src.sys.obj.mk plays games with MAKEOBJDIRPREFIX so that's
probably not a good option.
Perhaps:
diff --git a/share/mk/src.sys.obj.mk b/share/mk/src.sys.obj.mk
index 3b48fc3c5514..3c7e570dbdbd 100644
--- a/share/mk/src.sys.obj.mk
+++ b/share/mk/src.sys.obj.mk
@@ -67,6 +67,9 @@ SB_OBJROOT?= ${SB}/obj/
OBJROOT?= ${SB_OBJROOT}
.endif
OBJROOT?= ${_default_makeobjdirprefix}${SRCTOP}/
+# save the value before we mess with it
+_OBJROOT:= ${OBJROOT:tA}
+.export _OBJROOT
.if ${OBJROOT:M*/} != ""
OBJROOT:= ${OBJROOT:H:tA}/
.else
and then something like?
.MAKE.META.IGNORE_PATHS +=
${_OBJROOT}/${MACHINE}.${MACHINE_ARCH}/tmp/legacy/usr
> and still not right (MAKEOBJDIRPREFIX expanded
> to empty).
See above
> I still do not know notation to make .MAKE.META.IGNORE_PATHS
> effective for the specific list of tmp/legacy/usr/sbin/*
> examples in question.
>
> Effectively, it appears that the coverage of
> .MAKE.META.IGNORE_PATHS is just incomplete (via the
> notational constraints it is used within).
No, your problem has nothing to do with .MAKE.META.IGNORE_PATHS
but with the build's lack of a consistent definition of OBJTOP or
OBJROOT or whatever you want to call it.
You could I guess take note of .OBJDIR when setting
.MAKE.META.IGNORE_PATHS and if it match */sys/compile* and tweak
things accordingly so you actually get the value you want.
The messing about with MAKEOBJDIRPREFIX has been around so long I don't
know how you could go about fixing it at this point.
FWIW in our build we make a clear distinction between things build for
the "host" (all the tools you care about are actually host tools not
target tools I think), and those built for a target.
Everything built for host is found under ${HOST_OBJTOP}
and everything for the targets is under ${OBJTOP} which we define
consistently
mk -V OBJTOP -V HOST_OBJTOP:tA
/var/obj/FreeBSD/main/obj/amd64.amd64
/var/obj/FreeBSD/main/obj/freebsd13-amd64
(the :tA is needed there so you can see the relationship - HOST_OBJTOP
comes from environment)
mk -n buildworld -V OBJTOP
Setting legacy build env...
/var/obj/FreeBSD/main/obj/h/sjg/work/FreeBSD/main/src/amd64.amd64
and as you know, that value does not remain consistent thoughout the
tree which makes it of questionable value.
--sjg
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)
On Feb 23, 2023, at 23:33, Simon J. Gerraty wrote:
>
>> . . .
>
> Yes, if the value of MAKEOBJDIRPREFIX isn't consistent that's going to
> cause problems (I'd call it a bug). If so don't use MAKEOBJDIRPREFIX
> directly, set some other variable and export that.
> Hmm src.sys.obj.mk plays games with MAKEOBJDIRPREFIX so that's
> probably not a good option.
> Perhaps:
>
> diff --git a/share/mk/src.sys.obj.mk b/share/mk/src.sys.obj.mk
> index 3b48fc3c5514..3c7e570dbdbd 100644
> --- a/share/mk/src.sys.obj.mk
> +++ b/share/mk/src.sys.obj.mk
> @@ -67,6 +67,9 @@ SB_OBJROOT?= ${SB}/obj/
> OBJROOT?= ${SB_OBJROOT}
> .endif
> OBJROOT?= ${_default_makeobjdirprefix}${SRCTOP}/
> +# save the value before we mess with it
> +_OBJROOT:= ${OBJROOT:tA}
> +.export _OBJROOT
> .if ${OBJROOT:M*/} != ""
> OBJROOT:= ${OBJROOT:H:tA}/
> .else
>
> and then something like?
>
> .MAKE.META.IGNORE_PATHS +=
> ${_OBJROOT}/${MACHINE}.${MACHINE_ARCH}/tmp/legacy/usr
>
>> . . .
>
Thanks.
That has allowed me to set up the disabling of
specific . . ./tmp/legacy/usr/sbin/* programs
from having their dates lead directly to rebuild
activity (tested via -dM use in the make commands).
For reference, I'm using the below block of text
for the .MAKE.META.IGNORE_PATHS adjustments now.
# _OBJROOT is an addition to share/mk/src.sys.obj.mk
# provided by Simon J. Gerraty for my experimentation
# with this avoidance of some unnecessary build
# activity in META MODE:
#
# OBJROOT?= ${_default_makeobjdirprefix}${SRCTOP}/
# +# save the value before we mess with it
# +_OBJROOT:= ${OBJROOT:tA}
# +.export _OBJROOT
#
# TARGET.TARGET_ARCH for amd64 stays as amd64.amd64 for obj-lib32 (correct
for the purpose)
# MACHINE.MACHINE_ARCH for amd64 turns into i386.i386 for obj-lib32 (wrong
for the purpose)
#
IGNORELEGACY_NOSYMLINKPREFIX=
${_OBJROOT}/${TARGET}.${TARGET_ARCH}/tmp/legacy/usr
IGNOREOTHER_NOSYMLINKPREFIX= ${_OBJROOT}/${TARGET}.${TARGET_ARCH}/tmp/usr/bin
#
.for ignore_legacy_tool in awk basename cap_mkdb cat chmod cmp cp crunchgen
crunchide cut date dd dirname echo egrep env expr fgrep file2c find gencat grep
gzip head hostname jot lex lb ln ls m4 make
mkcsmapper mkdir mktemp mtree mv nawk patch realpath rm sed sh sort touch tr
truncate uudecode uuencode wc xargs
.MAKE.META.IGNORE_PATHS+=
${IGNORELEGACY_NOSYMLINKPREFIX}/sbin/${ignore_legacy_tool}
.endfor
#
.for ignore_other_tool in ctfconvert objcopy nm
.MAKE.META.IGNORE_PATHS+= ${IGNOREOTHER_NOSYMLINKPREFIX}/${ignore_other_tool}
.endfor
#
.MAKE.META.IGNORE_PATHS:= ${.MAKE.META.IGNORE_PATHS}
(It is in a file specified via env __MAKE_CONF= use.)
I've extended the experiment to span releng/13.0 ,
releng/13.1 , stable/13 as well, not just main
[so: 14]. And I've set up the aarch64 environment
as well for that range of system variations. (It also
builds for targeting armv7 which is also covered.)
We will see how it goes as I track system updates
periodically.
Thanks again.
For reference, the -dM "is newer than" notice counts
in the log for a buildworld buildkernel just after
a installworld installkernel now looks like:
1467 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib/Scrt1.o'
is newer than the target...
515 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib/crti.o'
is newer than the target...
236 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/crti.o'
is newer than the target...
70 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib/libgcc_s.so'
is newer than the target...
69 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/libgcc_s.so'
is newer than the target...
68 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib/crt1.o'
is newer than the target...
3 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/include/aio.h'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/libssl.so'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/libcxxrt.so'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/libctf.so'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/libcrypto.so'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/libc.so.7'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib32/Scrt1.o'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib/libssl.so'
is newer than the target...
1 file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/usr/lib/libcxxrt.so'
is newer t
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) [code level bug evidence]
cached_realpath only reports its "cached_realpath:" notice
(not the purging one) when it does not find the value via
HashTable_FindValue and so does a HashTable_Set :
const char *
cached_realpath(const char *pathname, char *resolved)
{
const char *rp;
if (pathname == NULL || pathname[0] == '\0')
return NULL;
rp = HashTable_FindValue(&cached_realpaths, pathname);
if (rp != NULL) {
/* a hit */
strncpy(resolved, rp, MAXPATHLEN);
resolved[MAXPATHLEN - 1] = '\0';
return resolved;
}
rp = realpath(pathname, resolved);
if (rp != NULL) {
HashTable_Set(&cached_realpaths, pathname, bmake_strdup(rp));
DEBUG2(DIR, "cached_realpath: %s -> %s\n", pathname, rp);
return resolved;
}
/* should we negative-cache? */
return NULL;
}
cached_realpaths is global:
static HashTable cached_realpaths;
So with -ddM why do I see lots of "cached_realpath:"
notices for the same path? For example:
# grep "tmp/legacy/usr/sbin/ln\>"
/usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:10:20:26
| more
cached_realpath:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
->
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
cached_realpath:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
->
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
Caching 02:49:37 Feb 23, 2023 for
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h.meta:
22: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
cached_realpath:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
->
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
Caching 02:49:37 Feb 23, 2023 for
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
cached_realpath:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
->
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
Caching 02:49:37 Feb 23, 2023 for
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
cached_realpath:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
->
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
Caching 02:49:37 Feb 23, 2023 for
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
cached_realpath:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
->
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
Caching 02:49:37 Feb 23, 2023 for
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
cached_realpath:
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
->
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
Caching 02:49:37 Feb 23, 2023 for
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h.meta:
22: file
'/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
is newer than the target...
. . .
A possible cause is something I ran into while looking around:
/* A read-only range of a character array, NOT null-terminated. */
typedef struct Substring {
const char *start;
const char *end;
} Substring;
. . .
MAKE_STATIC Substring
Substring_Init(const char *start, const char *end)
{
Substring sub;
sub.start = start;
sub.end = end;
return sub;
}
. . .
/* Find the entry corresponding to the key, or return NULL. */
HashEntry *
HashTable_FindEntry(HashTable *t, const char *key)
{
const char *keyEnd;
unsigned int h = Hash_String(key, &keyEnd);
return HashTable_Find(t, Substring_Init(key, keyEnd), h);
}
. . .
/* A read-only range of a character array, NOT null-terminated. */
typedef struct Substring {
const char *start;
const char *end;
} Substring;
. . .
MAKE_STATIC Substring
Substring_Init(const char *start, const char *end)
{
Substring sub;
sub.start = start;
sub.end = end;
return sub;
}
. . .
/* Find the entry corresponding to the key, or return NULL. */
HashEntry *
HashTable_FindEntry(HashTable *t, const char *key)
{
const char *keyEnd;
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) [code level bug evidence]
On Feb 23, 2023, at 11:53, Mark Millard wrote:
> cached_realpath only reports its "cached_realpath:" notice
> (not the purging one) when it does not find the value via
> HashTable_FindValue and so does a HashTable_Set :
>
> const char *
> cached_realpath(const char *pathname, char *resolved)
> {
>const char *rp;
>
>if (pathname == NULL || pathname[0] == '\0')
>return NULL;
>
>rp = HashTable_FindValue(&cached_realpaths, pathname);
>if (rp != NULL) {
>/* a hit */
>strncpy(resolved, rp, MAXPATHLEN);
>resolved[MAXPATHLEN - 1] = '\0';
>return resolved;
>}
>
>rp = realpath(pathname, resolved);
>if (rp != NULL) {
>HashTable_Set(&cached_realpaths, pathname, bmake_strdup(rp));
>DEBUG2(DIR, "cached_realpath: %s -> %s\n", pathname, rp);
>return resolved;
>}
>
>/* should we negative-cache? */
>return NULL;
> }
>
> cached_realpaths is global:
>
> static HashTable cached_realpaths;
>
> So with -ddM why do I see lots of "cached_realpath:"
> notices for the same path? For example:
>
> # grep "tmp/legacy/usr/sbin/ln\>"
> /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:10:20:26
> | more
> cached_realpath:
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> ->
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
> cached_realpath:
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> ->
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
> Caching 02:49:37 Feb 23, 2023 for
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h.meta:
> 22: file
> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
> is newer than the target...
> cached_realpath:
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> ->
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
> Caching 02:49:37 Feb 23, 2023 for
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> cached_realpath:
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> ->
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
> Caching 02:49:37 Feb 23, 2023 for
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> cached_realpath:
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> ->
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
> Caching 02:49:37 Feb 23, 2023 for
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> cached_realpath:
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> ->
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
> Caching 02:49:37 Feb 23, 2023 for
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> cached_realpath:
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> ->
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln
> Caching 02:49:37 Feb 23, 2023 for
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h.meta:
> 22: file
> '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln'
> is newer than the target...
> . . .
>
> A possible cause is something I ran into while looking around:
>
> /* A read-only range of a character array, NOT null-terminated. */
> typedef struct Substring {
>const char *start;
>const char *end;
> } Substring;
> . . .
> MAKE_STATIC Substring
> Substring_Init(const char *start, const char *end)
> {
>Substring sub;
>
>sub.start = start;
>sub.end = end;
>return sub;
> }
> . . .
> /* Find the entry corresponding to the key, or return NULL. */
> HashEntry *
> HashTable_FindEntry(HashTable *t, const char *key)
> {
>const char *keyEnd;
>unsigned int h = Hash_String(key, &keyEnd);
>return HashTable_Find(t, Substring_Init(key, keyEnd), h);
> }
> . . .
> /* A read-only range of a character array, NOT null-terminated. */
> typedef struct Substring {
>const char *start;
>const char *end;
> } Substring;
> . . .
> MAKE_STATIC Substring
> Substring_Init(const char *start, const char *end)
Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) [code level bug evidence]
Mark Millard wrote: > So with -ddM why do I see lots of "cached_realpath:" > notices for the same path? For example: how many separate bmake instances are in that log? > > # grep "tmp/legacy/usr/sbin/ln\>" > /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:10:20:26 > | more > cached_realpath: > /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln > -> > /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln > cached_realpath: > /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln > -> > /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/bin/ln >Caching 02:49:37 Feb 23, 2023 for > /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln > /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awkgram.tab.h.meta: > 22: file > '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln' > is newer than the target...
