Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
On 9/3/2015 11:46 AM, Simon J. Gerraty wrote: > Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an > initial set of tools. It then attempts to use that to build toolchain > for MACHINE=host which is currently failing because in-tree libelf and > libdwarf are needed and libelf needs sys/sys/elf_common.h but but that > doesn't currently get staged for host (we try to minimize what we put in > host stage tree). Using the special hack you came up with for lib/libelf, and a lot of other fixes for replacing binutils with elftoolchain, I've managed to get the targets/pseduo/bootstrap-tools build working again. I will commit this likely today or tomorrow. However... > > It may be better to simply skip targets/pseudo/toolchain.host > that will work so long as we set TOOLSDIR to point to the stuff built by > Makefile.inc1 > > Depending on where we are with external toolchian support that would > make sense - might be able to skip bootstrap-toolchain completely > which would be nice. I think this is more right because there's many more people thinking about, testing daily, and maintaining the bootstrap logic in Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently lead to bitrot and breakage with the elftoolchain replacement. It will very easily happen again. Right now the targets/pseduo/bootstrap-tools target builds legacy,build-tools,cross-tools from Makefile.inc1 for everything but the compiler and toolchain, and then building the toolchain itself. There's a subtle problem with this in that we end up building clang in targets/pseudo/toolchain with -nostdinc (from local.init.mk) which leads to a missing header (which actually is staged, just lacks a -I or --sysroot to be captured). But really there's no reason to do this work since Makefile.inc1 does it fine and is maintained well to handle it. There's another subtle thing here, because we set CC=HOST_CC=/usr/bin/cc when calling cross-tools from the pseudo/targets/bootstrap-tools build, it invokes the external compiler support in Makefile.inc1 which avoids building the bootstrap clang and some of the toolchain, leading us to have to do it from pseudo/targets/toolchain. I actually extended this logic to pass HOST_AS to avoid *more* redundant toolchain building in Makefile.inc1 from our own targets/pseduo/toolchain, before realizing this. What I plan to do is make pseudo/targets/bootstrap-tools always build all of cross-tools,etc, from Makefile.inc1 respecting its own internal logic for when to bootstrap the compiler (without us telling it to skip the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a global DIRDEP. Given the use of cookies, this will only be done once per meta build, but it at least won't be a manual step anymore. For the record, I do plan to extend .MAKE.MODE=meta to buildworld and its bootstrapping as well, which will give a lot of incremental build improvements to it. This means that the meta build will then default to bootstrapping the compiler, which we really must do to build the source tree reliably. There's really no reason the meta build should default to not bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler as the meta build does now. So there's no real problem for those people who want to skip the clang bootstrap. Then, I will work on the project of "building clang less" which will avoid building the bootstrap compiler for both the meta mode build and the buildworld build (and universe eventually) if /usr/bin/cc satisfies the needs (can cross compile the TARGET and is "new enough" compared to the version we want). This is not trivial but I think it is possible and it will make everyone happy! Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP) with buildworld leads to a broken build currently since it is the user basically asking to use their default external toolchain of /usr/bin/*, but the logic does not kick in to use --sysroot against WORLDTMP. I plan to fix this soon as well. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
Bryan Drewery wrote: > On 9/3/2015 11:46 AM, Simon J. Gerraty wrote: > > Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an > > initial set of tools. It then attempts to use that to build toolchain > > for MACHINE=host which is currently failing because in-tree libelf and > > libdwarf are needed and libelf needs sys/sys/elf_common.h but but that > > doesn't currently get staged for host (we try to minimize what we put in > > host stage tree). > > Using the special hack you came up with for lib/libelf, and a lot of > other fixes for replacing binutils with elftoolchain, I've managed to > get the targets/pseduo/bootstrap-tools build working again. I will > commit this likely today or tomorrow. Cool - thanks > However... > > It may be better to simply skip targets/pseudo/toolchain.host > > that will work so long as we set TOOLSDIR to point to the stuff built by > > Makefile.inc1 > > > > Depending on where we are with external toolchian support that would > > make sense - might be able to skip bootstrap-toolchain completely > > which would be nice. > > I think this is more right because there's many more people thinking > about, testing daily, and maintaining the bootstrap logic in > Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently > lead to bitrot and breakage with the elftoolchain replacement. It will > very easily happen again. I'd be favor of removing the need completely. All that is required is *some* means of producing a viable toolchain in such a way that you can point a variable at it. > What I plan to do is make pseudo/targets/bootstrap-tools always build > all of cross-tools,etc, from Makefile.inc1 respecting its own internal > logic for when to bootstrap the compiler (without us telling it to skip > the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a > global DIRDEP. Given the use of cookies, this will only be done once per > meta build, but it at least won't be a manual step anymore. For the That can still add a lot of time to build, and IIRC building clang alone poses problems for smaller machines (though using a mutex for linking clang bits could at least serialize the huge linker steps so as to avoid exhausting memory). I quite like the way NetBSD allow separating the tools from the rest of the build. I do the equivalent of 'make tools' very rarely - and so do not care how [in]efficient it is. IIRC it will throw an error if your tools are incompatible - prompting an update. I would suggest if you want to be able to hook bootstrap-tools in always, that you do it via a knob so it can be disabled. > record, I do plan to extend .MAKE.MODE=meta to buildworld and its > bootstrapping as well, which will give a lot of incremental build > improvements to it. WITH_META_FILES should give you improvements already in that regard. > This means that the meta build will then default to bootstrapping the > compiler, which we really must do to build the source tree reliably. > There's really no reason the meta build should default to not > bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or > WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler > as the meta build does now. So there's no real problem for those people > who want to skip the clang bootstrap. Ok, so long as there is a way to do so. Otherwise we'd need to add it locally for our own builds - where we need to use the toolchains provided by our compiler team. > Then, I will work on the project of "building clang less" which will > avoid building the bootstrap compiler for both the meta mode build and > the buildworld build (and universe eventually) if /usr/bin/cc satisfies > the needs (can cross compile the TARGET and is "new enough" compared to > the version we want). This is not trivial but I think it is possible and > it will make everyone happy! Indeed. As I say, NetBSD have this reasonably sorted. But of course they have 2k line shell script driving a lot of it ;-) > Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP) > with buildworld leads to a broken build currently since it is the user > basically asking to use their default external toolchain of /usr/bin/*, > but the logic does not kick in to use --sysroot against WORLDTMP. I plan > to fix this soon as well. Assuming that you have previously built the correct toolchain it should be valid to have something like -DWITH_TOOLSDIR -DWITHOUT_BOOTSTRAP_TOOLSDIR (or -DWITHOUT_TOOLSDIR_BOOTSTRAP) Such that the build logic is identical - all that is being skipped is the expense of re-generating the toolchain ___ freebsd-toolchain@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"
Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
On 11/12/2015 3:42 PM, Simon J. Gerraty wrote:
> Bryan Drewery wrote:
>
>> On 9/3/2015 11:46 AM, Simon J. Gerraty wrote:
>>> Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an
>>> initial set of tools. It then attempts to use that to build toolchain
>>> for MACHINE=host which is currently failing because in-tree libelf and
>>> libdwarf are needed and libelf needs sys/sys/elf_common.h but but that
>>> doesn't currently get staged for host (we try to minimize what we put in
>>> host stage tree).
>>
>> Using the special hack you came up with for lib/libelf, and a lot of
>> other fixes for replacing binutils with elftoolchain, I've managed to
>> get the targets/pseduo/bootstrap-tools build working again. I will
>> commit this likely today or tomorrow.
>
> Cool - thanks
>
>> However...
>
>>> It may be better to simply skip targets/pseudo/toolchain.host
>>> that will work so long as we set TOOLSDIR to point to the stuff built by
>>> Makefile.inc1
>>>
>>> Depending on where we are with external toolchian support that would
>>> make sense - might be able to skip bootstrap-toolchain completely
>>> which would be nice.
>>
>> I think this is more right because there's many more people thinking
>> about, testing daily, and maintaining the bootstrap logic in
>> Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently
>> lead to bitrot and breakage with the elftoolchain replacement. It will
>> very easily happen again.
>
> I'd be favor of removing the need completely.
>
> All that is required is *some* means of producing a viable toolchain in
> such a way that you can point a variable at it.
>
>> What I plan to do is make pseudo/targets/bootstrap-tools always build
>> all of cross-tools,etc, from Makefile.inc1 respecting its own internal
>> logic for when to bootstrap the compiler (without us telling it to skip
>> the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a
>> global DIRDEP. Given the use of cookies, this will only be done once per
>> meta build, but it at least won't be a manual step anymore. For the
>
> That can still add a lot of time to build, and IIRC building clang alone
> poses problems for smaller machines (though using a mutex for linking
> clang bits could at least serialize the huge linker steps so as to avoid
> exhausting memory).
>
> I quite like the way NetBSD allow separating the tools from the rest of
> the build. I do the equivalent of 'make tools' very rarely - and so
> do not care how [in]efficient it is.
> IIRC it will throw an error if your tools are incompatible - prompting
> an update.
>
> I would suggest if you want to be able to hook bootstrap-tools in
> always, that you do it via a knob so it can be disabled.
I would just add some knob like WITH_BOOTSTRAP_TOOLCHAIN or
WITH_EXTERNAL_TOOLCHAIN to make this and the clang bootstrap just be
skipped and resort to the way the meta build works now.
>
>> record, I do plan to extend .MAKE.MODE=meta to buildworld and its
>> bootstrapping as well, which will give a lot of incremental build
>> improvements to it.
>
> WITH_META_FILES should give you improvements already in that regard.
Yes, it's a step. We'll need cookies in a lot of places too. I wish
WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could
check for "META_MODE" in the buildworld world and for discussion sake.
It seems I can use ${.MAKE.MODE:M*meta*} but that :U is needed in all
the uses. I'm not sure yet if :U really is needed. We have some
${MK_META_MODE} checks now around some cookies that would need to change
for what I'm planning.
>
>> This means that the meta build will then default to bootstrapping the
>> compiler, which we really must do to build the source tree reliably.
>
>> There's really no reason the meta build should default to not
>> bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or
>> WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler
>> as the meta build does now. So there's no real problem for those people
>> who want to skip the clang bootstrap.
>
> Ok, so long as there is a way to do so.
> Otherwise we'd need to add it locally for our own builds - where we need
> to use the toolchains provided by our compiler team.
Yes for sure. As noted above. At Isilon when a developer is building
they have no interest in building the toolchain 99% of the time. This is
why I am so committed to making this automatically skip the toolchain if
possible. As for having an external compiler from a team, that's kind of
already in the "external compiler" realm, so a WITH_EXTERNAL_COMPILER
would make sense to short-circuit all of what I'm wanting. Then you can
add it to your local make.conf or local.sys.mk and not have any change
in behavior.
>
>> Then, I will work on the project of "building clang less" which will
>> avoid building the bootstrap compiler for both the meta mode build and
>> the buildworld build (and universe eventually) if /usr/bin/cc satisfies
>
Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
Bryan Drewery wrote:
> > WITH_META_FILES should give you improvements already in that regard.
>
> Yes, it's a step. We'll need cookies in a lot of places too. I wish
> WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could
WITH_DIRDEPS_BUILD would be more accurate.
Its not too late to rename/add it.
> check for "META_MODE" in the buildworld world and for discussion sake.
> It seems I can use ${.MAKE.MODE:M*meta*} but that :U is needed in all
> the uses. I'm not sure yet if :U really is needed. We have some
> ${MK_META_MODE} checks now around some cookies that would need to change
> for what I'm planning.
I think I mentioned the otherday of having something like:
.if ${MK_META_MODE} == "yes"
# we can safely use cookies to avoid always
# re-running targets.
META_COOKIE_TOUCH= touch ${COOKIE.${.TARGET}:U${.TARGET}}
.else
META_COOKIE_TOUCH=
.endif
in meta.sys.mk so you could just add ${META_COOKIE_TOUCH}
to the end of suitable targets.
> > Indeed. As I say, NetBSD have this reasonably sorted.
> > But of course they have 2k line shell script driving a lot of it ;-)
>
> Yes the NetBSD build, behavior wise, really impresses me.
I prefer building it in meta mode though ;-)
___
freebsd-toolchain@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"
Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
On 11/13/2015 9:22 AM, Simon J. Gerraty wrote:
> Bryan Drewery wrote:
>>> WITH_META_FILES should give you improvements already in that regard.
>>
>> Yes, it's a step. We'll need cookies in a lot of places too. I wish
>> WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could
>
> WITH_DIRDEPS_BUILD would be more accurate.
>
> Its not too late to rename/add it.
My hesitation was really only due to all of the documentation and
presentations you've done with it named "WITH_META_MODE". If you don't
mind renaming I would like to, if nothing else because it will help
remove some confusion on what "meta mode" is when discussing it (since
it's both a bmake feature and a new build).
>
>> check for "META_MODE" in the buildworld world and for discussion sake.
>> It seems I can use ${.MAKE.MODE:M*meta*} but that :U is needed in all
>> the uses. I'm not sure yet if :U really is needed. We have some
>> ${MK_META_MODE} checks now around some cookies that would need to change
>> for what I'm planning.
>
> I think I mentioned the otherday of having something like:
>
> .if ${MK_META_MODE} == "yes"
> # we can safely use cookies to avoid always
> # re-running targets.
> META_COOKIE_TOUCH= touch ${COOKIE.${.TARGET}:U${.TARGET}}
> .else
> META_COOKIE_TOUCH=
> .endif
>
> in meta.sys.mk so you could just add ${META_COOKIE_TOUCH}
> to the end of suitable targets.
Great, thanks. Definitely will be useful.
>
>>> Indeed. As I say, NetBSD have this reasonably sorted.
>>> But of course they have 2k line shell script driving a lot of it ;-)
>>
>> Yes the NetBSD build, behavior wise, really impresses me.
>
> I prefer building it in meta mode though ;-)
>
Yes. I do understand the want of a 'make tools' step as well. I just am
latching onto the "go into bin/cat, type make, and watch the magic" goal
without needing a "oh and btw you sometimes need to build a toolchain if
X, Y or Z". I won't push a forced toolchain into meta mode until I have
something more smart worked up.
--
Regards,
Bryan Drewery
signature.asc
Description: OpenPGP digital signature
Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
Bryan Drewery wrote:
> On 11/13/2015 9:22 AM, Simon J. Gerraty wrote:
> > Bryan Drewery wrote:
> >>> WITH_META_FILES should give you improvements already in that regard.
> >>
> >> Yes, it's a step. We'll need cookies in a lot of places too. I wish
> >> WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could
> >
> > WITH_DIRDEPS_BUILD would be more accurate.
> >
> > Its not too late to rename/add it.
>
> My hesitation was really only due to all of the documentation and
The diff is pretty small actually - only 8 files affected.
I suspect renaming tools/build/options/WITH_META_{FILES,MODE} might need
to be done in a couple of steps?
At least svn diff cannot correctly represent
svn mv WITH_META_MODE WITH_DIRDEPS_BUILD
svn mv WITH_META_FIES WITH_META_MODE
> > I think I mentioned the otherday of having something like:
> >
> > .if ${MK_META_MODE} == "yes"
> > # we can safely use cookies to avoid always
> > # re-running targets.
> > META_COOKIE_TOUCH= touch ${COOKIE.${.TARGET}:U${.TARGET}}
> > .else
> > META_COOKIE_TOUCH=
> > .endif
> >
> > in meta.sys.mk so you could just add ${META_COOKIE_TOUCH}
> > to the end of suitable targets.
>
> Great, thanks. Definitely will be useful.
Better name welcome
Also this probably wants to go somewhere other than meta.sys.mk
not needed during sys.mk anyway.
___
freebsd-toolchain@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"
Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
On 11/13/2015 1:38 PM, Simon J. Gerraty wrote:
> Bryan Drewery wrote:
>
>> On 11/13/2015 9:22 AM, Simon J. Gerraty wrote:
>>> Bryan Drewery wrote:
> WITH_META_FILES should give you improvements already in that regard.
Yes, it's a step. We'll need cookies in a lot of places too. I wish
WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could
>>>
>>> WITH_DIRDEPS_BUILD would be more accurate.
>>>
>>> Its not too late to rename/add it.
>>
>> My hesitation was really only due to all of the documentation and
>
> The diff is pretty small actually - only 8 files affected.
> I suspect renaming tools/build/options/WITH_META_{FILES,MODE} might need
> to be done in a couple of steps?
>
> At least svn diff cannot correctly represent
>
> svn mv WITH_META_MODE WITH_DIRDEPS_BUILD
> svn mv WITH_META_FIES WITH_META_MODE
1 commit here is fine. After the commit you would run something like
this from tools/build/options to update src.conf.5:
svn up . ../../../share/man/man5/src.conf.5
./makeman > ../../../share/man/man5/src.conf.5
svn commit -m "Regen" ../../../share/man/man5/src.conf.5
>
>>> I think I mentioned the otherday of having something like:
>>>
>>> .if ${MK_META_MODE} == "yes"
>>> # we can safely use cookies to avoid always
>>> # re-running targets.
>>> META_COOKIE_TOUCH= touch ${COOKIE.${.TARGET}:U${.TARGET}}
>>> .else
>>> META_COOKIE_TOUCH=
>>> .endif
>>>
>>> in meta.sys.mk so you could just add ${META_COOKIE_TOUCH}
>>> to the end of suitable targets.
>>
>> Great, thanks. Definitely will be useful.
>
> Better name welcome
> Also this probably wants to go somewhere other than meta.sys.mk
> not needed during sys.mk anyway.
>
--
Regards,
Bryan Drewery
signature.asc
Description: OpenPGP digital signature
