On 11/12/2015 3:42 PM, Simon J. Gerraty wrote: > Bryan Drewery <bdrew...@freebsd.org> wrote: > >> On 9/3/2015 11:46 AM, Simon J. Gerraty wrote: >>> Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an >>> initial set of tools. It then attempts to use that to build toolchain >>> for MACHINE=host which is currently failing because in-tree libelf and >>> libdwarf are needed and libelf needs sys/sys/elf_common.h but but that >>> doesn't currently get staged for host (we try to minimize what we put in >>> host stage tree). >> >> Using the special hack you came up with for lib/libelf, and a lot of >> other fixes for replacing binutils with elftoolchain, I've managed to >> get the targets/pseduo/bootstrap-tools build working again. I will >> commit this likely today or tomorrow. > > Cool - thanks > >> However... > >>> It may be better to simply skip targets/pseudo/toolchain.host >>> that will work so long as we set TOOLSDIR to point to the stuff built by >>> Makefile.inc1 >>> >>> Depending on where we are with external toolchian support that would >>> make sense - might be able to skip bootstrap-toolchain completely >>> which would be nice. >> >> I think this is more right because there's many more people thinking >> about, testing daily, and maintaining the bootstrap logic in >> Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently >> lead to bitrot and breakage with the elftoolchain replacement. It will >> very easily happen again. > > I'd be favor of removing the need completely. > > All that is required is *some* means of producing a viable toolchain in > such a way that you can point a variable at it. > >> What I plan to do is make pseudo/targets/bootstrap-tools always build >> all of cross-tools,etc, from Makefile.inc1 respecting its own internal >> logic for when to bootstrap the compiler (without us telling it to skip >> the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a >> global DIRDEP. Given the use of cookies, this will only be done once per >> meta build, but it at least won't be a manual step anymore. For the > > That can still add a lot of time to build, and IIRC building clang alone > poses problems for smaller machines (though using a mutex for linking > clang bits could at least serialize the huge linker steps so as to avoid > exhausting memory). > > I quite like the way NetBSD allow separating the tools from the rest of > the build. I do the equivalent of 'make tools' very rarely - and so > do not care how [in]efficient it is. > IIRC it will throw an error if your tools are incompatible - prompting > an update. > > I would suggest if you want to be able to hook bootstrap-tools in > always, that you do it via a knob so it can be disabled.
I would just add some knob like WITH_BOOTSTRAP_TOOLCHAIN or
WITH_EXTERNAL_TOOLCHAIN to make this and the clang bootstrap just be
skipped and resort to the way the meta build works now.
>
>> record, I do plan to extend .MAKE.MODE=meta to buildworld and its
>> bootstrapping as well, which will give a lot of incremental build
>> improvements to it.
>
> WITH_META_FILES should give you improvements already in that regard.
Yes, it's a step. We'll need cookies in a lot of places too. I wish
WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could
check for "META_MODE" in the buildworld world and for discussion sake.
It seems I can use ${.MAKE.MODE:M*meta*} but that :U is needed in all
the uses. I'm not sure yet if :U really is needed. We have some
${MK_META_MODE} checks now around some cookies that would need to change
for what I'm planning.
>
>> This means that the meta build will then default to bootstrapping the
>> compiler, which we really must do to build the source tree reliably.
>
>> There's really no reason the meta build should default to not
>> bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or
>> WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler
>> as the meta build does now. So there's no real problem for those people
>> who want to skip the clang bootstrap.
>
> Ok, so long as there is a way to do so.
> Otherwise we'd need to add it locally for our own builds - where we need
> to use the toolchains provided by our compiler team.
Yes for sure. As noted above. At Isilon when a developer is building
they have no interest in building the toolchain 99% of the time. This is
why I am so committed to making this automatically skip the toolchain if
possible. As for having an external compiler from a team, that's kind of
already in the "external compiler" realm, so a WITH_EXTERNAL_COMPILER
would make sense to short-circuit all of what I'm wanting. Then you can
add it to your local make.conf or local.sys.mk and not have any change
in behavior.
>
>> Then, I will work on the project of "building clang less" which will
>> avoid building the bootstrap compiler for both the meta mode build and
>> the buildworld build (and universe eventually) if /usr/bin/cc satisfies
>> the needs (can cross compile the TARGET and is "new enough" compared to
>> the version we want). This is not trivial but I think it is possible and
>> it will make everyone happy!
>
> Indeed. As I say, NetBSD have this reasonably sorted.
> But of course they have 2k line shell script driving a lot of it ;-)
Yes the NetBSD build, behavior wise, really impresses me.
>
>> Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP)
>> with buildworld leads to a broken build currently since it is the user
>> basically asking to use their default external toolchain of /usr/bin/*,
>> but the logic does not kick in to use --sysroot against WORLDTMP. I plan
>> to fix this soon as well.
>
> Assuming that you have previously built the correct toolchain
> it should be valid to have something like -DWITH_TOOLSDIR
> -DWITHOUT_BOOTSTRAP_TOOLSDIR (or -DWITHOUT_TOOLSDIR_BOOTSTRAP)
> Such that the build logic is identical - all that is being skipped is
> the expense of re-generating the toolchain
>
--
Regards,
Bryan Drewery
signature.asc
Description: OpenPGP digital signature
