Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]

[Bryan Drewery]

On 9/3/2015 11:46 AM, Simon J. Gerraty wrote:
> Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an
> initial set of tools.  It then attempts to use that to build toolchain
> for MACHINE=host which is currently failing because in-tree libelf and
> libdwarf are needed and libelf needs sys/sys/elf_common.h but but that
> doesn't currently get staged for host (we try to minimize what we put in
> host stage tree).

Using the special hack you came up with for lib/libelf, and a lot of
other fixes for replacing binutils with elftoolchain, I've managed to
get the targets/pseduo/bootstrap-tools build working again. I will
commit this likely today or tomorrow.

However...

> 
> It may be better to simply skip targets/pseudo/toolchain.host
> that will work so long as we set TOOLSDIR to point to the stuff built by
> Makefile.inc1 
> 
> Depending on where we are with external toolchian support that would
> make sense - might be able to skip bootstrap-toolchain completely
> which would be nice.

I think this is more right because there's many more people thinking
about, testing daily, and maintaining the bootstrap logic in
Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently
lead to bitrot and breakage with the elftoolchain replacement. It will
very easily happen again.

Right now the targets/pseduo/bootstrap-tools target builds
legacy,build-tools,cross-tools from Makefile.inc1 for everything but the
compiler and toolchain, and then building the toolchain itself. There's
a subtle problem with this in that we end up building clang in
targets/pseudo/toolchain with -nostdinc (from local.init.mk) which leads
to a missing header (which actually is staged, just lacks a -I or
--sysroot to be captured). But really there's no reason to do this work
since Makefile.inc1 does it fine and is maintained well to handle it.

There's another subtle thing here, because we set CC=HOST_CC=/usr/bin/cc
when calling cross-tools from the pseudo/targets/bootstrap-tools build,
it invokes the external compiler support in Makefile.inc1 which avoids
building the bootstrap clang and some of the toolchain, leading us to
have to do it from pseudo/targets/toolchain. I actually extended this
logic to pass HOST_AS to avoid *more* redundant toolchain building in
Makefile.inc1 from our own targets/pseduo/toolchain, before realizing this.

What I plan to do is make pseudo/targets/bootstrap-tools always build
all of cross-tools,etc, from Makefile.inc1 respecting its own internal
logic for when to bootstrap the compiler (without us telling it to skip
the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a
global DIRDEP. Given the use of cookies, this will only be done once per
meta build, but it at least won't be a manual step anymore. For the
record, I do plan to extend .MAKE.MODE=meta to buildworld and its
bootstrapping as well, which will give a lot of incremental build
improvements to it.

This means that the meta build will then default to bootstrapping the
compiler, which we really must do to build the source tree reliably.
There's really no reason the meta build should default to not
bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or
WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler
as the meta build does now. So there's no real problem for those people
who want to skip the clang bootstrap.

Then, I will work on the project of "building clang less" which will
avoid building the bootstrap compiler for both the meta mode build and
the buildworld build (and universe eventually) if /usr/bin/cc satisfies
the needs (can cross compile the TARGET and is "new enough" compared to
the version we want). This is not trivial but I think it is possible and
it will make everyone happy!

Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP)
with buildworld leads to a broken build currently since it is the user
basically asking to use their default external toolchain of /usr/bin/*,
but the logic does not kick in to use --sysroot against WORLDTMP. I plan
to fix this soon as well.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]

[Simon J. Gerraty]

Bryan Drewery  wrote:

> On 9/3/2015 11:46 AM, Simon J. Gerraty wrote:
> > Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an
> > initial set of tools.  It then attempts to use that to build toolchain
> > for MACHINE=host which is currently failing because in-tree libelf and
> > libdwarf are needed and libelf needs sys/sys/elf_common.h but but that
> > doesn't currently get staged for host (we try to minimize what we put in
> > host stage tree).
> 
> Using the special hack you came up with for lib/libelf, and a lot of
> other fixes for replacing binutils with elftoolchain, I've managed to
> get the targets/pseduo/bootstrap-tools build working again. I will
> commit this likely today or tomorrow.

Cool - thanks
 
> However...

> > It may be better to simply skip targets/pseudo/toolchain.host
> > that will work so long as we set TOOLSDIR to point to the stuff built by
> > Makefile.inc1 
> > 
> > Depending on where we are with external toolchian support that would
> > make sense - might be able to skip bootstrap-toolchain completely
> > which would be nice.
> 
> I think this is more right because there's many more people thinking
> about, testing daily, and maintaining the bootstrap logic in
> Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently
> lead to bitrot and breakage with the elftoolchain replacement. It will
> very easily happen again.

I'd be favor of removing the need completely.

All that is required is *some* means of producing a viable toolchain in
such a way that you can point a variable at it.

> What I plan to do is make pseudo/targets/bootstrap-tools always build
> all of cross-tools,etc, from Makefile.inc1 respecting its own internal
> logic for when to bootstrap the compiler (without us telling it to skip
> the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a
> global DIRDEP. Given the use of cookies, this will only be done once per
> meta build, but it at least won't be a manual step anymore. For the

That can still add a lot of time to build, and IIRC building clang alone
poses problems for smaller machines (though using a mutex for linking
clang bits could at least serialize the huge linker steps so as to avoid
exhausting memory).

I quite like the way NetBSD allow separating the tools from the rest of
the build.  I do the equivalent of 'make tools' very rarely - and so
do not care how [in]efficient it is.
IIRC it will throw an error if your tools are incompatible - prompting
an update.

I would suggest if you want to be able to hook bootstrap-tools in
always, that you do it via a knob so it can be disabled.

> record, I do plan to extend .MAKE.MODE=meta to buildworld and its
> bootstrapping as well, which will give a lot of incremental build
> improvements to it.

WITH_META_FILES should give you improvements already in that regard.

> This means that the meta build will then default to bootstrapping the
> compiler, which we really must do to build the source tree reliably.

> There's really no reason the meta build should default to not
> bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or
> WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler
> as the meta build does now. So there's no real problem for those people
> who want to skip the clang bootstrap.

Ok, so long as there is a way to do so.
Otherwise we'd need to add it locally for our own builds - where we need
to use the toolchains provided by our compiler team.

> Then, I will work on the project of "building clang less" which will
> avoid building the bootstrap compiler for both the meta mode build and
> the buildworld build (and universe eventually) if /usr/bin/cc satisfies
> the needs (can cross compile the TARGET and is "new enough" compared to
> the version we want). This is not trivial but I think it is possible and
> it will make everyone happy!

Indeed.  As I say, NetBSD have this reasonably sorted.
But of course they have 2k line shell script driving a lot of it ;-)

> Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP)
> with buildworld leads to a broken build currently since it is the user
> basically asking to use their default external toolchain of /usr/bin/*,
> but the logic does not kick in to use --sysroot against WORLDTMP. I plan
> to fix this soon as well.

Assuming that you have previously built the correct toolchain
it should be valid to have something like -DWITH_TOOLSDIR
-DWITHOUT_BOOTSTRAP_TOOLSDIR (or -DWITHOUT_TOOLSDIR_BOOTSTRAP)
Such that the build logic is identical - all that is being skipped is
the expense of re-generating the toolchain
___
freebsd-toolchain@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

amd64-gcc question

[William A. Mahaffey III]

Howdy, new to this list. I posted this to FreeBSD-users & was advised to 
try this list or the GNU GCC list, so here goes:



I pkg-installed amd64-gcc over the weekend hoping for Graphite 
(auto-loop parallelization) support, but no go. I looked around over the 
weekend & found that there was no port for that package, only the pkg. I 
just did a 'portsnap fetch upgrade' & there is now a port for amd64-gcc, 
but it includes no files & no pkg-descr file. I determined over the 
weekend that the gcc's from about V4.3 on can indeed be built w/ 
Graphite support, but you need to do it manually. I found a post dated 
2010 from someone who did it under linux: 
http://openwall.info/wiki/internal/gcc-local-build. I see no configure 
files for any of the gcc ports (I have the entire ports tree downloaded 
& local, & freshly updated as of a few min. ago). What is the 
canonical/BPP (FreeBSD 9.3R) way of recompiling a port with different 
config flags ?



I did find ports/pkgs for the 2 main components apparently needed for 
Graphite support (cloog & ppl) & pkg-installed them over the weekend, so 
I am ready to go on that front.



I have gotten as far as running 'make showconfig' in the various gcc* & 
amd64-gcc directories to see what info I could get on default config 
options. In all cases they gave options & said to run 'make config' to 
change options. I didn't even see a 'config:' entry in the Makefiles 
(probably included from elsewhere, but I didn't chase it). I only want 
to make the minimum # of config mods necessary (trusting that pkg/port 
maintainers probably know more than I about their various pkg's & ports) 
to add the cloog & ppl support & recompile.



I have been using pkg almost exclusively to maintain my (now 3) FreeBSD 
9.3R boxen, except for recompiling the linux-c6 flash plugin for this 
box whenever it get upgraded, so I have *no* experience with getting 
more nitty-gritty w/ FreeBSD ports than that :-/. TIA & have a good one.



BTW:

[wam@devbox, pre, 8:19:58am] 676 % uname -a
FreeBSD devbox 9.3-RELEASE-p24 FreeBSD 9.3-RELEASE-p24 #0: Sat Aug 22 
01:54:44 UTC 2015 
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

[wam@devbox, pre, 8:20:00am] 677 % sysctl -A | grep -A1 -B1 model
hw.machine: amd64
hw.model: AMD A8-6500 APU with Radeon(tm) HD Graphics
hw.ncpu: 4
--
dev.rgephy.0.%location: phyno=1
dev.rgephy.0.%pnpinfo: oui=0xe04c model=0x0 rev=0x0
dev.rgephy.0.%parent: miibus0
[wam@devbox, pre, 8:20:12am] 678 %


--

William A. Mahaffey III

 --

"The M1 Garand is without doubt the finest implement of war
 ever devised by man."
   -- Gen. George S. Patton Jr.

___
freebsd-questi...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

___
freebsd-toolchain@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"