Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Ed W]

On 23/09/2021 23:35, Peter Stuge wrote:
> I've built armv7a images using the armv7a-hardfloat-linux-gnueabi-emerge
> command installed by crossdev with the toolchain after setting up
> etc/portage/ in that root as needed.
>
> That cross emerge cross compiles. :)


What level of breakage do you see these days? Are there any common packages 
which fail?

I'm trying to keep things simple and standard between my native generated 
images and the cross
compiled ones. So the qemu-chroot is nice.


As per your previous email, it seems like the /usr/armv7-xxx and 
/usr/x86_64-pc-linux-gnu/armv7-xx
dirs contain a complete "fanken-compiler" (ie cross compiler) that I could use 
directly inside the
arm chroot. The bits I need to solve are only:

- Setting the binaries to use (could use bits of the crossdev-emerge script)

- How to insert it into the chroot (my preference is to use portage with 
"ROOT=/my-chroot emerge
crossdev-xxx" since that tracks the files and could be upgraded easily in the 
future)

- Getting required bits of libc from the host into the chroot, probably copy 
files manually. Could
do some horrible emerge thing here...


On top of the that it seems straightforward to get binutils and bash to be 
compiled natively/static
and insert that into the chroot. This should get me a chroot with native 
toolchain, that will
crosscompile to armv7a... Need to figure the last steps now


Ed W

Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Peter Stuge]

Hi Ed,

Ed W wrote:
> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64).
> So to keep things simple I have just been doing the whole thing using
> qemu up until now

..

> Now, I have a cross compiler built,

Did you build it with crossdev?


> but a) that's not static, so I
> would need to find a way to get native libc into the chroot, and b)
> I'm not clear how I would call it inside the chroot, could I just
> move a symlink to the other compiler into the path?
> How does it find things like libgcc*.so etc?
> 
> Or perhaps this is easier than this?

If you used crossdev to build e.g. a armv7a-hardfloat-linux-gnueabi
toolchain then look in /usr/armv7a-hardfloat-linux-gnueabi/ where
there should be the beginning of a root.

I've built armv7a images using the armv7a-hardfloat-linux-gnueabi-emerge
command installed by crossdev with the toolchain after setting up
etc/portage/ in that root as needed.

That cross emerge cross compiles. :)


> Any thoughts?

Unfortunately for me, this doesn't integrate with catalyst at all.


After emerging the packages you want you may want a script to
selectively create the final image.


//Peter

Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Ed W]

On 22/09/2021 21:03, Michael Jones wrote:
>
> Like I said, I think there'd be a big benefit to collaborating, but the image 
> builder is usable as
> is for your purpose, if I understand it correctly. Its just missing the 
> native binaries to speed
> things up.
>

OK, so have a look here for inspiration:

    https://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot


I think that lxc script won't function as is today, but the intent seems clear. 
Basically what I had
failed to notice is:

- The cross compiler is a mix of target arch gcc libs and native binaries, so 
this seems safe to
copy into the chroot

- the libc binary is tagged with the arch, at least when not on amd64, so it's 
possible to have both
arm and amd64 libcs in the chroot at the same time without a collision

- setting LD_LIBRARY_PATH is an interesting way to keep the other libs in an 
unusual directory


So I think it should be possible to simply copy in some/all of the files 
created by crossdev into
the arm chroot and then either simply setup your own gcc symlinks (as per 
distcc) or to use the guts
of the crossdev emerge wrapper scripts to do the same

Simplest way to accurately copy in would seem to be the usual trick (from the 
host):

    ROOT=/chroot_dir emerge -1vK --nodeps 
cross-armv7a-unknown-linux-musleabihf/gcc

Where the package at the end is your crossdev gcc

However, the symlinks would seem to support the case of upgrades in the host 
get carried through
into the target... Need to ponder which is simplest. Installing the package 
allows for uninstall and
tracking the files (I guess it completely replaces and overwrites the chroot 
compiler?). The symlink
method will clash and mask out some files in the chroot, accidentally upgrading 
in the chroot would
break everything.


As an aside, getting static exes is a bit of a pain on gentoo, so to note it 
here for the benefit of
google, you can get a static binutils with the following incantation:

  CFLAGS="$(portageq envvar CFLAGS) -static" EXTRA_ECONF="--disable-shared 
LDFLAGS=--static"
CXXFLAGS=$CFLAGS  ROOT=/tmp/test-binutils emerge -1v --nodeps sys-devel/binutils


Generally just the CFLAGS and CXXFLAGS parts are needed, the EXTRA_ECONF is 
just a trick needed for
binutils. My thought was just to install this static package direct into the 
chroot as per the above.


Ed W

Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Michael Jones]

On Wed, Sep 22, 2021, 14:54 Ed W  wrote:

> On 22/09/2021 20:26, Michael Jones wrote:
>
>
>
> On Wed, Sep 22, 2021 at 1:20 PM Ed W  wrote:
>
>> Hi all, traffic seems to have dropped off here significantly, but here
>> goes
>>
>> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64).
>> So to keep things simple I
>> have just been doing the whole thing using qemu up until now, by which I
>> mean I have an arm stage 3
>> somewhere, I chroot into it and then using userspace qemu binaries I just
>> run my whole script to
>> generate the target build from inside that chroot. This works but it's at
>> least a 5x slowdown from
>> native
>>
>> To optimise this I have tried
>>
>> - turning on the various compiler options for python (claimed to give a
>> 30% improvement) + LTO/PGO.
>> I don't notice any difference in the chroot - presume that the emulation
>> overhead is dominant effect
>>
>> - tried compiling qemu with -O3 and LTO (claimed to be supported since
>> 6.0). Doesn't give any
>> noticeable different in performance of emerge
>>
>> - Added a static compiled amd64 /bin/bash to the chroot - now this does
>> give a noticeable boost to
>> compile and emerge speeds. (random benchmark went from 26s to 22s)
>>
>>
>> So motivated by the last item I want to try and see how many native exes
>> I can push into the chroot
>> (since I'm running under usermode qemu! why not!). The obvious one is the
>> compiler
>>
>> Now, I have a cross compiler built, but a) that's not static, so I would
>> need to find a way to get
>> native libc into the chroot, and b) I'm not clear how I would call it
>> inside the chroot, could I
>> just move a symlink to the other compiler into the path? How does it find
>> things like libgcc*.so etc?
>>
>> Or perhaps this is easier than this? Can I just use some incantation in
>> the same way that the
>> crosscompiler must be working to build myself a straight gcc inside the
>> chroot which is native arch
>> and statically compiled? eg is it enough that assuming I can build gcc
>> static, can I just do this
>> from outside the chroot and overwrite the native:
>>
>> ROOT=$PWD emerge -1v --nodeps gcc
>>
>>
>> It seems to me that this should work at least for the gcc binaries, etc.
>> However, I'm completely
>> ignorant of whether I want things like the linker plugin in native arch
>> or target arch? What about
>> the libgcc*.so files? (They don't actually exist in my cross compiler
>> directories, but they are
>> linked in as dependencies in some binaries in target and exist in the
>> native compiler dir)
>>
>> Hacker news had someone do this recently and I believe meego used to do
>> something similar, so really
>> just trying to work out the details for this on gentoo. Any thoughts?
>>
>> Thanks
>>
>> Ed W
>>
>
>
> It's not clear to me if you're building gentoo images, or just building
> some application.
>
> If you're building gentoo images, you might consider this project
> https://github.com/GenPi64 , we'd love to work with you on the mixed arch
> situation, since we suffer the same problem.
>
>
> These are whole gentoo images. :-)
>
> So it's nothing special, but something like I drop into the arm chroot,
> then there is a whole pile of something like:
>
> ROOT=/mnt/new_image emerge $stuff
>
> And at the end of all of that you have a shiny image to boot from (on an
> imx based SOM as it happens).
>
> Nice thing about this approach is that I need to build the same system for
> i386, amd64 and 32bit arm, and basically it means only running the same
> build script in each individual chroot, so it's quite nice not needing to
> fixup stuff for each platform.
>
>
> There are arm64bit boxes you can rent from AWS and similar, but we see a
> few build oddities on this which still need fixing and at least as near as
> I can see they are still quite a bit slower than using an intel processor
> in native mode.
>
>
> I'm just about to (re) try using distcc, which basically achieves the
> required end goal, so that I can measure performance. So something like run
> up a side by side chroot using crossdev, then fire up distcc in there and
> talk to it from your arm chroot. This gives less speedup than you would
> like because it needs quite a lot of work on the arm qemu side and
> serialising stuff, etc. Also linking etc is still on the arm side.
>
> I think the replacing of the bash binary with a native static binary is
> giving a decent speedup. I'm about to try swapping in pypy to see how that
> behaves.
>
> However, there is no doubt that getting the native cross compiler into the
> chroot is the solution, but there are more than a few challenges here, such
> as how to get it statically compiled and how to insert some or all of it
> into the arm chroot.
>
> See here for inspiration and I guess also the meego stuff from history:
>
> https://news.ycombinator.com/item?id=28376447
>
>
> Thanks for any tips!
>
> Ed W
>


The genpi64 project does use distcc for building images 

Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Ed W]

On 22/09/2021 20:26, Michael Jones wrote:
>
>
> On Wed, Sep 22, 2021 at 1:20 PM Ed W  > wrote:
>
> Hi all, traffic seems to have dropped off here significantly, but here 
> goes
>
> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64). 
> So to keep things
> simple I
> have just been doing the whole thing using qemu up until now, by which I 
> mean I have an arm
> stage 3
> somewhere, I chroot into it and then using userspace qemu binaries I just 
> run my whole script to
> generate the target build from inside that chroot. This works but it's at 
> least a 5x slowdown from
> native
>
> To optimise this I have tried
>
> - turning on the various compiler options for python (claimed to give a 
> 30% improvement) +
> LTO/PGO.
> I don't notice any difference in the chroot - presume that the emulation 
> overhead is dominant
> effect
>
> - tried compiling qemu with -O3 and LTO (claimed to be supported since 
> 6.0). Doesn't give any
> noticeable different in performance of emerge
>
> - Added a static compiled amd64 /bin/bash to the chroot - now this does 
> give a noticeable boost to
> compile and emerge speeds. (random benchmark went from 26s to 22s)
>
>
> So motivated by the last item I want to try and see how many native exes 
> I can push into the
> chroot
> (since I'm running under usermode qemu! why not!). The obvious one is the 
> compiler
>
> Now, I have a cross compiler built, but a) that's not static, so I would 
> need to find a way to get
> native libc into the chroot, and b) I'm not clear how I would call it 
> inside the chroot, could I
> just move a symlink to the other compiler into the path? How does it find 
> things like
> libgcc*.so etc?
>
> Or perhaps this is easier than this? Can I just use some incantation in 
> the same way that the
> crosscompiler must be working to build myself a straight gcc inside the 
> chroot which is native
> arch
> and statically compiled? eg is it enough that assuming I can build gcc 
> static, can I just do this
> from outside the chroot and overwrite the native:
>
>     ROOT=$PWD emerge -1v --nodeps gcc
>
>
> It seems to me that this should work at least for the gcc binaries, etc. 
> However, I'm completely
> ignorant of whether I want things like the linker plugin in native arch 
> or target arch? What about
> the libgcc*.so files? (They don't actually exist in my cross compiler 
> directories, but they are
> linked in as dependencies in some binaries in target and exist in the 
> native compiler dir)
>
> Hacker news had someone do this recently and I believe meego used to do 
> something similar, so
> really
> just trying to work out the details for this on gentoo. Any thoughts?
>
> Thanks
>
> Ed W
>
>
>
> It's not clear to me if you're building gentoo images, or just building some 
> application.
>
> If you're building gentoo images, you might consider this project 
> https://github.com/GenPi64
>  , we'd love to work with you on the mixed arch 
> situation, since we
> suffer the same problem.


These are whole gentoo images. :-)

So it's nothing special, but something like I drop into the arm chroot, then 
there is a whole pile
of something like:

    ROOT=/mnt/new_image emerge $stuff

And at the end of all of that you have a shiny image to boot from (on an imx 
based SOM as it happens). 

Nice thing about this approach is that I need to build the same system for 
i386, amd64 and 32bit
arm, and basically it means only running the same build script in each 
individual chroot, so it's
quite nice not needing to fixup stuff for each platform.


There are arm64bit boxes you can rent from AWS and similar, but we see a few 
build oddities on this
which still need fixing and at least as near as I can see they are still quite 
a bit slower than
using an intel processor in native mode.


I'm just about to (re) try using distcc, which basically achieves the required 
end goal, so that I
can measure performance. So something like run up a side by side chroot using 
crossdev, then fire up
distcc in there and talk to it from your arm chroot. This gives less speedup 
than you would like
because it needs quite a lot of work on the arm qemu side and serialising 
stuff, etc. Also linking
etc is still on the arm side.

I think the replacing of the bash binary with a native static binary is giving 
a decent speedup. I'm
about to try swapping in pypy to see how that behaves.

However, there is no doubt that getting the native cross compiler into the 
chroot is the solution,
but there are more than a few challenges here, such as how to get it statically 
compiled and how to
insert some or all of it into the arm chroot.

See here for inspiration and I guess also the meego stuff from history:

    https://news.ycombinator.com/item?id=28376447


Thanks 

Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Michael Jones]

On Wed, Sep 22, 2021 at 1:20 PM Ed W  wrote:

> Hi all, traffic seems to have dropped off here significantly, but here goes
>
> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64).
> So to keep things simple I
> have just been doing the whole thing using qemu up until now, by which I
> mean I have an arm stage 3
> somewhere, I chroot into it and then using userspace qemu binaries I just
> run my whole script to
> generate the target build from inside that chroot. This works but it's at
> least a 5x slowdown from
> native
>
> To optimise this I have tried
>
> - turning on the various compiler options for python (claimed to give a
> 30% improvement) + LTO/PGO.
> I don't notice any difference in the chroot - presume that the emulation
> overhead is dominant effect
>
> - tried compiling qemu with -O3 and LTO (claimed to be supported since
> 6.0). Doesn't give any
> noticeable different in performance of emerge
>
> - Added a static compiled amd64 /bin/bash to the chroot - now this does
> give a noticeable boost to
> compile and emerge speeds. (random benchmark went from 26s to 22s)
>
>
> So motivated by the last item I want to try and see how many native exes I
> can push into the chroot
> (since I'm running under usermode qemu! why not!). The obvious one is the
> compiler
>
> Now, I have a cross compiler built, but a) that's not static, so I would
> need to find a way to get
> native libc into the chroot, and b) I'm not clear how I would call it
> inside the chroot, could I
> just move a symlink to the other compiler into the path? How does it find
> things like libgcc*.so etc?
>
> Or perhaps this is easier than this? Can I just use some incantation in
> the same way that the
> crosscompiler must be working to build myself a straight gcc inside the
> chroot which is native arch
> and statically compiled? eg is it enough that assuming I can build gcc
> static, can I just do this
> from outside the chroot and overwrite the native:
>
> ROOT=$PWD emerge -1v --nodeps gcc
>
>
> It seems to me that this should work at least for the gcc binaries, etc.
> However, I'm completely
> ignorant of whether I want things like the linker plugin in native arch or
> target arch? What about
> the libgcc*.so files? (They don't actually exist in my cross compiler
> directories, but they are
> linked in as dependencies in some binaries in target and exist in the
> native compiler dir)
>
> Hacker news had someone do this recently and I believe meego used to do
> something similar, so really
> just trying to work out the details for this on gentoo. Any thoughts?
>
> Thanks
>
> Ed W
>


It's not clear to me if you're building gentoo images, or just building
some application.

If you're building gentoo images, you might consider this project
https://github.com/GenPi64 , we'd love to work with you on the mixed arch
situation, since we suffer the same problem.