On Wed, Sep 22, 2021, 14:54 Ed W <li...@wildgooses.com> wrote: > On 22/09/2021 20:26, Michael Jones wrote: > > > > On Wed, Sep 22, 2021 at 1:20 PM Ed W <li...@wildgooses.com> wrote: > >> Hi all, traffic seems to have dropped off here significantly, but here >> goes >> >> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64). >> So to keep things simple I >> have just been doing the whole thing using qemu up until now, by which I >> mean I have an arm stage 3 >> somewhere, I chroot into it and then using userspace qemu binaries I just >> run my whole script to >> generate the target build from inside that chroot. This works but it's at >> least a 5x slowdown from >> native >> >> To optimise this I have tried >> >> - turning on the various compiler options for python (claimed to give a >> 30% improvement) + LTO/PGO. >> I don't notice any difference in the chroot - presume that the emulation >> overhead is dominant effect >> >> - tried compiling qemu with -O3 and LTO (claimed to be supported since >> 6.0). Doesn't give any >> noticeable different in performance of emerge >> >> - Added a static compiled amd64 /bin/bash to the chroot - now this does >> give a noticeable boost to >> compile and emerge speeds. (random benchmark went from 26s to 22s) >> >> >> So motivated by the last item I want to try and see how many native exes >> I can push into the chroot >> (since I'm running under usermode qemu! why not!). The obvious one is the >> compiler >> >> Now, I have a cross compiler built, but a) that's not static, so I would >> need to find a way to get >> native libc into the chroot, and b) I'm not clear how I would call it >> inside the chroot, could I >> just move a symlink to the other compiler into the path? How does it find >> things like libgcc*.so etc? >> >> Or perhaps this is easier than this? Can I just use some incantation in >> the same way that the >> crosscompiler must be working to build myself a straight gcc inside the >> chroot which is native arch >> and statically compiled? eg is it enough that assuming I can build gcc >> static, can I just do this >> from outside the chroot and overwrite the native: >> >> ROOT=$PWD emerge -1v --nodeps gcc >> >> >> It seems to me that this should work at least for the gcc binaries, etc. >> However, I'm completely >> ignorant of whether I want things like the linker plugin in native arch >> or target arch? What about >> the libgcc*.so files? (They don't actually exist in my cross compiler >> directories, but they are >> linked in as dependencies in some binaries in target and exist in the >> native compiler dir) >> >> Hacker news had someone do this recently and I believe meego used to do >> something similar, so really >> just trying to work out the details for this on gentoo. Any thoughts? >> >> Thanks >> >> Ed W >> > > > It's not clear to me if you're building gentoo images, or just building > some application. > > If you're building gentoo images, you might consider this project > https://github.com/GenPi64 , we'd love to work with you on the mixed arch > situation, since we suffer the same problem. > > > These are whole gentoo images. :-) > > So it's nothing special, but something like I drop into the arm chroot, > then there is a whole pile of something like: > > ROOT=/mnt/new_image emerge $stuff > > And at the end of all of that you have a shiny image to boot from (on an > imx based SOM as it happens). > > Nice thing about this approach is that I need to build the same system for > i386, amd64 and 32bit arm, and basically it means only running the same > build script in each individual chroot, so it's quite nice not needing to > fixup stuff for each platform. > > > There are arm64bit boxes you can rent from AWS and similar, but we see a > few build oddities on this which still need fixing and at least as near as > I can see they are still quite a bit slower than using an intel processor > in native mode. > > > I'm just about to (re) try using distcc, which basically achieves the > required end goal, so that I can measure performance. So something like run > up a side by side chroot using crossdev, then fire up distcc in there and > talk to it from your arm chroot. This gives less speedup than you would > like because it needs quite a lot of work on the arm qemu side and > serialising stuff, etc. Also linking etc is still on the arm side. > > I think the replacing of the bash binary with a native static binary is > giving a decent speedup. I'm about to try swapping in pypy to see how that > behaves. > > However, there is no doubt that getting the native cross compiler into the > chroot is the solution, but there are more than a few challenges here, such > as how to get it statically compiled and how to insert some or all of it > into the arm chroot. > > See here for inspiration and I guess also the meego stuff from history: > > https://news.ycombinator.com/item?id=28376447 > > > Thanks for any tips! > > Ed W >
The genpi64 project does use distcc for building images when configured. Like I said, I think there'd be a big benefit to collaborating, but the image builder is usable as is for your purpose, if I understand it correctly. Its just missing the native binaries to speed things up. >
