Hi Dave, This discussion about cross-compilation didn't go anywhere. We're thinking of talking about the cross compilation in the monthly VPP call tomorrow, now that latest release is done.. I want to get some agreement on how to properly do the cross-compilation - how feasible the qemu emulation is, how much of the current ebuild system should (or shouldn't) we use or do we need to do something completely different (that would presumably be a lot simpler).
Thanks, Juraj From: Juraj Linkeš <juraj.lin...@pantheon.tech> Sent: Tuesday, December 17, 2019 10:25 AM To: Juraj Linkeš <juraj.lin...@pantheon.tech>; dmar...@me.com; Benoit Ganne (bganne) <bga...@cisco.com> Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Change in vpp[master]: ebuild: Cross compilation aarch64 Ubuntu support I tried a more basic thing in my previous patch: https://gerrit.fd.io/r/c/vpp/+/21035 The patch has one file with all cross-compile arguments/parameters: build-data/platforms/aarch64-generic.mk. It also modifies the ebuild system to propagate those into both VPP and external (dpdk, rdma and such) builds. If we don't want to modify the the ebuild system in this way, I can create separate make targets like Damjan did here: https://gerrit.fd.io/r/c/vpp/+/23153 If anyone's interested in trying it out, there's a script that builds aarch64 dpdk and other external libs along with VPP libraries: build-root/scripts/aarch64-crossbuild.sh. The binaries should be in the same repository. What do you think about this approach? That is, splitting the cross compilation from the environment setup. It seems like a good first step if we're not going to do the emulation builds (see my previous e-mail for reasons why I think it's inadequate). Thanks, Juraj From: Juraj Linkeš <juraj.lin...@pantheon.tech<mailto:juraj.lin...@pantheon.tech>> Sent: Wednesday, December 4, 2019 3:36 PM To: dmar...@me.com<mailto:dmar...@me.com>; Benoit Ganne (bganne) <bga...@cisco.com<mailto:bga...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Change in vpp[master]: ebuild: Cross compilation aarch64 Ubuntu support I looked into this and there are some problems. The first problem is the inability to fine tune any parameters we might want to for target cpu/microarchitecture (for arm, that would be building packages with specifics for ThunderX, McBin, Raspberry PI etc.). I'm not sure how Qemu does the emulation, but I doubt there's extensive support for a myriad of cpus. And on top of that, it's slow. The other thing is with using the x86 version of aarch64-linux-gnu-gcc. In that case it's just regular cross-compilation (x86 version of aarch64-linux-gnu-gcc is the cross-compiler) and what we gain from these target platform containers is having the proper libraries that VPP build depends on. For Ubuntu I've found that it's actually not worth it - it's easier to have an x86 container and install arm dependency packages that to install x86 cross compiler inside an aarch64 container. Another thing I've observed (in htop) is that even when using the x86 version of aarch64-linux-gnu-gcc inside the aarch64 container, it was still going through the emulator, but maybe I did something wrong. Damjan, you mentioned that my current patch doesn't solve anything. It certainly isn't a comprehensive solution, but it does one thing and that is it allows users to specify platform specific config args (well, at least some of the supported ones in build-data/platforms/<platform>.mk) which then get propagated to all parts of the build and it's possible to do cross compilation given that the environemnt has been already set up. It modifies the current ebuild system, but that might not be the appropriate place to do that. However, I don't how else would we do this. I'm not sure what all of this means, but the docker solution is certainly incomplete, if not outright unsuitable. Maybe we could use containers for just environment setup (e.g. for Ubuntu, installing both host and target packages) and then could run cross-compilation in them with a solution that would do something like my patch (i.e. cross-compile DPDK and VPP with config args defined in one file). Thoughts? Juraj From: Damjan Marion via Lists.Fd.Io <dmarion=me....@lists.fd.io<mailto:dmarion=me....@lists.fd.io>> Sent: Thursday, October 31, 2019 1:49 PM To: Benoit Ganne (bganne) <bga...@cisco.com<mailto:bga...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Change in vpp[master]: ebuild: Cross compilation aarch64 Ubuntu support > On 31 Oct 2019, at 13:18, Benoit Ganne (bganne) > <bga...@cisco.com<mailto:bga...@cisco.com>> wrote: > >> I was going to remain silent, but since there's now multiple people saying >> this sounds good -- I think this sounds horrible. :) >> To wit, it seems too complex and too much setup/overhead. I'll try and >> look closer at this soon to see if I can feed back our local changes that >> seem to be working. > > It is not that bad in my opinion [1] : > 1) add support for multiarch (must be done once after reboot) > ~# docker run --rm --privileged multiarch/qemu-user-static --reset > --persistent yes --credential yes > 2) create your chroot (must be done once - I am sharing my homedir with my > chroot and same UID/GID) > ~# docker run --name aarch64_u1804 --privileged --net host -v $HOME:$HOME -v > /dev:/dev -v/lib/modules:/lib/modules/host:ro -td arm64v8/ubuntu:18.04 > /bin/bash > ~# docker container exec aarch64_u1804 sh -c "apt -qy update && apt > dist-upgrade -qy && apt install -qy vim sudo make git && groupadd -g $(id > -rg) $USER && useradd -u $(id -ru) -g $(id -rg) -M -d $HOME -s /bin/bash > $USER && echo '$USER ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && echo > aarch64_u1804 > /etc/debian_chroot" > 3) compile vpp (I already checked out VPP in $HOME/src/vpp but you can > checkout it there too if you prefer) > ~# docker container exec aarch64_u1804 su "$USER" -l -c "UNATTENTED=y make -C > src/vpp install-dep install-ext-deps pkg-deb" > [...] > dpkg-deb: building package 'libvppinfra-dev' in > '../libvppinfra-dev_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-dbg' in > '../vpp-dbg_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'libvppinfra' in > '../libvppinfra_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-api-python' in > '../vpp-api-python_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp' in > '../vpp_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-plugin-dpdk' in > '../vpp-plugin-dpdk_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'python3-vpp-api' in > '../python3-vpp-api_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-dev' in > '../vpp-dev_20.01-rc0~538-gbb41ee925_arm64.deb'. > dpkg-deb: building package 'vpp-plugin-core' in > '../vpp-plugin-core_20.01-rc0~538-gbb41ee925_arm64.deb'. > make[2]: Leaving directory > '/home/bganne/src/vpp/build-root/build-vpp-native/vpp' > dpkg-genbuildinfo --build=binary > dpkg-genchanges --build=binary >../vpp_20.01-rc0~538-gbb41ee925_arm64.changes > dpkg-genchanges: info: binary-only upload (no source code included) > dpkg-source --after-build vpp > dpkg-buildpackage: info: binary-only upload (no source included) > make[1]: Leaving directory '/home/bganne/src/vpp/build-root' > make: Leaving directory '/home/bganne/src/vpp' > > Enjoy :) This is so cool :) I wanted to invest time to investigate how to do exactly this, so thanks for saving my time Only nice thing will be to find way to run x86 version of aarch64-linux-gnu-gcc instead of aarch64 version of gcc so it runs faster...
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15372): https://lists.fd.io/g/vpp-dev/message/15372 Mute This Topic: https://lists.fd.io/mt/39781289/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
