Re: Summary of the Arm ports BoF at DC17
On Mon, Sep 18, 2017 at 10:50:39AM +0100, Ian Campbell wrote: > On Mon, 2017-09-18 at 10:15 +0100, Edmund Grimley Evans wrote: > > > But it did remind me that on some platforms writing "2" to > > > /proc/sys/abi/cp15_barrier will enable hw support for these > > > instructions, since some platforms do support them even thought > > > they > > > are deprecated. It's certainly worth investigating what your > > > hardware > > > supports. > > > > Not necessarily disagreeing with the content of that but the > > terminology seems wrong: I don't have a reference to hand but I'm > > fairly sure that hardware is *required* to support a "deprecated" > > instruction. "Deprecated" usually means something like: unless you > > know what you're doing you probably shouldn't use this feature > > because > > it might perform badly and it might not be available in a future > > version (of the Arm architecture in this case). > > In principal I agree with your interpretation of "Deprecated" but the > wording in Linux's Documentation/arm64/legacy_instructions.txt which > documents this stuff is: > > * Hardware Execution > Value: 2 > Although marked as deprecated, some implementations may support the > enabling/disabling of hardware support for the execution of these > instructions. Using hardware execution generally provides better > performance, but at the loss of ability to gather runtime statistics > about the use of the deprecated instructions. > > Which in "some implementations may" at least implies the possibility > that some hardware may not support these instructions. I don't seem to > be able to download an ARM ARM right now to see what the actual wording > used there is. I believe it says some hardware might support disabling the support for the deprecated instruction. It does not say they don't support the deprecated instruction. Having the ability to disable a feature that you ideally shouldn't use can be very helpful to make sure you aren't accidentally using it after all. Nothing in that text says that some hardware might not implement the deprecated instruction. The might is talking about the ability to turn off deprecated instructions. -- Len Sorensen
Re: Summary of the Arm ports BoF at DC17
On Mon, 2017-09-18 at 10:15 +0100, Edmund Grimley Evans wrote: > > But it did remind me that on some platforms writing "2" to > > /proc/sys/abi/cp15_barrier will enable hw support for these > > instructions, since some platforms do support them even thought > > they > > are deprecated. It's certainly worth investigating what your > > hardware > > supports. > > Not necessarily disagreeing with the content of that but the > terminology seems wrong: I don't have a reference to hand but I'm > fairly sure that hardware is *required* to support a "deprecated" > instruction. "Deprecated" usually means something like: unless you > know what you're doing you probably shouldn't use this feature > because > it might perform badly and it might not be available in a future > version (of the Arm architecture in this case). In principal I agree with your interpretation of "Deprecated" but the wording in Linux's Documentation/arm64/legacy_instructions.txt which documents this stuff is: * Hardware Execution Value: 2 Although marked as deprecated, some implementations may support the enabling/disabling of hardware support for the execution of these instructions. Using hardware execution generally provides better performance, but at the loss of ability to gather runtime statistics about the use of the deprecated instructions. Which in "some implementations may" at least implies the possibility that some hardware may not support these instructions. I don't seem to be able to download an ARM ARM right now to see what the actual wording used there is. Ian.
Re: Summary of the Arm ports BoF at DC17
> But it did remind me that on some platforms writing "2" to > /proc/sys/abi/cp15_barrier will enable hw support for these > instructions, since some platforms do support them even thought they > are deprecated. It's certainly worth investigating what your hardware > supports. Not necessarily disagreeing with the content of that but the terminology seems wrong: I don't have a reference to hand but I'm fairly sure that hardware is *required* to support a "deprecated" instruction. "Deprecated" usually means something like: unless you know what you're doing you probably shouldn't use this feature because it might perform badly and it might not be available in a future version (of the Arm architecture in this case).
Re: Summary of the Arm ports BoF at DC17
On Sun, 2017-09-17 at 12:59 -0700, Vagrant Cascadian wrote: > On 2017-09-15, Ben Hutchings wrote: > > On Thu, 2017-09-14 at 03:40 +0100, Steve McIntyre wrote: > > [...] > >> There is optional kernel support to trap the exceptions here > >> and emulate the instructions, but it's really not recommended for > >> serious use (e.g. on a build machine!). > > [...] > > > > Why is it not recommended? Terrible performance, or known bugs? > > On arm64 kernels building armhf packages for the reproducible builds > builders, I'm seeing hundreds or even thousands of kernel log warnings > per second when building anything that makes use of > CP15_BARRIER_EMULATION (e.g. anything using ghc): > > [85740.553537] cp15barrier_handler: 115316 callbacks suppressed > [85740.559358] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction > at 0xf5beaa7c > [85740.567344] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction > at 0xf5beaa9c > > That *might* be sufficient to actually impact performance; it certainly > makes it hard to read other kernel log messages on those machines... I bet it does! I was a bit surprised to find in the docs that there was no "silently emulate" option in the proc interface, seems my memory was faulty there. But it did remind me that on some platforms writing "2" to /proc/sys/abi/cp15_barrier will enable hw support for these instructions, since some platforms do support them even thought they are deprecated. It's certainly worth investigating what your hardware supports. Ian.
Re: Summary of the Arm ports BoF at DC17
On Sun, 2017-09-17 at 21:30 +0100, Colin Watson wrote: > On Sun, Sep 17, 2017 at 12:59:04PM -0700, Vagrant Cascadian wrote: > > On 2017-09-15, Ben Hutchings wrote: > > > On Thu, 2017-09-14 at 03:40 +0100, Steve McIntyre wrote: > > > [...] > > > > There is optional kernel support to trap the exceptions here > > > > and emulate the instructions, but it's really not recommended for > > > > serious use (e.g. on a build machine!). > > > > > > [...] > > > > > > Why is it not recommended? Terrible performance, or known bugs? > > > > On arm64 kernels building armhf packages for the reproducible builds > > builders, > > Steve was mostly talking about armel, I thought? [...] > I can't really speak for how it would be on armel, but I rather suspect > that decent ARMv8 hardware would be sufficiently much faster than > ARMv7-or-earlier kit to more than cancel out performance losses from > instruction emulation. For armel, we can't assume there *are* any memory barrier instructions. Instead, atomic operations libraries or code generators should use the kuser helper memory_barrier(), which will do the right thing on an SMP system and nothing on a UP system. (kuser helpers are user-mode functions exposed by the kernel.) So far as I can see, there are three sets of deprecated and emulated instructions and I think they should be replaced as follows: min. arch armelarmhf - -- swp/swpb ARMv4? kuser cmpxchg() ldrex+strex cp15 memory barrier ARMv6 N/A dmb/dsb/isb setend ARMv6 N/A rev/rev16 (I think swp/swpb were originally specified to bypass the data cache, which meant they was never useful in user mode armel code.) Ben. -- Ben Hutchings friends: People who know you well, but like you anyway. signature.asc Description: This is a digitally signed message part
Re: Summary of the Arm ports BoF at DC17
On Fri, Sep 15, 2017 at 07:11:38PM +0100, Ben Hutchings wrote: >On Thu, 2017-09-14 at 03:40 +0100, Steve McIntyre wrote: >[...] >> There is optional kernel support to trap the exceptions here >> and emulate the instructions, but it's really not recommended for >> serious use (e.g. on a build machine!). >[...] > >Why is it not recommended? Terrible performance, or known bugs? I've heard estimates for performance slowdown of a factor of 20 or more, maybe worse. It's very dependent on the hardware and the code in question obviously. Sorry, I don't have any references, I'm afraid. :-/ Of course, the massive log overheard that Vagrant reports won't help much either! I'm not aware of any bugs OTTOMH. -- Steve McIntyre, Cambridge, UK.st...@einval.com There's no sensation to compare with this Suspended animation, A state of bliss
Re: Summary of the Arm ports BoF at DC17
On Sun, 2017-09-17 at 12:59 -0700, Vagrant Cascadian wrote: > On 2017-09-15, Ben Hutchings wrote: > > On Thu, 2017-09-14 at 03:40 +0100, Steve McIntyre wrote: > > [...] > > > There is optional kernel support to trap the exceptions here > > > and emulate the instructions, but it's really not recommended for > > > serious use (e.g. on a build machine!). > > > > [...] > > > > Why is it not recommended? Terrible performance, or known bugs? > > On arm64 kernels building armhf packages for the reproducible builds > builders, I'm seeing hundreds or even thousands of kernel log warnings > per second when building anything that makes use of > CP15_BARRIER_EMULATION (e.g. anything using ghc): > > [85740.553537] cp15barrier_handler: 115316 callbacks suppressed > [85740.559358] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction > at 0xf5beaa7c > [85740.567344] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction > at 0xf5beaa9c > > That *might* be sufficient to actually impact performance; it certainly > makes it hard to read other kernel log messages on those machines... The log rate-limiting is done in the kernel, so the log daemon doesn't have to process all those messages. However, the rate-limiting state is protected with a mutex (actually a spin-lock) so even when the log message is suppressed it involves an additional two memory barriers. And I agree that this kind of rate-limiting isn't really sufficient. Warning once per process might be more reasonable, but there's currently no mechanism for that in the kernel. Ben. > There was a bug reported on ghc related to this: > > https://bugs.debian.org/864847 > > > live well, > vagrant -- Ben Hutchings friends: People who know you well, but like you anyway. signature.asc Description: This is a digitally signed message part
Re: Summary of the Arm ports BoF at DC17
On Sun, Sep 17, 2017 at 12:59:04PM -0700, Vagrant Cascadian wrote: > On 2017-09-15, Ben Hutchings wrote: > > On Thu, 2017-09-14 at 03:40 +0100, Steve McIntyre wrote: > > [...] > >> There is optional kernel support to trap the exceptions here > >> and emulate the instructions, but it's really not recommended for > >> serious use (e.g. on a build machine!). > > [...] > > > > Why is it not recommended? Terrible performance, or known bugs? > > On arm64 kernels building armhf packages for the reproducible builds > builders, Steve was mostly talking about armel, I thought? > I'm seeing hundreds or even thousands of kernel log warnings > per second when building anything that makes use of > CP15_BARRIER_EMULATION (e.g. anything using ghc): > > [85740.553537] cp15barrier_handler: 115316 callbacks suppressed > [85740.559358] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction > at 0xf5beaa7c > [85740.567344] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction > at 0xf5beaa9c > > > That *might* be sufficient to actually impact performance; it certainly > makes it hard to read other kernel log messages on those machines... We've been building armhf packages in arm64 VMs for some time in Ubuntu and it generally seems to work fine; I haven't looked *very* closely but I haven't noticed any particularly significant performance difference between armhf and arm64 builds in that environment, including builds that use GHC. (I would have to go to some effort to look at kernel logs, so I haven't.) We're using HP ProLiant m400 cartridges running as OpenStack compute nodes, and I gather that hardware supports a few more ARMv7 instructions than some ARMv8 hardware does. We also run with a small kernel tweak [1] to set "uname -m" output to "armv7l" when running under linux32(1), which makes some dodgy configure scripts behave themselves a bit better. But apart from that, it's been a pretty good experience on armhf and definitely much faster than what we had before. [1] https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/kernel/sys.c?id=fff84367fa2a19f20c2b9094b47bc3db51fed183 I can't really speak for how it would be on armel, but I rather suspect that decent ARMv8 hardware would be sufficiently much faster than ARMv7-or-earlier kit to more than cancel out performance losses from instruction emulation. -- Colin Watson [cjwat...@debian.org]
Re: Summary of the Arm ports BoF at DC17
On 2017-09-15, Ben Hutchings wrote: > On Thu, 2017-09-14 at 03:40 +0100, Steve McIntyre wrote: > [...] >> There is optional kernel support to trap the exceptions here >> and emulate the instructions, but it's really not recommended for >> serious use (e.g. on a build machine!). > [...] > > Why is it not recommended? Terrible performance, or known bugs? On arm64 kernels building armhf packages for the reproducible builds builders, I'm seeing hundreds or even thousands of kernel log warnings per second when building anything that makes use of CP15_BARRIER_EMULATION (e.g. anything using ghc): [85740.553537] cp15barrier_handler: 115316 callbacks suppressed [85740.559358] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction at 0xf5beaa7c [85740.567344] "ghc-pkg" (29572) uses deprecated CP15 Barrier instruction at 0xf5beaa9c That *might* be sufficient to actually impact performance; it certainly makes it hard to read other kernel log messages on those machines... There was a bug reported on ghc related to this: https://bugs.debian.org/864847 live well, vagrant signature.asc Description: PGP signature
Re: Summary of the Arm ports BoF at DC17
On 2017-09-14 12:58 +0800, Paul Wise wrote: > Wookey, could you add something about the motivation for arm64ilp32 to > the wiki page about it? Will do. But the short version is that it's only useful if you need to run 32-bit code on hardware that only supports the 64-bit execution mode. Such hardware does exist (Cavium ThunderX), and it can't run armhf. It's not clear if more manufacturers will make such chips. And then you need a reason to run as 32-bit, but where you can rebuild code. So it's no use for old binaries, but it is useful for codebases that are not 64-bit safe. There might be arguments about memory usage (as for x32) where 32-bit makes sense, but I've not seen any data on that. As of today (about 3 hours ago :-), there is an ILP32 debian stable debootrappable set of packages, so it's possible to do said tests. Wookey -- Principal hats: Linaro, Debian, Wookware, ARM http://wookware.org/ signature.asc Description: Digital signature
Re: Summary of the Arm ports BoF at DC17
On Thu, 2017-09-14 at 03:40 +0100, Steve McIntyre wrote: [...] > There is optional kernel support to trap the exceptions here > and emulate the instructions, but it's really not recommended for > serious use (e.g. on a build machine!). [...] Why is it not recommended? Terrible performance, or known bugs? Ben. -- Ben Hutchings Kids! Bringing about Armageddon can be dangerous. Do not attempt it in your own home. - Terry Pratchett and Neil Gaiman, `Good Omens' signature.asc Description: This is a digitally signed message part
Re: Summary of the Arm ports BoF at DC17
On Thu, Sep 14, 2017 at 12:06:13PM +0200, Marco d'Itri wrote:
>On Sep 14, Steve McIntyre wrote:
>
>> The Pine64 [6] is another alternative, based on a mobile CPU. It's
>> therefore got limited RAM and I/O. Upstreaming has taken a while, but
>> is getting there in current kernel releases. U-Boot head will work on
>> the board, including the UEFI implementation mentioned earlier.
>> There's the related PineBook project [7] too - a small laptop-style
>> machine based around the Pine64 board.
>There are also scary warnings and discussions about the Ethernet port
>being half-broken at 1 Gbps.
>
>> Answering this question ("I want something that just works, what
>> should I buy?") is always much harder than it should be... :-(
>On the lower end I strongly recommend the Olimex Lime 2, which is open
>hardware and works well with the mainline kernel and U-Boot from stretch
>(as long as you do not care about 3D, obviously):
>https://www.armbian.com/olimex-lime-2/
ACK, thanks for the recommendation!
>BTW... I like installing my little ARM systems using debootstrap
>--foreign, but I am unable to do a 100% self-contained installation
>(without using qemu, which would be cheating, or copying the initramfs
>from another system) because the initramfs has not been built yet.
>Do we distribute anyware one that may be used for the first boot?
Not that I'm aware of, no.
--
Steve McIntyre, Cambridge, UK.st...@einval.com
"C++ ate my sanity" -- Jon Rabone
Re: Summary of the Arm ports BoF at DC17
On Thu, Sep 14, 2017 at 12:58:02PM +0800, Paul Wise wrote: >On Thu, Sep 14, 2017 at 10:40 AM, Steve McIntyre wrote: > >> It's possible to replace the installed U-Boot >> on many boards, but that depends on hardware support being properly >> upstreamed; lack of that upstreaming work is another common bugbear in >> vendor U-Boot binaries. If you want to use your arm64 machine as a >> *computer*, my recommended route is UEFI; this is why all the server >> vendors have gone that route. > >Doesn't the same problem of u-boot upstreaming also apply to >UEFI/TianoCore upstreaming (but worse)? AFAICT we do not have *any* >support in our edk2 source package for anything other than virtual >machines (x86/ARMv7/ARMv8). It seems to me that u-boot is in a much >better position wrt upstream support for ARM *and* for support for >UEFI than TianoCore or other options that aren't in Debian like >coreboot. Let's be straight - the UEFI boot support in U-Boot is very much a limited project to ease booting; it's not *at all* a complete implementation of UEFI like Tianocore. It doesn't support any of the runtime interfaces, for example. I stand by my recommendation: for normal use for an end user on an arm64 machine, UEFI is a better bet than U-Boot. ... >> Last question: what should people do to keep armel? Main things: >> >> * deal with armel-specific bug reports > >For some reason there isn't an armel usertag but there is an eabi one: > >https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=eabi&user=debian-arm%40lists.debian.org > >> * get involved in toolchains and make sure they continue to work > >Join the #debian-toolchain IRC channel and tell doko you are there to >support armel. +1 >Improve the wiki page for armel, move it under the Ports hierarchy and >base it off the port template: > >https://wiki.debian.org/ArmEabiPort >https://wiki.debian.org/Ports >https://wiki.debian.org/PortTemplate > >Improve the web page for the ARM ports. > >https://www.debian.org/ports/arm/ Alternatively, give up on the ports web pages altogether - IMHO they're massively more effort to maintain than stuff in the wiki and that's one of the reasons why they're always so out-dated. -- Steve McIntyre, Cambridge, UK.st...@einval.com You raise the blade, you make the change... You re-arrange me 'til I'm sane...
Re: Summary of the Arm ports BoF at DC17
On Sep 14, Steve McIntyre wrote:
> The Pine64 [6] is another alternative, based on a mobile CPU. It's
> therefore got limited RAM and I/O. Upstreaming has taken a while, but
> is getting there in current kernel releases. U-Boot head will work on
> the board, including the UEFI implementation mentioned earlier.
> There's the related PineBook project [7] too - a small laptop-style
> machine based around the Pine64 board.
There are also scary warnings and discussions about the Ethernet port
being half-broken at 1 Gbps.
> Answering this question ("I want something that just works, what
> should I buy?") is always much harder than it should be... :-(
On the lower end I strongly recommend the Olimex Lime 2, which is open
hardware and works well with the mainline kernel and U-Boot from stretch
(as long as you do not care about 3D, obviously):
https://www.armbian.com/olimex-lime-2/
BTW... I like installing my little ARM systems using debootstrap
--foreign, but I am unable to do a 100% self-contained installation
(without using qemu, which would be cheating, or copying the initramfs
from another system) because the initramfs has not been built yet.
Do we distribute anyware one that may be used for the first boot?
--
ciao,
Marco
signature.asc
Description: PGP signature
Re: Summary of the Arm ports BoF at DC17
On Thu, Sep 14, 2017 at 10:40 AM, Steve McIntyre wrote: > It's possible to replace the installed U-Boot > on many boards, but that depends on hardware support being properly > upstreamed; lack of that upstreaming work is another common bugbear in > vendor U-Boot binaries. If you want to use your arm64 machine as a > *computer*, my recommended route is UEFI; this is why all the server > vendors have gone that route. Doesn't the same problem of u-boot upstreaming also apply to UEFI/TianoCore upstreaming (but worse)? AFAICT we do not have *any* support in our edk2 source package for anything other than virtual machines (x86/ARMv7/ARMv8). It seems to me that u-boot is in a much better position wrt upstream support for ARM *and* for support for UEFI than TianoCore or other options that aren't in Debian like coreboot. > Hector asked about the current state of the less well-known Arm ports: > big-endian versions and arm64ilp32. Lots of different BE ports have > happened over the years, for various reasons. arm64ilp32 [8] is the > arm64 equivalent of x32 on x64-64; IMHO there is no need for it in > Debian as a mainstream port. Wookey is working on it in Linaro at the > moment. There is a BE version too. While x32 *might* have some > performance wins over i386 (depending on code), it's not clear that > arm64ilp32 is likely to be any better than armhf (A32) in most cases. Wookey, could you add something about the motivation for arm64ilp32 to the wiki page about it? > Last question: what should people do to keep armel? Main things: > > * deal with armel-specific bug reports For some reason there isn't an armel usertag but there is an eabi one: https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=eabi&user=debian-arm%40lists.debian.org > * get involved in toolchains and make sure they continue to work Join the #debian-toolchain IRC channel and tell doko you are there to support armel. Also: Test that d-i and other installation methods work. Test that u-boot works for armel devices: https://packages.debian.org/search?keywords=u-boot Fix the issues mentioned on the buildd page: https://buildd.debian.org/status/architecture.php?a=armel&suite=sid Improve the release team's arch qualification status for armel: https://release.debian.org/testing/arch_qualify.html Monitor the rebootstrap for armel and fix any issues that come up, join #debian-bootstrap and mention your interest to helmut. https://wiki.debian.org/HelmutGrohne/rebootstrap Package and test the RPi open firmware: https://github.com/christinaa/rpi-open-firmware Improve the wiki page for armel, move it under the Ports hierarchy and base it off the port template: https://wiki.debian.org/ArmEabiPort https://wiki.debian.org/Ports https://wiki.debian.org/PortTemplate Improve the web page for the ARM ports. https://www.debian.org/ports/arm/ Extend architecture support for some services: https://ci.debian.net/status/ https://tests.reproducible-builds.org/debian/reproducible.html http://cdimage.debian.org/cdimage/weekly-live-builds/ -- bye, pabs https://wiki.debian.org/PaulWise
Summary of the Arm ports BoF at DC17
[ Please note the cross-post and Reply-To ] Hi folks, As promised, here's a quick summary of what was discussed at the Arm ports BoF session in Montréal. Apologies for the delay in posting... Thanks to the awesome efforts of our video team, the session is already online [1]. I've taken a copy of the Gobby notes too, alongside my small set of slides for the session. [2] We spoke about the past/present/future state of Arm ports in Debian. arm64 = Most recent Arm port, first released with Jessie. We're starting to see more devices becoming available that will run Debian arm64 well. We're also starting to get real server hardware turning up. arm64 hardware often follows better standards than some of the older generations of Arm hardware, meaning that most of the machines coming (assuming kernel support!) should work well out of the box using device tree or ACPI. Please use ACPI if you can - the two are mostly equivalent in terms of what they describe, but ACPI is a better choice for the future where possible, with better standards. armhf = Much as in previous BoFs in terms of support and future! Current port, first released with Wheezy. Due to cross-distro effort, this setup (ARMv7 EABI using VFPv3D16) is the default supported 32-bit Arm architecture in all distros now. This means that there's even a fair chance of binaries built for one distro running on others! We've got a couple of multi-platform kernel variants (armmp and armmp-lpae for large-memory systems) that will support just about any new devices shipping, given updated drivers and device tree support. UEFI is now a thing for some ARMv7 platforms. Some people are looking into using UEFI by default, plus there's also the ongoing work by Alexander Graf on the UEFI shim layer in modern U-Boot. This allows for easier, consistent boot support (e.g. booting off USB automatically for installation). armel = armel is the longest-running of the three current Arm ports in Debian, starting in 2007 with Lenny. We've been debating the future of armel for some time in Debian now. It's typically still supported by most upstreams (Linux kernel, gcc, etc.), but we're more frequently finding issues with support for it in newer tools and languages and applications further up the stack. It's difficult (in some cases almost impossible) to run many larger applications like browsers on ARMv4t, as the upstream developers just don't support it any more. Based on this, I described a plan to drop armel from testing and *not* release it with Buster. This is *not* a done deal, and there's still scope for interested developers to step up and make sure that armel is supported in future. We'd been asking for that for the last couple of years and it had not happened. We'd been talking in the past few years about options for armel, including a partial armel port, dropping support for some packages that are difficult to port or not considered useful. While that might sound attractive as a possible way to go, nobody appears to have done any of the work needed to make it possible. In the background, almost in parallel there has been some discussion on the mailing list about this topic and it seems we now *do* have some developers wanting to keep armel alive. See the thread starting at [3] for more information, or if you'd like to help. I still have personal concerns about the viability of armel in the future, but I wish any other volunteers luck in keeping it alive. Whether we continue with armel or not, it's probably worthwhile for users to think about migrating to newer hardware if they care about it in the longer term. Buildds and hardware The existing build machines for armel and armhf are all still ARMv7 based, using donated Marvell Armada XP machines. They're working very well as build machines - they're quite fast and take a reasonable amount of memory. Scaleway [4] use similar hardware for Arm-based hosting. We also still have an older Freescale imx53 porter box for people on armhf who might need to debug NEON issues, as the Armada XP machines don't include it. arm64 gives us more options for sensible build machines: AMD Seattle, APM Mustang, Arm Junos. We're still looking at more and better hardware in the future, including potential offers from arm64 server hardware developers. We're wanting to move on to more and more arm64 hardware in the future for build machines, including for building armhf. Using real server hardware will help us, as the machines will then fit in a rack, with remote management options too. Unfortunately, you can't rely on these for building for armel. There are old deprecated instructions (mainly barriers and atomics) that are no longer supported in the ARMv8 hardware. There is optional kernel support to trap the exceptions here and emulate the i
