Hi! On Fri, May 29, 2020 at 07:03:48PM +0000, Will Springer wrote: > Hey all, a couple of us over in #talos-workstation on freenode have been > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been > designated for this
https://www.polyomino.org.uk/publications/2011/Power-Arch-32-bit-ABI-supp-1.0-Embedded.pdf > (unless you count the patchset from a decade ago [1]), so The original sysv PowerPC supplement http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf supports LE as well, and most powerpcle ports use that. But, the big-endian Linux ABI differs in quite a few places, and it of course makes a lot better sense if powerpcle-linux follows that. > it's pretty much uncharted territory as far as Linux is concerned. We want to > sync up with libc and the relevant kernel folks to establish the best path > forward. > > The practical application that drove these early developments (as you might > expect) is x86 emulation. The box86 project [2] implements a translation layer > for ia32 library calls to native architecture ones; this way, emulation > overhead is significantly reduced by relying on native libraries where > possible (libc, libGL, etc.) instead of emulating an entire x86 userspace. > box86 is primarily targeted at ARM, but it can be adapted to other > architectures—so long as they match ia32's 32-bit, little-endian nature. Hence > the need for a ppcle userland; modern POWER brought ppc64le as a supported > configuration, but without a 32-bit equivalent there is no option for a 32/64 > multilib environment, as seen with ppc/ppc64 and arm/aarch64. > > Surprisingly, beyond minor patching of gcc to get crosscompile going, What patches did you need? I regularly build >30 cross compilers (on both BE and LE hosts; I haven't used 32-bit hosts for a long time, but in the past those worked fine as well). I also cross-built powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64, from various x86). > bootstrapping the initial userland was not much of a problem. The work has > been done on top of the Void Linux PowerPC project [3], and much of that is > now present in its source package tree [4]. > > The first issue with running the userland came from the ppc32 signal handler > forcing BE in the MSR, causing any 32LE process receiving a signal (such as a > shell receiving SIGCHLD) to terminate with SIGILL. This was trivially > patched, Heh :-) > along with enabling the 32-bit vDSO on ppc64le kernels [5]. (Given that this > behavior has been in place since 2006, I don't think anyone has been using > the > kernel in this state to run ppcle userlands.) Almost no project that used 32-bit PowerPC in LE mode has sent patches to the upstreams. > The next problem concerns the ABI more directly. The failure mode was `file` > surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a > garbage value for `pos`, which didn't appear to be caused by anything in > `file`. Initially it seemed as though the 32-bit components of the arg were > getting swapped, and we made hacky fixes to glibc and musl to put them in the > "right order"; however, we weren't sure if that was the correct approach, or > if there were knock-on effects we didn't know about. So we found the relevant > compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where > there exists this comment: > > > /* > > * long long munging: > > * The 32 bit ABI passes long longs in an odd even register pair. > > */ > > It seems that the opposite is true in LE mode, and something is expecting long > longs to start on an even register. I realized this after I tried swapping hi/ > lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that > switches which syscalls use padding arguments depending on endianness, while > hopefully remaining tidy enough to be unobtrusive. (I took some liberties with > variable names/types so that the macro could be consistent.) The ABI says long longs are passed in the same order in registers as it would be in memory; so the high part and the low part are swapped between BE and LE. Which registers make up a pair is exactly the same between the two. (You can verify this with an existing powerpcle-* compiler, too; I did, and we implement it correctly as far as I can see). > This was enough to fix up the `file` bug. I'm no seasoned kernel hacker, > though, and there is still concern over the right way to approach this, > whether it should live in the kernel or libc, etc. Frankly, I don't know the > ABI structure enough to understand why the register padding has to be > different in this case, or what lower-level component is responsible for it. > For comparison, I had a look at the mips tree, since it's bi-endian and has a > similar 32/64 situation. There is a macro conditional upon endianness that is > responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead > of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of > that. > (It also simply swaps registers for LE, unlike what I did for ppc.) But you should :-) > Also worth noting is the one other outstanding bug, where the time-related > syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an > endian bug to me, and it doesn't affect standard syscalls (which is why if you > run `date` on musl it prints the correct time, unlike on glibc). The vDSO time > functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/ > gettimeofday.S), and I've never touched the stuff, so if anyone has a clue > I'm > all ears. > > Again, I'd appreciate feedback on the approach to take here, in order to > touch/special-case only the minimum necessary, while keeping the kernel/libc > folks happy. A huge factor in having good GCC support for powerpcle-linux (or anything else) is someone needs to regularly test it, and share test results with us (via gcc-testresults@). Hint hint hint :-) That way we know it is in good shape, know when we are regressing it, know there is interest in it. gl;hf, Segher
