On Thu, Oct 7, 2021 at 8:56 AM Alex Bennée <alex.ben...@linaro.org> wrote:
> Hi, > > I came across a use-case this week for ARM although this may be also > applicable to architectures where QEMU's emulation is ahead of the > hardware currently widely available - for example if you want to > exercise SVE code on AArch64. When the linux-user architecture is not > the same as the host architecture then binfmt_misc works perfectly fine. > > However in the case you are running same-on-same you can't use > binfmt_misc to redirect execution to using QEMU because any attempt to > trap native binaries will cause your userspace to hang as binfmt_misc > will be invoked to run the QEMU binary needed to run your application > and a deadlock ensues. > > There are some hacks you can apply at a local level like tweaking the > elf header of the binaries you want to run under emulation and adjusting > the binfmt_mask appropriately. This works but is messy and a faff to > set-up. > > An ideal setup would be would be for the kernel to catch a SIGILL from a > failing user space program and then to re-launch the process using QEMU > with the old processes maps and execution state so it could continue. > However I suspect there are enough moving parts to make this very > fragile (e.g. what happens to the results of library feature probing > code). So two approaches I can think of are: > 32-bit arm had an 'eabi' section in ELF binaries. There it would have been possible to look at that and make a decision before the binary starts executing to see whether it should just run, or fork the linux-user binary. It would take kernel changes, though. > Trap execve in QEMU linux-user > ------------------------------ > > We could add a flag to QEMU so at the point of execve it manually > invokes the new process with QEMU, passing on the flag to persist this > behaviour. > The bsd-user code differs a little from linux-user in that it looks at the binary being exec'd to determine what to do. It works OK, but isn't really for this situation (we use it to optimize our package builds with additional path processing for our mixed binary situation where we have native binaries execing emulated binaries that then exec native binaries again. It is a bit of a hack, though, and I'm not completely happy with it. Add path mask to binfmt_misc > ---------------------------- > > The other option would be to extend binfmt_misc to have a path mask so > it only applies it's alternative execution scheme to binaries in a > particular section of the file-system (or maybe some sort of pattern?). > > Are there any other approaches you could take? Which do you think has > the most merit? > In by-gone times, brandelf has bene used for situations where you wanted to run an ELF binary with one emulation that looks like another. But that's also kernel hacks and also touching the local binary. There's also the option of doing a VM86-like thing that allowed people to run 16-bit x86 binaries on 32-bit processors. There the system calls would SEGV and you'd decode them inline, execute the emulation and move the IP to execute the next instruction after the INT XX system call. You could create a loader that knows how to load load the binaries and catch SIGILL and then emulate the new instructions on the old processor, but that's somewhat different than how qemu user-mode works today. But knowing you'd need to do this is hard, potentially. But one could expand the kernel to load in SIGILL handlers on-demand for programs that do this, but that wouldn't work with old kernels and just feels weird... Warner
