> On Sat, 11 Mar 2017, ropers wrote:
> > Was 32-on-64 compatibility somehow easier to achieve on the Linux side?
> > Or did they just keep throwing code and more code at the problem because
> > they felt they really, really had to have this? It's that kind of idle
> > curiosity. If nobody's interested in explaining or hearing this
> > explained, then sorry for the noise.
>
> There's (at least) three parts to such compat for the kernel:
>
> 1) the low-level kernel<->userspace boundary handling.
> A 32-bit process has a different pmap, adding code and requiring the
> uvm layer to indirect or test which is in play for its operations. The
> syscall/trap interface is quite different too; amd64 is sooooo much
> nicer for syscall entry and %fs/%gs handling than i386. You don't
> want that shit in your nice 64bit kernel!
>
> 2) ABI mapping
> So you called some syscall and it has the result to copy out to
> userspace. Oh wait, which ABI is this process using, the 64bit ABI
> that matches the kernel's structure layouts or a completely different
> ABI that requires checking for overflow and then repacking structures
> to match the 32bit ABI? Yay, we get to think about that every time we
> do copyin/copyout! Let's see, 490 calls to those two functions in 119
> files (not counting sys/arch/*). OSes that support this tend to
> introduce an abstraction layer to reduce the number of those that are
> needed, but that's still cognitive load.
>
> 3) interprocess handling
> a) 64bit -> 32bit
> You weren't really planning on having gdb64 and gdb32, so at least
> ptrace() has to be capable to letting a 64bit process manipulate a
> 32bit process. Go look at FreeBSD's sys/kern/sys_process.c for
> example and examine the COMPAT_FREEBSD32 blocks. They've done it,
> obviously, but it's not simple, beautiful code. A comment like
>
> * This CPP subterfuge is to try and reduce the number of ifdefs in
> * the body of the code.
>
> is a warning that feature is costing you a chunk.
>
> b) 32bit -> 64bit
> Maybe you ban 32bit ptrace() of 64bit processes (most do IIRC), but
> you still want 32bit processes to be able to use sysctl(KERN_PROC)
> and see 64bit processes
>
>
> Implementations have jumped through all those hoops, but at what cost in
> complexity and security holes, and there *have* a been a bunch! A google
> search for "security hole in 32bit compat" immediately turned up this
> article:
> https://lwn.net/Articles/406466/
>
> Adding "FreeBSD" or "NetBSD" to the search turns up hits for them too.
> This stuff is *hard* and the benefit is...?
>
> Nope, don't want it.
Thanks Philip.
By the way the history of OS compat in *BSD is interesting.
Chris Torek built his LBL sparc code to the SunOS ABI. It was
incomplete. But enough to bring the system up.
I pulled his sparc code into NetBSD. At the time NetBSD was
transition and even extending the machine-independent system call API
in the not-yet-released 4.4BSD-lite2, rather than having every
architecture contain it's own table.
Torek's API wasn't good enough to handle structures which needed
modification on the way to/from kernel/userland, so I created the idea
of a zone on the stack where arguments could be tweaked before being
passed on.
Thus was born the first real compat layer. Compat HPUX was created
very soon, then compat for SCO Unix, some FreeBSD compat, and then
Linux compat which was far nastier because their socket system calls
had been added in a really adhoc fashion.
There's another problem Philip hasn't brought up.
4) The people who rely on the compat layers don't care enough to
maintain it. The people who work on the mainline system don't
care about the compat layers because they don't use them. The
cultures aren't aligned in the same direction. Compat layers
rot very quickly.
