Re: Can we drop upstream Linux x32 support?
On 14.12.18 22:16, Thomas Schöbel-Theuer wrote: Hi, > Currently, we have a few thousands of servers relying on 32bit ABIs in> some > thousands of VMs and/or containers of various types (LXC, OpenVZ,> etc). Similar w/ my clients, but in Industrie/Embedded/i4.0 world. We have thousands of devices in the field, that need support for at least a decade. Right now, I'm working on a new device, that - even having a 64bit cpu - will still run 32bit userland for various reasons (memory footprint is one of them). Even on the data center side, where we have *a lot* of containers (eg. container-based microservices on minimal distros like alpine or ptxdist - as well as many different, often highly specialized, legacy applications), 32bit userland indeed is a major factor. Dropping it would have huge economic consequences: * massive costs for re-deployment / migration of thousands of different applications * field roll of thousands of different embedded / industrial devices, as a major cpu arch * huge increase in memory consumption and io load in container- based microservice environments * loosing x86 for a huge portion of the embedded / small devices world I don't even dare to estimate the economic damage. (not even speaking about reputation loss of the Linux community) > Here is my private opinion, not speaking for 1&1: at some point the> future, > 32bit userspace support needs to be dropped anyway, somewhen in> future. This is inevitable in the very long term. Very, very, long term. Just the devices I'm coping w/ will have remaining lifetime of at least 10..15 years. That's a really long time in IT world. Nobody knows whether x86 at all will play a big role then. > 1) please release / declare a new LTS kernel, with upstream support for > at least 5 years (as usual). Currently, only 4.4 and 4.9 are marked as > LTS. Either mark another existing stable kernel as LTS, or a future one. That might be okay in enterprise world (even though, here I'm regularily stumbling across much older code in production), but in industrial world, the product lifetimes are much longer - 20+yrs years usualstandard.are pretty common. --mtx -- Enrico Weigelt, metux IT consult Free software and Linux embedded engineering i...@metux.net -- +49-151-27565287
Re: Can we drop upstream Linux x32 support?
On Wed, Jan 09, 2019 at 01:41:14PM +0100, Florian Weimer wrote: > * Thomas Schöbel-Theuer: > > > 2) please _announce_ _now_ that after the _next_ LTS kernel (whichever > > you want to declare as such), you will _afterwards_ drop the legacy > > 32bit support for 64 kernels (I am deliberately using "management > > speak" here). > > > > => result: the industry should have to fair chance to deal with such a > > roadmap. Yes, it will hurt some people, but they will have enough time > > for their migration projects. > > > > Example: I know that out of several millions of customers of > > webhosting, a very low number of them have some very old legacy 32bit > > software installed in their webspace. This cannot be supported > > forever. But the number of such cases is very small, and there just > > needs to be enough time for finding a solution for those few > > customers. > > > > 3) the next development kernel _after_ that LTS release can then > > immediately drop the 32bit support. Enterprise users should have > > enough time for planning, and for lots of internal projects > > modernizing their infrastructure. Usually, they will need to do this > > anyway in the long term. > > We've already phased out support for all 32-bit architectures except > i386 in our products, and i386 is obviously next. (We never supported > x32 in the first place.) > > It becomes increasingly difficult to build a 32-bit userspace that meets > backwards-compatibility needs. We want to use SSE2 (to avoid excess > precision for double) and avoid relying on stack realignment (for > compatibility with applications that use the old i386 ABI which did not > require stack realignment). We also have to build the distribution with > a full complement of hardening flags. This results in a combination of > flags that is poorly tested in upstream GCC. The i386 register file > isn't large enough to support all these features at the same time and > combine them with function arguments passed in registers (which some > programs enable manually via function attributes). > > So even if we keep the kernel interface, in the forseeable future, I > expect that it will be difficult to build a full, contemporary 32-bit > userspace on i386. I guess that's informative of how one company's distro process works, but it's not representative. Your customers are enterprise and big-server (probably mostly the former) oriented which is exactly the domain where 32-bit is of course irrelevant except for running legacy applications. Where it matters are embedded and other systems striving for resource efficiency. For what it's worth, 32-bit archs including i386 and many others are well-supported in Debian with no forseeable EOL I'm aware of, and most if not all of the musl-libc-based distros I'm familiar with support 32-bit archs including i386. I don't think waning relevance of 32-bit is a reasonable argument against x32 since x32's relevance is in exactly the places where 32-bit is already relevant and preferred for important reasons. Rich
Re: Can we drop upstream Linux x32 support?
* Thomas Schöbel-Theuer: > 2) please _announce_ _now_ that after the _next_ LTS kernel (whichever > you want to declare as such), you will _afterwards_ drop the legacy > 32bit support for 64 kernels (I am deliberately using "management > speak" here). > > => result: the industry should have to fair chance to deal with such a > roadmap. Yes, it will hurt some people, but they will have enough time > for their migration projects. > > Example: I know that out of several millions of customers of > webhosting, a very low number of them have some very old legacy 32bit > software installed in their webspace. This cannot be supported > forever. But the number of such cases is very small, and there just > needs to be enough time for finding a solution for those few > customers. > > 3) the next development kernel _after_ that LTS release can then > immediately drop the 32bit support. Enterprise users should have > enough time for planning, and for lots of internal projects > modernizing their infrastructure. Usually, they will need to do this > anyway in the long term. We've already phased out support for all 32-bit architectures except i386 in our products, and i386 is obviously next. (We never supported x32 in the first place.) It becomes increasingly difficult to build a 32-bit userspace that meets backwards-compatibility needs. We want to use SSE2 (to avoid excess precision for double) and avoid relying on stack realignment (for compatibility with applications that use the old i386 ABI which did not require stack realignment). We also have to build the distribution with a full complement of hardening flags. This results in a combination of flags that is poorly tested in upstream GCC. The i386 register file isn't large enough to support all these features at the same time and combine them with function arguments passed in registers (which some programs enable manually via function attributes). So even if we keep the kernel interface, in the forseeable future, I expect that it will be difficult to build a full, contemporary 32-bit userspace on i386. Thanks, Florian
Re: Can we drop upstream Linux x32 support?
> On Dec 14, 2018, at 11:41 PM, Thomas Schoebel-Theuer > wrote: > >> On 12/14/18 22:41, Thomas Schöbel-Theuer wrote: >>> On 12/14/18 22:24, Andy Lutomirski wrote: >>> >>> I'm talking about x32, which is a different beast. >>> >> >> So from my viewpoint the mentioned roadmap / timing requirements will remain >> the same, whatever you are dropping. >> >> Enterprise-critical use cases will probably need to be migrated to KVM/qemu >> together with their old kernel versions, anyway (because the original >> hardware will be no longer available in a few decades). >> > > Here is a systematic approach to the problem. > This is way off topic. Let’s not discuss it on this thread, please.
Re: Can we drop upstream Linux x32 support?
On 12/14/18 22:41, Thomas Schöbel-Theuer wrote: On 12/14/18 22:24, Andy Lutomirski wrote: I'm talking about x32, which is a different beast. So from my viewpoint the mentioned roadmap / timing requirements will remain the same, whatever you are dropping. Enterprise-critical use cases will probably need to be migrated to KVM/qemu together with their old kernel versions, anyway (because the original hardware will be no longer available in a few decades). Here is a systematic approach to the problem. AFAICS legacy 32bit userspace code (which exists in some notable masses) can be executed at least in the following ways: 1) natively on 32bit-capable hardware, under 32bit kernels. Besides legacy hardware, this also encompasses most current Intel / AMD 64bit hardware in 32bit compatibility mode. 2) under 64bit kernels, using the 32bit compat layer from practically any kernel version. 3) under KVM/qemu. When you just drop 1), users have a fair chance by migrating to any of the other two possibilities. As explained, a time frame of ~5 years should work for the vast majority. If you clearly explain the migration paths to your users (and to the press), I think it will be acceptable. [side note: I know of a single legacy instance which is now ~20 years old, but makes a revenue of several millions per month. These guys have large quantities of legacy hardware in stock. And they have enough money to hire a downstream maintainer in case of emergency.] Fatal problems would only arise if you would drop all three possibilities in the very long term. In ~100 years, possibility 3) should be sufficient for handling use cases like preservation of historic documents. The latter is roughly equivalent to running binary-only MSDOS, Windows NT, and similar, even in 100 years, and even non-natively under future hardware architectures.
Re: Can we drop upstream Linux x32 support?
Andy Lutomirski dixit: >x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's >IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't Yeah, I was looking at this from userspace PoV, as I said I’m not a Linux kernel programmer. In BSD we have register_t which is probably the equivalent to your __kernel_long_t? Maybe removing the “long” from the name helps. But yes, x32 is just a (second to i386) ILP32 userspace API in an amd64 kernel. This does imply mapping on the userspace (x32) to kernel (amd64) boundary and back. I would have thought full struct member mapping, as dalias described, to be the most robust. >something similar to work using the normal x86_64 syscalls. And I'm But those would require the longer structs etc. and therefore lose all the benefits of x32… bye, //mirabilos -- „Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund, mksh auf jedem System zu installieren.“ -- XTaran auf der OpenRheinRuhr, ganz begeistert (EN: “[…]uhr.gz is a reason to install mksh on every system.”)
Re: Can we drop upstream Linux x32 support?
On 12/14/18 22:24, Andy Lutomirski wrote: I should clarify again: I am not suggesting that we drop 32-bit support in the forseeable future. (Well, I might eventually suggest that we drop support for 32-bit *hardware* at some point, but not for 32-bit compat software.) Linux's compat code is quite robust and is even fairly maintainable. I'm talking about x32, which is a different beast. OK, this does not really make a big difference for large-scale enterprise users. Some day in the very distant future any legacy system will disappear, eventually. So from my viewpoint the mentioned roadmap / timing requirements will remain the same, whatever you are dropping. Enterprise-critical use cases will probably need to be migrated to KVM/qemu together with their old kernel versions, anyway (because the original hardware will be no longer available in a few decades).
Re: Can we drop upstream Linux x32 support?
On Fri, Dec 14, 2018 at 12:13 PM Linus Torvalds wrote: > > On Fri, Dec 14, 2018 at 10:58 AM Andy Lutomirski wrote: > > > > Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long? > > It *needs* to be long long, since the headers are used for builds in > user mode using ILP32. > > Since __kernel_long_t is a 64-bit (the _kernel_ is not ILP32), you > need to use "long long" when building in ILP32. > > Obviously, it could be something like > >#ifdef __KERNEL__ > typedef long __kernel_long_t; >#else > typedef long long __kernel_long_t; >#endif > > or similar to make it more obvious what's going on. > > Or we could encourage all the uapi header files to always just use > explicit sizing like __u64, but some of the structures really end up > being "kernel long size" for sad historical reasons. Not lovely, but > there we are.. > This is probably water under the bridge, but I disagree with you here. Or rather, I agree with you in principle but I really don't like the way it turned out. For legacy uapi structs (and probably some new ones too, sigh), as a practical matter, user code is going to shove them at the C compiler, and the C compiler is going to interpret them in the usual way, and either we need a usermode translation layer or the kernel needs to deal with the result. It's a nice thought that, by convincing an x32 compiler that __kernel_long_t is 64 bits, we end up with the x32 struct being compatible with the native Linux struct, but it only works for structs where the only ABI-dependent type is long. But the real structs in uapi aren't all like this. We have struct iovec: struct iovec { void __user *iov_base; /* BSD uses caddr_t (1003.1g requires void *) */ __kernel_size_t iov_len; /* Must be size_t (1003.1g) */ }; Whoops, this one looks the same on x32 and i386, but neither one of them match x86_64, and, just randomly grepping around a bit, I see: struct snd_hwdep_dsp_image { unsigned int index; /* W: DSP index */ unsigned char name[64]; /* W: ID (e.g. file name) */ unsigned char __user *image;/* W: binary image */ size_t length; /* W: size of image in bytes */ unsigned long driver_data; /* W: driver-specific data */ }; struct __sysctl_args { int __user *name; int nlen; void __user *oldval; size_t __user *oldlenp; void __user *newval; size_t newlen; unsigned long __unused[4]; }; If these had been switched from "unsigned long" to __kernel_ulong_t, they would have had three different layouts on i386, x32, and x86_64. So now we have a situation where, if we were to make x32 work 100%, the whole kernel would need to recognize that there are three possible ABIs, not two. And this sucks. So I think it would have been a better choice to let long be 32-bit on x32 and to therefore make x32 match the x86_32 "compat" layout as much as possible. Sure, this would make x32 be more of a second-class citizen, but I think it would have worked better, had fewer bugs, and been more maintainable. --Andy
Re: Can we drop upstream Linux x32 support?
On Fri, Dec 14, 2018 at 1:22 PM Thomas Schöbel-Theuer wrote: > > On 12/11/18 02:23, Andy Lutomirski wrote: > > I'm seriously considering sending a patch to remove x32 support from > > upstream Linux. > > I am downstream maintainer of several self-patched kernels at 1&1 Ionos. > The kernels are rolled out to several tenthousands of production servers > running in several datacenters and in multiple continents. > > Currently, we have a few thousands of servers relying on 32bit ABIs in > some thousands of VMs and/or containers of various types (LXC, OpenVZ, etc). I should clarify again: I am not suggesting that we drop 32-bit support in the forseeable future. (Well, I might eventually suggest that we drop support for 32-bit *hardware* at some point, but not for 32-bit compat software.) Linux's compat code is quite robust and is even fairly maintainable. I'm talking about x32, which is a different beast.
Re: Can we drop upstream Linux x32 support?
On 12/11/18 02:23, Andy Lutomirski wrote: I'm seriously considering sending a patch to remove x32 support from upstream Linux. I am downstream maintainer of several self-patched kernels at 1&1 Ionos. The kernels are rolled out to several tenthousands of production servers running in several datacenters and in multiple continents. Currently, we have a few thousands of servers relying on 32bit ABIs in some thousands of VMs and/or containers of various types (LXC, OpenVZ, etc). Here is my private opinion, not speaking for 1&1: at some point the future, 32bit userspace support needs to be dropped anyway, somewhen in future. This is inevitable in the very long term. Thus the discussion should be about _timing_ / _roadmaps_, but not about the fact as such. My suggestion: 1) please release / declare a new LTS kernel, with upstream support for at least 5 years (as usual). Currently, only 4.4 and 4.9 are marked as LTS. Either mark another existing stable kernel as LTS, or a future one. 2) please _announce_ _now_ that after the _next_ LTS kernel (whichever you want to declare as such), you will _afterwards_ drop the legacy 32bit support for 64 kernels (I am deliberately using "management speak" here). => result: the industry should have to fair chance to deal with such a roadmap. Yes, it will hurt some people, but they will have enough time for their migration projects. Example: I know that out of several millions of customers of webhosting, a very low number of them have some very old legacy 32bit software installed in their webspace. This cannot be supported forever. But the number of such cases is very small, and there just needs to be enough time for finding a solution for those few customers. 3) the next development kernel _after_ that LTS release can then immediately drop the 32bit support. Enterprise users should have enough time for planning, and for lots of internal projects modernizing their infrastructure. Usually, they will need to do this anyway in the long term. A roadmap should be _reliable_ for planning, and there should be no "unexpected surprises". That's the most important requirements. Notice: 5 years is what I know will very likely work for 1&1 Ionos. I cannot speak for other large-scale enterprise users, but I think most of them should be able to deal with suchalike intervals in a clear roadmap. Addendum / side note: I know of cases where _critical_ / sometimes even proprietary 32bit enterprise software needs to run for several _decades_. Please don't drop KVM/qemu support for 32bit guests using _old_ (unchanged) 32bit kernels, which are just happen to run on 64bit hypervisors. But AFAICS that was not the intention of this initiative. Just keep the latter in mind as another (independent) requirement. Cheers, Thomas
Re: Can we drop upstream Linux x32 support?
On Fri, Dec 14, 2018 at 10:58 AM Andy Lutomirski wrote: > > Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long? It *needs* to be long long, since the headers are used for builds in user mode using ILP32. Since __kernel_long_t is a 64-bit (the _kernel_ is not ILP32), you need to use "long long" when building in ILP32. Obviously, it could be something like #ifdef __KERNEL__ typedef long __kernel_long_t; #else typedef long long __kernel_long_t; #endif or similar to make it more obvious what's going on. Or we could encourage all the uapi header files to always just use explicit sizing like __u64, but some of the structures really end up being "kernel long size" for sad historical reasons. Not lovely, but there we are.. Linus
Re: Can we drop upstream Linux x32 support?
My impression is it was mostly a desire to reuse existing x86_64 system calls as much as possible without modification or additional compat layer work. The 64-bit time requirement seems to have come from an lkml discussion, which has quite a bit of interesting background about x32: https://lkml.org/lkml/2011/8/26/415 https://lkml.org/lkml/2011/8/26/453 On Fri, Dec 14, 2018 at 2:05 PM Andy Lutomirski wrote: > > > > > On Dec 14, 2018, at 8:55 AM, Rich Felker wrote: > > > >> On Fri, Dec 14, 2018 at 05:38:33PM +0100, Florian Weimer wrote: > >> * Rich Felker: > >> > >>> This is all useless (and wrong since tv_nsec is required to have type > >>> long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug > >>> in glibc's x32). > >> > >> We should be able to fix standards if they prove unworkable in practice. > >> In my opinion, if standards require complex solutions where an obvious > >> and simple solution exists, then standards are wrong. > > > > The requirement doesn't mandate complex solutions. There's nothing > > complex about tv_nsec being long. long is the smallest type that C > > guarantees to be large enough to store the range of values, which is > > forever fixed and can't grow (because the definition of "nano" prefix > > is fixed :). The type has been long ever since the structure was > > introduced, and its being long means that there's lots of (correct!) > > code using %ld (e.g. ".%.9ld" to format results as a decimal without > > using floating point approximations) to print it. There might also be > > code taking pointers to it to pass to functions, etc. > > > > The only reason a "complex" need arises is that Linux did something > > horribly wrong here, ignoring the specified type, when introducing an > > obscure subarch that almost nobody uses. This kind of mistake is > > becoming a theme in Linux (see also: msghdr). Application authors > > should not have to pay the price for fixing this by retrofitting yet > > another silly type like "snseconds_t" or something into programs to > > accommodate the mistakes of x32. > > > > > > Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long? > I assume that this is where this bug, and most of the other bugs, came from. > > This may be silly, but the kernel could plausibly add a x32v2 where long is > genuinely 32-bit, and then maybe we could drop the old x32 at some point. > From all the discussion so far, it seems like there really is some demand for > ILP32, but it’s still not really clear that preserving compatibility with > existing x32 binaries in the long run is critical.
Re: Can we drop upstream Linux x32 support?
> On Dec 14, 2018, at 8:55 AM, Rich Felker wrote: > >> On Fri, Dec 14, 2018 at 05:38:33PM +0100, Florian Weimer wrote: >> * Rich Felker: >> >>> This is all useless (and wrong since tv_nsec is required to have type >>> long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug >>> in glibc's x32). >> >> We should be able to fix standards if they prove unworkable in practice. >> In my opinion, if standards require complex solutions where an obvious >> and simple solution exists, then standards are wrong. > > The requirement doesn't mandate complex solutions. There's nothing > complex about tv_nsec being long. long is the smallest type that C > guarantees to be large enough to store the range of values, which is > forever fixed and can't grow (because the definition of "nano" prefix > is fixed :). The type has been long ever since the structure was > introduced, and its being long means that there's lots of (correct!) > code using %ld (e.g. ".%.9ld" to format results as a decimal without > using floating point approximations) to print it. There might also be > code taking pointers to it to pass to functions, etc. > > The only reason a "complex" need arises is that Linux did something > horribly wrong here, ignoring the specified type, when introducing an > obscure subarch that almost nobody uses. This kind of mistake is > becoming a theme in Linux (see also: msghdr). Application authors > should not have to pay the price for fixing this by retrofitting yet > another silly type like "snseconds_t" or something into programs to > accommodate the mistakes of x32. > > Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long? I assume that this is where this bug, and most of the other bugs, came from. This may be silly, but the kernel could plausibly add a x32v2 where long is genuinely 32-bit, and then maybe we could drop the old x32 at some point. From all the discussion so far, it seems like there really is some demand for ILP32, but it’s still not really clear that preserving compatibility with existing x32 binaries in the long run is critical.
Re: Can we drop upstream Linux x32 support?
On Fri, Dec 14, 2018 at 05:38:33PM +0100, Florian Weimer wrote: > * Rich Felker: > > > This is all useless (and wrong since tv_nsec is required to have type > > long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug > > in glibc's x32). > > We should be able to fix standards if they prove unworkable in practice. > In my opinion, if standards require complex solutions where an obvious > and simple solution exists, then standards are wrong. The requirement doesn't mandate complex solutions. There's nothing complex about tv_nsec being long. long is the smallest type that C guarantees to be large enough to store the range of values, which is forever fixed and can't grow (because the definition of "nano" prefix is fixed :). The type has been long ever since the structure was introduced, and its being long means that there's lots of (correct!) code using %ld (e.g. ".%.9ld" to format results as a decimal without using floating point approximations) to print it. There might also be code taking pointers to it to pass to functions, etc. The only reason a "complex" need arises is that Linux did something horribly wrong here, ignoring the specified type, when introducing an obscure subarch that almost nobody uses. This kind of mistake is becoming a theme in Linux (see also: msghdr). Application authors should not have to pay the price for fixing this by retrofitting yet another silly type like "snseconds_t" or something into programs to accommodate the mistakes of x32. Rich
Re: Can we drop upstream Linux x32 support?
* Rich Felker: > This is all useless (and wrong since tv_nsec is required to have type > long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug > in glibc's x32). We should be able to fix standards if they prove unworkable in practice. In my opinion, if standards require complex solutions where an obvious and simple solution exists, then standards are wrong. Thanks, Florian
Re: Can we drop upstream Linux x32 support?
On 14/12/2018 17:17, Rich Felker wrote: > On Fri, Dec 14, 2018 at 03:13:10PM +0100, Bernd Petrovitsch wrote: [..] >> FWIW I have >> snip >> #if defined __x86_64__ >> # if defined __ILP32__ // x32 >> # define PRI_time_t "lld" // for time_t >> # define PRI_nsec_t "lld" // for tv_nsec in struct timespec >> # else // x86_64 >> # define PRI_time_t "ld"// for time_t >> # define PRI_nsec_t "ld"// for tv_nsec in struct timespec >> # endif >> #else// i[3-6]68 >> # define PRI_time_t "ld"// for time_t >> # define PRI_nsec_t "ld"// for tv_nsec in struct timespec >> #endif >> snip >> in my userspace code for printf() and friends - I don't know how libc's >> react to such a patch (and I don't care for the name of the macros as >> long it's obviously clear for which type they are). >> I assume/fear we won't get additional modifiers into the relevant >> standards for libc types (as they are far more like pid_t, uid_t etc.). >> And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't >> look appealing to me to "solve" such issues. > > This is all useless (and wrong since tv_nsec is required to have type > long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug Thanks. OK, I didn't know that - and 32bit is enough to represent 1E9 (as a second won't have more nanosecs). Hmm, can we fix that in the x32 world? Sry, I'm not the expert on glibc vs ABI va syscall interface vs breakage there though. > in glibc's x32). Just do: > > printf("%jd", (intmax_t)t); > > Saving 2 or 3 insns (for sign or zero extension) around a call to > printf is not going to make any measurable difference to performance Until someone comes up with hardware with ASIC support for 1k bit int's and (ab)uses intmax_t for that. SCNR > or any significant difference to size, and it's immeasurably more > readable than the awful PRI* macros and the > adjacent-string-concatenation they rely on. One gets used to the PRI_* macros over time (and there no calculated format strings in my world) - and type casts are not better in my eyes ... MfG, Bernd -- "I dislike type abstraction if it has no real reason. And saving on typing is not a good reason - if your typing speed is the main issue when you're coding, you're doing something seriously wrong." - Linus Torvalds pEpkey.asc Description: application/pgp-keys
Re: Can we drop upstream Linux x32 support?
On Fri, Dec 14, 2018 at 03:13:10PM +0100, Bernd Petrovitsch wrote: > On 13/12/2018 17:02, Rich Felker wrote: > > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: > >> I can't say anything about the syscall interface. However, what I do know > >> is that the weird combination of a 32-bit userland with a 64-bit kernel > >> interface is sometimes causing issues. For example, application code > >> usually > >> expects things like time_t to be 32-bit on a 32-bit system. However, this > > IMHO this just historically grown (as in "it has been forever that way" > - it sounds way better in Viennese dialect though;-). > > >> isn't the case for x32 which is why code fails to build. > > > > I don't see any basis for this claim about expecting time_t to be > > 32-bit. I've encountered some programs that "implicitly assume" this > > by virtue of assuming they can cast time_t to long to print it, or > > similar. IIRC this was an issue in busybox at one point; I'm not sure > > if it's been fixed. But any software that runs on non-Linux unices has > > long been corrected. If not, 2038 is sufficiently close that catching > > and correcting any such remaining bugs is more useful than covering > > them up and making the broken code work as expected. > > Yup, unconditionally providing 64bit > time_t/timespec/timeval/...-equivalents with libc and syscall support > also for 32bit architectures (and deprecating all 32bit versions) should > be the way to go. > > FWIW I have > snip > #if defined __x86_64__ > # if defined __ILP32__// x32 > # define PRI_time_t "lld" // for time_t > # define PRI_nsec_t "lld" // for tv_nsec in struct timespec > # else// x86_64 > # define PRI_time_t "ld"// for time_t > # define PRI_nsec_t "ld"// for tv_nsec in struct timespec > # endif > #else // i[3-6]68 > # define PRI_time_t "ld"// for time_t > # define PRI_nsec_t "ld"// for tv_nsec in struct timespec > #endif > snip > in my userspace code for printf() and friends - I don't know how libc's > react to such a patch (and I don't care for the name of the macros as > long it's obviously clear for which type they are). > I assume/fear we won't get additional modifiers into the relevant > standards for libc types (as they are far more like pid_t, uid_t etc.). > And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't > look appealing to me to "solve" such issues. This is all useless (and wrong since tv_nsec is required to have type long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug in glibc's x32). Just do: printf("%jd", (intmax_t)t); Saving 2 or 3 insns (for sign or zero extension) around a call to printf is not going to make any measurable difference to performance or any significant difference to size, and it's immeasurably more readable than the awful PRI* macros and the adjacent-string-concatenation they rely on. Rich
Re: Can we drop upstream Linux x32 support?
On Fri, Dec 14, 2018 at 12:42:34PM +0100, Florian Weimer wrote: > * Rich Felker: > > > On Thu, Dec 13, 2018 at 05:04:59PM +0100, Florian Weimer wrote: > >> * Rich Felker: > >> > >> >> If the compiler can handle the zeroing, that would be great, though not > >> >> sure how (some __attribute__((zero)) which generates a type constructor > >> >> for such structure; it kind of departs from what the C language offers). > >> > > >> > The compiler fundamentally can't. At the very least it would require > >> > effective type tracking, which requires shadow memory and is even more > >> > controversial than -fstrict-aliasing (because in a sense it's a > >> > stronger version thereof). > >> > >> It's possible to do it with the right types. See _Bool on 32-bit Darwin > >> PowerPC for an example, which is four bytes instead of the usual one. > >> > >> Similarly, we could have integer types with trap representations. > >> Whether it is a good idea is a different matter, but the amount of > >> compiler magic required is actually limited. > > > > If you do this you just have LP64 with value range restricted to > > 32-bit. > > You have to a type different from long int for the relevant struct > fields. This type would have zero padding. Just upthread (Message-ID: <20181212165237.gt23...@brightrain.aerifal.cx>) I explained why this does not work: >>> If on the other hand you tried to make just some pointers "wide >>> pointers", you'd also be completely breaking the specified API >>> contracts of standard interfaces. For example in struct iovec's >>> iov_base, &foo->iov_base is no longer a valid pointer to an object of >>> type void* that you can pass to interfaces expecting void**. Sloppy >>> misunderstandings like what you're making now are exactly why x32 is >>> already broken and buggy (&foo->tv_nsec already has wrong type for >>> struct timespec foo). Rich
Re: Can we drop upstream Linux x32 support?
Am Freitag, 14. Dezember 2018, 15:38:53 CET schrieb David Laight: > From: Richard Weinberger > > Sent: 13 December 2018 09:05 > > On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton wrote: > > > > > > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: > > > ... > > > > I can't say anything about the syscall interface. However, what I do > > > > know > > > > is that the weird combination of a 32-bit userland with a 64-bit kernel > > > > interface is sometimes causing issues. For example, application code > > > > usually > > > > expects things like time_t to be 32-bit on a 32-bit system. However, > > > > this > > > > isn't the case for x32 which is why code fails to build. > > > > > > OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have > > > had for four or five years at this point. > > > > They can also do flag-day changes and break existing applications, Linux > > not. > > Not true at all. > The binary compatibility in NetBSD is probably better than that in Linux > and goes back a long way. > > For the time_t changes new system calls numbers were assigned where needed. > The system headers and libc were updated so that recompiled code would > use the new system calls. > > The only real advantage that NetBSD has is that its libc (and standard > utilities) are released with the kernel making it much easier to get > applications to use the new features. > > This was also done a very long time ago when file offsets were extended > to 64 bits. > > Some of the system calls have quite a few 'compatibility' versions. > As well as the ones for emulations of other operating systems. > It has been possible to run copies of firefox compiled for Linux > under NetBSD. I stand corrected, I was under the impression that NetBSD went the same path as OpenBSD did. Thanks for pointing this out. Thanks, //richard
RE: Can we drop upstream Linux x32 support?
From: Richard Weinberger > Sent: 13 December 2018 09:05 > On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton wrote: > > > > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: > > ... > > > I can't say anything about the syscall interface. However, what I do know > > > is that the weird combination of a 32-bit userland with a 64-bit kernel > > > interface is sometimes causing issues. For example, application code > > > usually > > > expects things like time_t to be 32-bit on a 32-bit system. However, this > > > isn't the case for x32 which is why code fails to build. > > > > OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have > > had for four or five years at this point. > > They can also do flag-day changes and break existing applications, Linux not. Not true at all. The binary compatibility in NetBSD is probably better than that in Linux and goes back a long way. For the time_t changes new system calls numbers were assigned where needed. The system headers and libc were updated so that recompiled code would use the new system calls. The only real advantage that NetBSD has is that its libc (and standard utilities) are released with the kernel making it much easier to get applications to use the new features. This was also done a very long time ago when file offsets were extended to 64 bits. Some of the system calls have quite a few 'compatibility' versions. As well as the ones for emulations of other operating systems. It has been possible to run copies of firefox compiled for Linux under NetBSD. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: Can we drop upstream Linux x32 support?
On 13/12/2018 17:02, Rich Felker wrote: > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: >> I can't say anything about the syscall interface. However, what I do know >> is that the weird combination of a 32-bit userland with a 64-bit kernel >> interface is sometimes causing issues. For example, application code usually >> expects things like time_t to be 32-bit on a 32-bit system. However, this IMHO this just historically grown (as in "it has been forever that way" - it sounds way better in Viennese dialect though;-). >> isn't the case for x32 which is why code fails to build. > > I don't see any basis for this claim about expecting time_t to be > 32-bit. I've encountered some programs that "implicitly assume" this > by virtue of assuming they can cast time_t to long to print it, or > similar. IIRC this was an issue in busybox at one point; I'm not sure > if it's been fixed. But any software that runs on non-Linux unices has > long been corrected. If not, 2038 is sufficiently close that catching > and correcting any such remaining bugs is more useful than covering > them up and making the broken code work as expected. Yup, unconditionally providing 64bit time_t/timespec/timeval/...-equivalents with libc and syscall support also for 32bit architectures (and deprecating all 32bit versions) should be the way to go. FWIW I have snip #if defined __x86_64__ # if defined __ILP32__ // x32 # define PRI_time_t"lld" // for time_t # define PRI_nsec_t"lld" // for tv_nsec in struct timespec # else // x86_64 # define PRI_time_t"ld"// for time_t # define PRI_nsec_t"ld"// for tv_nsec in struct timespec # endif #else // i[3-6]68 # define PRI_time_t "ld"// for time_t # define PRI_nsec_t "ld"// for tv_nsec in struct timespec #endif snip in my userspace code for printf() and friends - I don't know how libc's react to such a patch (and I don't care for the name of the macros as long it's obviously clear for which type they are). I assume/fear we won't get additional modifiers into the relevant standards for libc types (as they are far more like pid_t, uid_t etc.). And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't look appealing to me to "solve" such issues. MfG, Bernd -- "I dislike type abstraction if it has no real reason. And saving on typing is not a good reason - if your typing speed is the main issue when you're coding, you're doing something seriously wrong." - Linus Torvalds pEpkey.asc Description: application/pgp-keys
Re: Can we drop upstream Linux x32 support?
* Rich Felker: > On Thu, Dec 13, 2018 at 05:04:59PM +0100, Florian Weimer wrote: >> * Rich Felker: >> >> >> If the compiler can handle the zeroing, that would be great, though not >> >> sure how (some __attribute__((zero)) which generates a type constructor >> >> for such structure; it kind of departs from what the C language offers). >> > >> > The compiler fundamentally can't. At the very least it would require >> > effective type tracking, which requires shadow memory and is even more >> > controversial than -fstrict-aliasing (because in a sense it's a >> > stronger version thereof). >> >> It's possible to do it with the right types. See _Bool on 32-bit Darwin >> PowerPC for an example, which is four bytes instead of the usual one. >> >> Similarly, we could have integer types with trap representations. >> Whether it is a good idea is a different matter, but the amount of >> compiler magic required is actually limited. > > If you do this you just have LP64 with value range restricted to > 32-bit. You have to a type different from long int for the relevant struct fields. This type would have zero padding. Florian
Re: Can we drop upstream Linux x32 support?
Hi. I reported several significant speedups gained by the x32 ABI for Scheme-based applications (compiled via a gcc backend), e.g. https://lists.nongnu.org/archive/html/chicken-users/2015-09/msg0.html https://www.mail-archive.com/bigloo@inria.fr/msg00427.html https://forum.manjaro.org/t/why-is-there-no-kernel-support-for-the-x32-abi-config-x86-x32/55156/9 As a tech startup in cognitive search and NLP (https://www.sempria.de/), the benefits of the x32 ABI in terms of saved compute time and costs are vital for us. It's a pity that x32 has not gained the momentum that it deserves. Two suggestions to change this: 1. One could extend a prominent page like https://sites.google.com/site/x32abi/ with positive and negative results for applications w.r.t. the x32 ABI. I am willing to rewrite and update the results that I reported over the years. 2. The name x32 is too short, not search-engine friendly, and ambiguous (most people think it's about i386 ...). Is not there a better name? Ciao Sven
Re: Can we drop upstream Linux x32 support?
On Thu, 13 Dec 2018, Florian Weimer wrote: > Similarly, we could have integer types with trap representations. C++2a will require two's complement representation for integer types, with no trap representations (other than for bool, where only 0 and 1 are valid representations). It seems very plausible that C will follow. -- Joseph S. Myers jos...@codesourcery.com
Re: Can we drop upstream Linux x32 support?
On Thu, Dec 13, 2018 at 05:04:59PM +0100, Florian Weimer wrote: > * Rich Felker: > > >> If the compiler can handle the zeroing, that would be great, though not > >> sure how (some __attribute__((zero)) which generates a type constructor > >> for such structure; it kind of departs from what the C language offers). > > > > The compiler fundamentally can't. At the very least it would require > > effective type tracking, which requires shadow memory and is even more > > controversial than -fstrict-aliasing (because in a sense it's a > > stronger version thereof). > > It's possible to do it with the right types. See _Bool on 32-bit Darwin > PowerPC for an example, which is four bytes instead of the usual one. > > Similarly, we could have integer types with trap representations. > Whether it is a good idea is a different matter, but the amount of > compiler magic required is actually limited. If you do this you just have LP64 with value range restricted to 32-bit. This does not accomplish anything useful since it uses just as much memory as normal LP64 but you don't get anything beneficial out of it. Rich
Re: Can we drop upstream Linux x32 support?
I will also complain. I use the 32 bit x86 environment regularly for testing. I have found several bugs and worked with upstream developers to create fixes. The developers never said anything like "this is a waste of time because it was found on 32 bit". Instead, they were universally supportive. As an example, Thomas Gleixner said that the problem we found would only show up on older hardware, and he affirmed that it was a bug that certainly needed to be fixed ("NOHZ: Check for nohz active instead of nohz enabled"). I actually think this is an _opportunity_ to engage aspiring kernel programmers. Give them some guidance and have them actively test and do simple maintenance of 32 bit x86. Yes, I am prepared to take the lead on this if it is something you want to pursue. - Matthew Whitehead
Re: Can we drop upstream Linux x32 support?
On Mon, 2018-12-10 at 17:40 -0800, Linus Torvalds wrote: > On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski > wrote: > > I'm seriously considering sending a patch to remove x32 support > > from > > upstream Linux. Here are some problems with it: > > I talked to Arnd (I think - we were talking about all the crazy > ABI's, > but maybe it was with somebody else) about exactly this in Edinburgh. > > Apparently the main real use case is for extreme benchmarking. It's > the only use-case where the complexity of maintaining a whole > development environment and distro is worth it, it seems. Apparently > a > number of Spec submissions have been done with the x32 model. > > I'm not opposed to trying to sunset the support, but let's see who > complains.. FWIW, the Yocto Project has working x32 support which it regularly builds, tests and has as a supported configuration. We did help identify and fix many of the places userspace didn't support it. As with open source in general, its hard to know how many people are actually using it. I don't think there are huge numbers but we do occasionally get bug reports. Cheers, Richard
Re: Can we drop upstream Linux x32 support?
* Rich Felker: >> If the compiler can handle the zeroing, that would be great, though not >> sure how (some __attribute__((zero)) which generates a type constructor >> for such structure; it kind of departs from what the C language offers). > > The compiler fundamentally can't. At the very least it would require > effective type tracking, which requires shadow memory and is even more > controversial than -fstrict-aliasing (because in a sense it's a > stronger version thereof). It's possible to do it with the right types. See _Bool on 32-bit Darwin PowerPC for an example, which is four bytes instead of the usual one. Similarly, we could have integer types with trap representations. Whether it is a good idea is a different matter, but the amount of compiler magic required is actually limited. Thanks, Florian
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: > I can't say anything about the syscall interface. However, what I do know > is that the weird combination of a 32-bit userland with a 64-bit kernel > interface is sometimes causing issues. For example, application code usually > expects things like time_t to be 32-bit on a 32-bit system. However, this > isn't the case for x32 which is why code fails to build. I don't see any basis for this claim about expecting time_t to be 32-bit. I've encountered some programs that "implicitly assume" this by virtue of assuming they can cast time_t to long to print it, or similar. IIRC this was an issue in busybox at one point; I'm not sure if it's been fixed. But any software that runs on non-Linux unices has long been corrected. If not, 2038 is sufficiently close that catching and correcting any such remaining bugs is more useful than covering them up and making the broken code work as expected. Rich
Re: Can we drop upstream Linux x32 support?
On Thu, Dec 13, 2018 at 12:40:25PM +, Catalin Marinas wrote: > On Wed, Dec 12, 2018 at 10:03:30AM -0800, Andy Lutomirski wrote: > > On Wed, Dec 12, 2018 at 8:52 AM Rich Felker wrote: > > > On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote: > > > > I'm proposing another alternative. Given that x32 already proves that > > > > the user bitness model doesn't have to match the kernel model (in x32, > > > > user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm > > > > proposing extending this to just make the kernel ABI be LP64. So > > > > __kernel_size_t would be 64-bit and pointers in kernel data structures > > > > would be 64-bit. In other words, most or all of the kernel ABI would > > > > just match x86_64. > > > > > > > > As far as I can tell, the only thing that really needs unusual > > > > toolchain features here is that C doesn't have an extra-wide pointer > > > > type. The kernel headers would need a way to say "this pointer is > > > > still logically a pointer, and user code may assume that it's 32 bits, > > > > but it has 8-byte alignment." > > > > > > None of this works on the userspace/C side, nor should any attempt be > > > made to make it work. Types fundamentally cannot have alignments > > > larger than their size. If you want to make the alignment of some > > > pointers 8, you have to make their size 8, and then you just have LP64 > > > again if you did it for all pointers. > > > > > > If on the other hand you tried to make just some pointers "wide > > > pointers", you'd also be completely breaking the specified API > > > contracts of standard interfaces. For example in struct iovec's > > > iov_base, &foo->iov_base is no longer a valid pointer to an object of > > > type void* that you can pass to interfaces expecting void**. Sloppy > > > misunderstandings like what you're making now are exactly why x32 is > > > already broken and buggy (&foo->tv_nsec already has wrong type for > > > struct timespec foo). > > > > I don't think it's quite that broken. For the struct iovec example, > > we currently have: > > > >struct iovec { > >void *iov_base;/* Starting address */ > >size_t iov_len; /* Number of bytes to transfer */ > >}; > > > > we could have, instead: (pardon any whitespace damage) > > > >struct iovec { > >void *iov_base;/* Starting address */ > >uint32_t __pad0; > >size_t iov_len; /* Number of bytes to transfer */ > >uint32_t __pad1; > >} __attribute__((aligned(8)); > > > > or the same thing but where iov_len is uint64_t. A pointer to > > iov_base still works exactly as expected. Something would need to be > > done to ensure that the padding is all zeroed, which might be a real > > problem. > > We looked at this approach briefly for arm64/ILP32 and zeroing the pads > was the biggest problem. User programs would not explicitly zero the pad > and I'm not sure the compiler would be any smarter. This means it's the > kernel's responsibility to zero the pad (around get_user, > copy_from_user), so it doesn't actually simplify the kernel side of the > syscall interface. > > If the data flow goes the other way (kernel to user), this approach > works fine. > > > No one wants to actually type all the macro gunk into the headers to > > make this work, but this type of transformation is what I have in mind > > when the compiler is asked to handle the headers. Or there could > > potentially be a tool that automatically consumes the uapi headers and > > spits out modified headers like this. > > If the compiler can handle the zeroing, that would be great, though not > sure how (some __attribute__((zero)) which generates a type constructor > for such structure; it kind of departs from what the C language offers). The compiler fundamentally can't. At the very least it would require effective type tracking, which requires shadow memory and is even more controversial than -fstrict-aliasing (because in a sense it's a stronger version thereof). But even effective type tracking would not help, since you can have things like: struct iovec *iov = malloc(sizeof *iov); scanf("%p %zu", &iov->iov_base, &iov->iov_len); where no store to the object via the struct type ever happens and the only stores that do happen are invisible across translation unit boundaries. (Ignore that scanf here is awful; it's just a canonical example of a function that would store the members via pointers to them.) The kernel-side approach could work if the kernel had some markup for fields that need to be zero- or sign-extended when copied from user in a 32-bit process and applied them at copy time. That could also fix the existing tv_nsec issue. I'm not sure how difficult/costly it would be though. Rich
Re: Can we drop upstream Linux x32 support?
On Thu, Dec 13, 2018 at 9:39 AM Olof Johansson wrote: > > On Tue, Dec 11, 2018 at 9:40 AM Linus Torvalds > wrote: > > > > On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski wrote: > > > > > > I'm seriously considering sending a patch to remove x32 support from > > > upstream Linux. Here are some problems with it: > > > > I talked to Arnd (I think - we were talking about all the crazy ABI's, > > but maybe it was with somebody else) about exactly this in Edinburgh. > > > > Apparently the main real use case is for extreme benchmarking. It's > > the only use-case where the complexity of maintaining a whole > > development environment and distro is worth it, it seems. Apparently a > > number of Spec submissions have been done with the x32 model. > > > > I'm not opposed to trying to sunset the support, but let's see who > > complains.. > > I'm just a single user. I do rely on it though, FWIW. > > I hadn't finished my benchmarking in Edinburgh, but for my new machine > that does kernel builds 24/7, I ended up going with x32 userspace (in > a container). > > Main reason is that it's a free ~10% improvement in runtime over > 64-bit. I.e. GCC-as-a-workload is quite a bit faster as x32, > supposedly mostly due to smaller D cache footprints (I ran out of > cycles to tinker with back and forth perf data collection and settled > down on just running it). > > Running classic 32-bit (i386? i686? whatever it's called) is about > half as good. I.e. even then I get a ~5% performance win. Less than > x32, but still better than 64-bit userspace. > > > -Olof I'm familiar with two embedded Linux systems using x32 ABI for the following reasons: - Significant performance benefits over i386 ABI. - Smaller memory footprint than x86_64 (pointer-heavy data structures) - Large legacy software base with many ILP32 assumptions, much smaller effort to move to x32 than x86_64. Some examples of (relatively minor) problems encountered with x32: - Time-related data type mismatch in socket options (fixed) https://patchwork.ozlabs.org/patch/904254/ - Userspace overrun with select() (patch proposed) https://patchwork.kernel.org/patch/10245677/ - glibc get_phys_pages() doesn't work correctly with x32 (assumes that struct sysinfo fields are not wider than "long"). So, one small vote for keeping x32 with the hope that support for it can continue to be improved... - Lance
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 9:40 AM Linus Torvalds wrote: > > On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski wrote: > > > > I'm seriously considering sending a patch to remove x32 support from > > upstream Linux. Here are some problems with it: > > I talked to Arnd (I think - we were talking about all the crazy ABI's, > but maybe it was with somebody else) about exactly this in Edinburgh. > > Apparently the main real use case is for extreme benchmarking. It's > the only use-case where the complexity of maintaining a whole > development environment and distro is worth it, it seems. Apparently a > number of Spec submissions have been done with the x32 model. > > I'm not opposed to trying to sunset the support, but let's see who complains.. I'm just a single user. I do rely on it though, FWIW. I hadn't finished my benchmarking in Edinburgh, but for my new machine that does kernel builds 24/7, I ended up going with x32 userspace (in a container). Main reason is that it's a free ~10% improvement in runtime over 64-bit. I.e. GCC-as-a-workload is quite a bit faster as x32, supposedly mostly due to smaller D cache footprints (I ran out of cycles to tinker with back and forth perf data collection and settled down on just running it). Running classic 32-bit (i386? i686? whatever it's called) is about half as good. I.e. even then I get a ~5% performance win. Less than x32, but still better than 64-bit userspace. -Olof
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 10:03:30AM -0800, Andy Lutomirski wrote: > On Wed, Dec 12, 2018 at 8:52 AM Rich Felker wrote: > > On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote: > > > I'm proposing another alternative. Given that x32 already proves that > > > the user bitness model doesn't have to match the kernel model (in x32, > > > user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm > > > proposing extending this to just make the kernel ABI be LP64. So > > > __kernel_size_t would be 64-bit and pointers in kernel data structures > > > would be 64-bit. In other words, most or all of the kernel ABI would > > > just match x86_64. > > > > > > As far as I can tell, the only thing that really needs unusual > > > toolchain features here is that C doesn't have an extra-wide pointer > > > type. The kernel headers would need a way to say "this pointer is > > > still logically a pointer, and user code may assume that it's 32 bits, > > > but it has 8-byte alignment." > > > > None of this works on the userspace/C side, nor should any attempt be > > made to make it work. Types fundamentally cannot have alignments > > larger than their size. If you want to make the alignment of some > > pointers 8, you have to make their size 8, and then you just have LP64 > > again if you did it for all pointers. > > > > If on the other hand you tried to make just some pointers "wide > > pointers", you'd also be completely breaking the specified API > > contracts of standard interfaces. For example in struct iovec's > > iov_base, &foo->iov_base is no longer a valid pointer to an object of > > type void* that you can pass to interfaces expecting void**. Sloppy > > misunderstandings like what you're making now are exactly why x32 is > > already broken and buggy (&foo->tv_nsec already has wrong type for > > struct timespec foo). > > I don't think it's quite that broken. For the struct iovec example, > we currently have: > >struct iovec { >void *iov_base;/* Starting address */ >size_t iov_len; /* Number of bytes to transfer */ >}; > > we could have, instead: (pardon any whitespace damage) > >struct iovec { >void *iov_base;/* Starting address */ >uint32_t __pad0; >size_t iov_len; /* Number of bytes to transfer */ >uint32_t __pad1; >} __attribute__((aligned(8)); > > or the same thing but where iov_len is uint64_t. A pointer to > iov_base still works exactly as expected. Something would need to be > done to ensure that the padding is all zeroed, which might be a real > problem. We looked at this approach briefly for arm64/ILP32 and zeroing the pads was the biggest problem. User programs would not explicitly zero the pad and I'm not sure the compiler would be any smarter. This means it's the kernel's responsibility to zero the pad (around get_user, copy_from_user), so it doesn't actually simplify the kernel side of the syscall interface. If the data flow goes the other way (kernel to user), this approach works fine. > No one wants to actually type all the macro gunk into the headers to > make this work, but this type of transformation is what I have in mind > when the compiler is asked to handle the headers. Or there could > potentially be a tool that automatically consumes the uapi headers and > spits out modified headers like this. If the compiler can handle the zeroing, that would be great, though not sure how (some __attribute__((zero)) which generates a type constructor for such structure; it kind of departs from what the C language offers). > Realistically, I think a much better model would be to use true ILP32 > code, where all the memory layouts in the uapi match i386. The conclusion we came to on arm64 was that an ILP32 ABI should not really be any different from a _new_ 32-bit architecture ABI. It differs from arm32 a bit (different syscall numbers, off_t is 64-bit, sigcontext) but not significantly as it is still able to use the majority of the compat_sys_* wrappers. -- Catalin
Re: Can we drop upstream Linux x32 support?
On Thu, Dec 13, 2018 at 10:05:14AM +0100, Richard Weinberger wrote: > On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton wrote: > > > > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: > > ... > > > I can't say anything about the syscall interface. However, what I do know > > > is that the weird combination of a 32-bit userland with a 64-bit kernel > > > interface is sometimes causing issues. For example, application code > > > usually > > > expects things like time_t to be 32-bit on a 32-bit system. However, this > > > isn't the case for x32 which is why code fails to build. > > > > OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have > > had for four or five years at this point. > > They can also do flag-day changes and break existing applications, Linux not. Sure, but the point is that most widely-used software has probably by now come in to contact with systems where time_t is bigger than long. - Kevin
Re: Can we drop upstream Linux x32 support?
On Thu, Dec 13, 2018 at 10:37:31AM +0100, Sven Hartrumpf wrote: > Will the proposed patch ("only") remove the possibility to build x32 kernels > or will it make impossible to compile and run any x32 binaries? There's no such thing as x32 kernels. It's an ABI atop amd64 kernels; the kernel is always 64-bit -- short pointers apply only to userspace and syscalls. It could be interesting to block regular amd64 syscalls and leave x32 only -- for "security through rarity", but the kernel itself would still look the same. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Ivan was a worldly man: born in St. Petersburg, raised in ⢿⡄⠘⠷⠚⠋⠀ Petrograd, lived most of his life in Leningrad, then returned ⠈⠳⣄ to the city of his birth to die.
Re: Can we drop upstream Linux x32 support?
Will the proposed patch ("only") remove the possibility to build x32 kernels or will it make impossible to compile and run any x32 binaries? Sven
Re: Can we drop upstream Linux x32 support?
On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton wrote: > > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: > ... > > I can't say anything about the syscall interface. However, what I do know > > is that the weird combination of a 32-bit userland with a 64-bit kernel > > interface is sometimes causing issues. For example, application code usually > > expects things like time_t to be 32-bit on a 32-bit system. However, this > > isn't the case for x32 which is why code fails to build. > > OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have > had for four or five years at this point. They can also do flag-day changes and break existing applications, Linux not. -- Thanks, //richard
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote: ... > I can't say anything about the syscall interface. However, what I do know > is that the weird combination of a 32-bit userland with a 64-bit kernel > interface is sometimes causing issues. For example, application code usually > expects things like time_t to be 32-bit on a 32-bit system. However, this > isn't the case for x32 which is why code fails to build. OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have had for four or five years at this point. - Kevin
Re: Can we drop upstream Linux x32 support?
My main reason on staying at Pentium 4 is that it does not contain Intel ME backdoor (it came starting with Core 2 Duo) ср, 12 дек. 2018 г. в 22:12, Andy Lutomirski : > > On Wed, Dec 12, 2018 at 10:50 AM Ivan Ivanov wrote: > > > > Please don't drop x86 support from the Linux kernel ! You may not > > realize it, but there are plenty of people who care about x86 systems. > > I have a good old Pentium 4 based computer which is 32-bit, using it > > for some weird Linux experiments + retro gaming, and if I can't run > > new Linux kernels there - it would be SAD > > Just to clarify: no one is proposing to drop 32-bit hardware support > or normal x86 compatibility support from new kernels. > > That being said, you should seriously consider replacing that P4. > Unless you live somewhere with *very* cheap power or you barely use > it, replacing it will pay for itself quite quickly.
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 10:50 AM Ivan Ivanov wrote: > > Please don't drop x86 support from the Linux kernel ! You may not > realize it, but there are plenty of people who care about x86 systems. > I have a good old Pentium 4 based computer which is 32-bit, using it > for some weird Linux experiments + retro gaming, and if I can't run > new Linux kernels there - it would be SAD Just to clarify: no one is proposing to drop 32-bit hardware support or normal x86 compatibility support from new kernels. That being said, you should seriously consider replacing that P4. Unless you live somewhere with *very* cheap power or you barely use it, replacing it will pay for itself quite quickly.
Re: Can we drop upstream Linux x32 support?
Please don't drop x86 support from the Linux kernel ! You may not realize it, but there are plenty of people who care about x86 systems. I have a good old Pentium 4 based computer which is 32-bit, using it for some weird Linux experiments + retro gaming, and if I can't run new Linux kernels there - it would be SAD ср, 12 дек. 2018 г. в 21:15, Joseph Myers : > > On Wed, 12 Dec 2018, Arnd Bergmann wrote: > > > > MIPS had o32, n32, n64 since like forever. > > > > o32 and n32 are practically the same, the only difference on the > > syscall ABI that I can see are the actual syscall numbers, and > > the 'struct sigcontext' definition. > > And for syscalls that have 64-bit arguments, n32 generally passes those in > a single register (like n64), not pairs of registers (like o32). But, > yes, userspace structure layout for n32 is generally much closer to o32 > than to n64. > > -- > Joseph S. Myers > jos...@codesourcery.com
Re: Can we drop upstream Linux x32 support?
On Wed, 12 Dec 2018, Arnd Bergmann wrote: > > MIPS had o32, n32, n64 since like forever. > > o32 and n32 are practically the same, the only difference on the > syscall ABI that I can see are the actual syscall numbers, and > the 'struct sigcontext' definition. And for syscalls that have 64-bit arguments, n32 generally passes those in a single register (like n64), not pairs of registers (like o32). But, yes, userspace structure layout for n32 is generally much closer to o32 than to n64. -- Joseph S. Myers jos...@codesourcery.com
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 8:52 AM Rich Felker wrote: > > On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote: > > > On Dec 11, 2018, at 6:33 PM, Thorsten Glaser wrote: > > > > > > Andy Lutomirski dixit: > > > > > > > > > > >> IMO the real right solution would be to push the whole problem to > > >> userspace: get an ILP32 system working with almost or entirely LP64 > > > > > > Is this a reflex of Linux kernel developers? ;-) > > > > > > I doubt that userspace is the right place for this, remember > > > the recent glibc vs. syscalls debate. It would also need to > > > multiply across various libcs. > > > > > >> How hard would it be to have __attribute__((ilp64)), with an optional > > >> warning if any embedded structs are not ilp64? This plus a wrapper to > > > > > > You mean LP64. Impossible, because LP64 vs. ILP32 is not the only > > > difference between amd64 and x32. > > > > I mean LP64. And I'm not suggesting that ILP32 is the only difference > > between x32 and x86_64, nor am I suggesting that a technique like this > > would implement x32 -- I'm suggesting it would implement something > > better than x32. > > > > The kernel, as a practical matter, supports two ABIs on 64-bit builds: > > LP64 and ILP32. ILP32 is what the kernel calls "compat". ("compat" > > comes with other baggage -- it generally has a 32-bit signal frame, > > syscall arguments are mostly limited to 32 bits, etc.) Allowing a > > user program that runs in 64-bit mode to issue compat syscalls is not > > really a big deal. x86_64 has allowed this forever using int $0x80 -- > > it's just slow. Adding a faster mechanism would be straightforward. > > As I understand it, the arm64 ilp32 proposal involves using a genuine > > ILP32 model for user code, so the syscalls will all (except for signal > > handling) go through the compat path. > > > > x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's > > IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't > > really support this. The only good things I can think of about it are > > that (a) it proves that somewhat odd ABIs are possible, at least in > > principle, and (b) three users have come out of the woodwork to say > > that they use it. > > > > I'm proposing another alternative. Given that x32 already proves that > > the user bitness model doesn't have to match the kernel model (in x32, > > user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm > > proposing extending this to just make the kernel ABI be LP64. So > > __kernel_size_t would be 64-bit and pointers in kernel data structures > > would be 64-bit. In other words, most or all of the kernel ABI would > > just match x86_64. > > > > As far as I can tell, the only thing that really needs unusual > > toolchain features here is that C doesn't have an extra-wide pointer > > type. The kernel headers would need a way to say "this pointer is > > still logically a pointer, and user code may assume that it's 32 bits, > > but it has 8-byte alignment." > > None of this works on the userspace/C side, nor should any attempt be > made to make it work. Types fundamentally cannot have alignments > larger than their size. If you want to make the alignment of some > pointers 8, you have to make their size 8, and then you just have LP64 > again if you did it for all pointers. > > If on the other hand you tried to make just some pointers "wide > pointers", you'd also be completely breaking the specified API > contracts of standard interfaces. For example in struct iovec's > iov_base, &foo->iov_base is no longer a valid pointer to an object of > type void* that you can pass to interfaces expecting void**. Sloppy > misunderstandings like what you're making now are exactly why x32 is > already broken and buggy (&foo->tv_nsec already has wrong type for > struct timespec foo). I don't think it's quite that broken. For the struct iovec example, we currently have: struct iovec { void *iov_base;/* Starting address */ size_t iov_len; /* Number of bytes to transfer */ }; we could have, instead: (pardon any whitespace damage) struct iovec { void *iov_base;/* Starting address */ uint32_t __pad0; size_t iov_len; /* Number of bytes to transfer */ uint32_t __pad1; } __attribute__((aligned(8)); or the same thing but where iov_len is uint64_t. A pointer to iov_base still works exactly as expected. Something would need to be done to ensure that the padding is all zeroed, which might be a real problem. No one wants to actually type all the macro gunk into the headers to make this work, but this type of transformation is what I have in mind when the compiler is asked to handle the headers. Or there could potentially be a tool that automatically consumes the uapi headers and spits out modified headers like this. Realistically, I think a much better model wo
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote: > > On Dec 11, 2018, at 6:33 PM, Thorsten Glaser wrote: > > > > Andy Lutomirski dixit: > > > > > > >> IMO the real right solution would be to push the whole problem to > >> userspace: get an ILP32 system working with almost or entirely LP64 > > > > Is this a reflex of Linux kernel developers? ;-) > > > > I doubt that userspace is the right place for this, remember > > the recent glibc vs. syscalls debate. It would also need to > > multiply across various libcs. > > > >> How hard would it be to have __attribute__((ilp64)), with an optional > >> warning if any embedded structs are not ilp64? This plus a wrapper to > > > > You mean LP64. Impossible, because LP64 vs. ILP32 is not the only > > difference between amd64 and x32. > > I mean LP64. And I'm not suggesting that ILP32 is the only difference > between x32 and x86_64, nor am I suggesting that a technique like this > would implement x32 -- I'm suggesting it would implement something > better than x32. > > The kernel, as a practical matter, supports two ABIs on 64-bit builds: > LP64 and ILP32. ILP32 is what the kernel calls "compat". ("compat" > comes with other baggage -- it generally has a 32-bit signal frame, > syscall arguments are mostly limited to 32 bits, etc.) Allowing a > user program that runs in 64-bit mode to issue compat syscalls is not > really a big deal. x86_64 has allowed this forever using int $0x80 -- > it's just slow. Adding a faster mechanism would be straightforward. > As I understand it, the arm64 ilp32 proposal involves using a genuine > ILP32 model for user code, so the syscalls will all (except for signal > handling) go through the compat path. > > x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's > IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't > really support this. The only good things I can think of about it are > that (a) it proves that somewhat odd ABIs are possible, at least in > principle, and (b) three users have come out of the woodwork to say > that they use it. > > I'm proposing another alternative. Given that x32 already proves that > the user bitness model doesn't have to match the kernel model (in x32, > user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm > proposing extending this to just make the kernel ABI be LP64. So > __kernel_size_t would be 64-bit and pointers in kernel data structures > would be 64-bit. In other words, most or all of the kernel ABI would > just match x86_64. > > As far as I can tell, the only thing that really needs unusual > toolchain features here is that C doesn't have an extra-wide pointer > type. The kernel headers would need a way to say "this pointer is > still logically a pointer, and user code may assume that it's 32 bits, > but it has 8-byte alignment." None of this works on the userspace/C side, nor should any attempt be made to make it work. Types fundamentally cannot have alignments larger than their size. If you want to make the alignment of some pointers 8, you have to make their size 8, and then you just have LP64 again if you did it for all pointers. If on the other hand you tried to make just some pointers "wide pointers", you'd also be completely breaking the specified API contracts of standard interfaces. For example in struct iovec's iov_base, &foo->iov_base is no longer a valid pointer to an object of type void* that you can pass to interfaces expecting void**. Sloppy misunderstandings like what you're making now are exactly why x32 is already broken and buggy (&foo->tv_nsec already has wrong type for struct timespec foo). If you wanted to do this and have it work, libc would have to completely wrap and convert all the types from userspace structures with the right types matching the specified API contracts to kernel-interface structures that can be passed to the LP64 syscalls. This could be done, but it's a big task, and I don't think anyone wants to do it, especially if it would be yet another new thing that doesn't exactly match the already-existing x32. The cost-benefit ratio is just too high. > There's an added benefit to my proposal over x32. With my proposal, > an ILP32 program could plausibly call into a true 64-bit C library > using a straightforward FFI translation. Unless it's a thin, "pure" library that doesn't need anything from libc, or needs sufficiently little that it could be satisfied by some shims, this would necessarily require having two libcs in the same process, which is not going to work. Rich
Re: Can we drop upstream Linux x32 support?
> On Dec 11, 2018, at 6:33 PM, Thorsten Glaser wrote: > > Andy Lutomirski dixit: > > >> IMO the real right solution would be to push the whole problem to >> userspace: get an ILP32 system working with almost or entirely LP64 > > Is this a reflex of Linux kernel developers? ;-) > > I doubt that userspace is the right place for this, remember > the recent glibc vs. syscalls debate. It would also need to > multiply across various libcs. > >> How hard would it be to have __attribute__((ilp64)), with an optional >> warning if any embedded structs are not ilp64? This plus a wrapper to > > You mean LP64. Impossible, because LP64 vs. ILP32 is not the only > difference between amd64 and x32. I mean LP64. And I'm not suggesting that ILP32 is the only difference between x32 and x86_64, nor am I suggesting that a technique like this would implement x32 -- I'm suggesting it would implement something better than x32. The kernel, as a practical matter, supports two ABIs on 64-bit builds: LP64 and ILP32. ILP32 is what the kernel calls "compat". ("compat" comes with other baggage -- it generally has a 32-bit signal frame, syscall arguments are mostly limited to 32 bits, etc.) Allowing a user program that runs in 64-bit mode to issue compat syscalls is not really a big deal. x86_64 has allowed this forever using int $0x80 -- it's just slow. Adding a faster mechanism would be straightforward. As I understand it, the arm64 ilp32 proposal involves using a genuine ILP32 model for user code, so the syscalls will all (except for signal handling) go through the compat path. x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't really support this. The only good things I can think of about it are that (a) it proves that somewhat odd ABIs are possible, at least in principle, and (b) three users have come out of the woodwork to say that they use it. I'm proposing another alternative. Given that x32 already proves that the user bitness model doesn't have to match the kernel model (in x32, user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm proposing extending this to just make the kernel ABI be LP64. So __kernel_size_t would be 64-bit and pointers in kernel data structures would be 64-bit. In other words, most or all of the kernel ABI would just match x86_64. As far as I can tell, the only thing that really needs unusual toolchain features here is that C doesn't have an extra-wide pointer type. The kernel headers would need a way to say "this pointer is still logically a pointer, and user code may assume that it's 32 bits, but it has 8-byte alignment." So I guess I'm proposing that people who really like x32 and are willing to invest effort it in instead invest some effort into getting something similar to work using the normal x86_64 syscalls. And I'm hinting to the ARM folks on the thread that they consider this approach for arm64. There's an added benefit to my proposal over x32. With my proposal, an ILP32 program could plausibly call into a true 64-bit C library using a straightforward FFI translation. > > bye, > //mirabilos > -- > I believe no one can invent an algorithm. One just happens to hit upon it > when God enlightens him. Or only God invents algorithms, we merely copy them. > If you don't believe in God, just consider God as Nature if you won't deny > existence.-- Coywolf Qi Hunt
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 02:46:00PM +, Steven Newbury wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > On Wed, 2018-12-12 at 09:01 -0500, Rich Felker wrote: > > On Wed, Dec 12, 2018 at 09:12:34AM +, Steven Newbury wrote: > > > -BEGIN PGP SIGNED MESSAGE- > > > Hash: SHA256 > > > > > > First off I'd like to request: Please don't break my userspace! > > > > > > I have a number of systems running with x32-abi as native. They > > > work > > > well, I've no want or desire to upgrade their memory or CPUs to > > > make > > > keep them working as well as they do now. Let alone the hassle of > > > having to redeploy with a completely different ABI. > > > > > > > > > I've been working on getting as much userspace as I can working > > > with > > > x32 since it first became somewhat usable, I've sent patches > > > upstream > > > and pushed to encourage projects to write portable code. Sometimes > > > that has meant just little tweaks to build systems, or sometimes > > > disabling asm where I consider it isn't worth the effort of making > > > it > > > work. For other projects I've converted the asm or even in some > > > cases > > > got the JIT working, mozjs17 for example. > > > > > > So I'm both a user and a developer. > > > > > > Breaking support for x32 would be really bad for me, but if I'm the > > > only one using it I suppose I can't really justify it being kept on > > > that basis. I know CERN has sucessfully experimented with it over > > > the > > > years though, so I wouldn't be surprised if there are other users > > > hiding out there. > > > > > > I can't help but wonder if it wouldn't make more sense to drop x86 > > > support from long mode than x32. AMD64 x86 support was always > > > intended > > > as a compatibility feature, I very much doubt there are more > > > people > > > out there using an x86 native userspace on a 64 bit kernel than > > > there > > > > I am. The only reason I'm using a 64-bit kernel with it is to get the > > full 4GB address space for userspace rather than just 3GB. I suspect > > there are more users like me than like you. > > > You may well be right, I lack any data either way. I just find it hard > to believe I'm, what, one of two users of x32? > > > Unlike x32, i386 is actually widely supported and works, and achieves > > most of the benefits of x32, but with marginally worse performance > > due > x32 works, and is widely supported by the fact that the vast majority > of free software is just a compile away. Now, there are holes, I've > been trying to get Chromium/qtwebengine ported for the last couple of > weeks. > > The performance isn't marginally worse, it's much worse, unless the > code in question is hand optimized SIMD or is otherwise not really > standard "IA32 ISA" anyway. When the baseline improvement is roughly 2x (because you can double N in make -jN), I consider 10-20% "marginal". Obviously there's a perspective where 10-20% is huge. Have you actually observed x32 gcc (since gcc is the example I'm using here, where it makes a big difference) running that much faster than i386 gcc, though? What I meant by "widely supported" is that pretty much any kernel that's shipped can run i386 binaries. On the other hand, at least as I understand it,lots of distros have x32 disabled because it's "not used" or because there were a couple historic vulns exposed by having it enabled and they just decided it wasn't worth it. > > performance sensitive it's the right choice for shipping static > > binaries, especially if they may be memory-hungry, since it works on > > pretty much any box. > > > When it comes to power usage and latency any code that runs often is > peformance senstitive. I can't argue about shipping i386 static > binaries, they'll work on pretty much any *x86* box, I agree. OK, so we're on the same page there. > > > are native x32 users. x86 support could be implemented via KVM > > > and/or > > > qemu-user. There is no reason to keep the extra complexity in the > > > kernel for what is effectively an emulation layer anyway. > > > > qemu-user is a non-solution. Not only does it result in being much > > slower and using more resources than just running a 64-bit userspace > > (defeating the purpose); it's also incapable of correctly emulating > > lots of corner cases. I've never been able to get it to work with > > thread cancellation, and doing so is fundamentally hard because it > > would require heavy emulation of signals rather than passing them > > through. > > > What's the purpose? Running IA32 binaries is for legacy. You don't > run i386/IA32 binaries for the performance. You use i386 as x32 was > intended, that's succifient for you. Great. I get the benefits I want > from an x32 userspace with amd64 binaries and libraries where it makes > sense as I understand does Thorsten. Are you saying this wrong, > broken(?), and should be removed? No, I'm not really taking a position here except that i386 (IA3
Re: Can we drop upstream Linux x32 support?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Wed, 2018-12-12 at 09:01 -0500, Rich Felker wrote: > On Wed, Dec 12, 2018 at 09:12:34AM +, Steven Newbury wrote: > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > First off I'd like to request: Please don't break my userspace! > > > > I have a number of systems running with x32-abi as native. They > > work > > well, I've no want or desire to upgrade their memory or CPUs to > > make > > keep them working as well as they do now. Let alone the hassle of > > having to redeploy with a completely different ABI. > > > > > > I've been working on getting as much userspace as I can working > > with > > x32 since it first became somewhat usable, I've sent patches > > upstream > > and pushed to encourage projects to write portable code. Sometimes > > that has meant just little tweaks to build systems, or sometimes > > disabling asm where I consider it isn't worth the effort of making > > it > > work. For other projects I've converted the asm or even in some > > cases > > got the JIT working, mozjs17 for example. > > > > So I'm both a user and a developer. > > > > Breaking support for x32 would be really bad for me, but if I'm the > > only one using it I suppose I can't really justify it being kept on > > that basis. I know CERN has sucessfully experimented with it over > > the > > years though, so I wouldn't be surprised if there are other users > > hiding out there. > > > > I can't help but wonder if it wouldn't make more sense to drop x86 > > support from long mode than x32. AMD64 x86 support was always > > intended > > as a compatibility feature, I very much doubt there are more > > people > > out there using an x86 native userspace on a 64 bit kernel than > > there > > I am. The only reason I'm using a 64-bit kernel with it is to get the > full 4GB address space for userspace rather than just 3GB. I suspect > there are more users like me than like you. > You may well be right, I lack any data either way. I just find it hard to believe I'm, what, one of two users of x32? > Unlike x32, i386 is actually widely supported and works, and achieves > most of the benefits of x32, but with marginally worse performance > due x32 works, and is widely supported by the fact that the vast majority of free software is just a compile away. Now, there are holes, I've been trying to get Chromium/qtwebengine ported for the last couple of weeks. The performance isn't marginally worse, it's much worse, unless the code in question is hand optimized SIMD or is otherwise not really standard "IA32 ISA" anyway. > to register scarcity, lack of native 64-bit arithmetic, bad function > call ABI, and bad PC-relative addressing. For applications that > aren't Exactly, much worse. > performance sensitive it's the right choice for shipping static > binaries, especially if they may be memory-hungry, since it works on > pretty much any box. > When it comes to power usage and latency any code that runs often is peformance senstitive. I can't argue about shipping i386 static binaries, they'll work on pretty much any *x86* box, I agree. > > are native x32 users. x86 support could be implemented via KVM > > and/or > > qemu-user. There is no reason to keep the extra complexity in the > > kernel for what is effectively an emulation layer anyway. > > qemu-user is a non-solution. Not only does it result in being much > slower and using more resources than just running a 64-bit userspace > (defeating the purpose); it's also incapable of correctly emulating > lots of corner cases. I've never been able to get it to work with > thread cancellation, and doing so is fundamentally hard because it > would require heavy emulation of signals rather than passing them > through. > What's the purpose? Running IA32 binaries is for legacy. You don't run i386/IA32 binaries for the performance. You use i386 as x32 was intended, that's succifient for you. Great. I get the benefits I want from an x32 userspace with amd64 binaries and libraries where it makes sense as I understand does Thorsten. Are you saying this wrong, broken(?), and should be removed? My point about qemu-user is that there is an alternative (non-)solution to legacy support in long mode, in addition to simply running an i386 kernel with or without virtualisattion. If that's insufficient for your use case of running an i386 userspace on an amd64 kernel that's fair enough, and from your point of view is a *really* good reason for not dropping i386 binary support... I feel the same way about x32! ;-) -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEEWa1B4K0Hk12RDstE+lAa6BzrmeAFAlwRHykACgkQ+lAa6Bzr meAZegf/WnQN8rOekchuSDbohDlsnW114/E2cEpnyfMAR4T6pNbzjoPronTfYO+Y yOWlQVgkImuJyjkupqKNa4FfQgCqEVav+p79G5SEeS+42kL5GyNlIgnYEjEjU2/i hKZxpk9YNSgTVX2rMZiRJ9UoLmk8cVz7DhaRhlSvH0ZYeJD8vZYzBLaZvs9Z9Qps xlh8S8sJA5v2vXHGg/iRMxmYT3nKfaH63cY9hCcpvxrcoIGV7UH5a4IDJ6ISCPzO U8KEfllu7a6slHMLOzw52TVH4jk1XtmzfjNKQdAHP0lArbvNRI+JNLO
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 09:12:34AM +, Steven Newbury wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > First off I'd like to request: Please don't break my userspace! > > I have a number of systems running with x32-abi as native. They work > well, I've no want or desire to upgrade their memory or CPUs to make > keep them working as well as they do now. Let alone the hassle of > having to redeploy with a completely different ABI. > > > I've been working on getting as much userspace as I can working with > x32 since it first became somewhat usable, I've sent patches upstream > and pushed to encourage projects to write portable code. Sometimes > that has meant just little tweaks to build systems, or sometimes > disabling asm where I consider it isn't worth the effort of making it > work. For other projects I've converted the asm or even in some cases > got the JIT working, mozjs17 for example. > > So I'm both a user and a developer. > > Breaking support for x32 would be really bad for me, but if I'm the > only one using it I suppose I can't really justify it being kept on > that basis. I know CERN has sucessfully experimented with it over the > years though, so I wouldn't be surprised if there are other users > hiding out there. > > I can't help but wonder if it wouldn't make more sense to drop x86 > support from long mode than x32. AMD64 x86 support was always intended > as a compatibility feature, I very much doubt there are more people > out there using an x86 native userspace on a 64 bit kernel than there I am. The only reason I'm using a 64-bit kernel with it is to get the full 4GB address space for userspace rather than just 3GB. I suspect there are more users like me than like you. Unlike x32, i386 is actually widely supported and works, and achieves most of the benefits of x32, but with marginally worse performance due to register scarcity, lack of native 64-bit arithmetic, bad function call ABI, and bad PC-relative addressing. For applications that aren't performance sensitive it's the right choice for shipping static binaries, especially if they may be memory-hungry, since it works on pretty much any box. > are native x32 users. x86 support could be implemented via KVM and/or > qemu-user. There is no reason to keep the extra complexity in the > kernel for what is effectively an emulation layer anyway. qemu-user is a non-solution. Not only does it result in being much slower and using more resources than just running a 64-bit userspace (defeating the purpose); it's also incapable of correctly emulating lots of corner cases. I've never been able to get it to work with thread cancellation, and doing so is fundamentally hard because it would require heavy emulation of signals rather than passing them through. Rich
Re: Can we drop upstream Linux x32 support?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Wed, 2018-12-12 at 10:48 +, Thorsten Glaser wrote: > Steven Newbury dixit: > > >I can't help but wonder if it wouldn't make more sense to drop x86 > >support from long mode than x32. AMD64 x86 support was always > intended > > Do you mean i386? > > x86 = { i386, x32, amd64 } > Yes, sorry to be unclear. I mean the "IA32 ISA". > No, please don’t. I use i386 as “companion architecture” to x32, > only the kernel and actually memory-hungry things (qemu) are > amd64 binaries on my system, and this works very well. > Well, if you have amd64 qemu anyway, why can't you use binfmt_misc with qemu-user to simulate i386? I'm just saying that makes more sense to me than dropping x32. -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEEWa1B4K0Hk12RDstE+lAa6BzrmeAFAlwRDLIACgkQ+lAa6Bzr meB2FQgAxF85pmFK90op1tY/lPKFqnQZdbx1zq1gPQVS1N30Kqt+FTJYSi0HNfPd Q0l1TVx7BCF2VFzqQkLWPwtjeK8OP8SY9D8ShbYo2/Ul0e0tfc2L9+YJpa3QCzvE p3G6SE92+wZZZlPoT+bVGj4heRUrCzUi77/bTIRO4JkSelJFDSZEVoZqTabYuveh lBtbDZ6WvFxAGZg3fSjpZwq31C0cV/W7S0FJhutb+rAhIpoL4jHmddnhUSq+mM0+ lHNQ3O8WmbzR8FY6lezhhmBir29iW/2gJ/+Z5kSSdBV3buk6O8LB2WbFORhpGSYb pWb2drFGVZeQhuOrboG95ZQ1d+4YnQ== =skLP -END PGP SIGNATURE-
Re: Can we drop upstream Linux x32 support?
Steven Newbury dixit: >I can't help but wonder if it wouldn't make more sense to drop x86 >support from long mode than x32. AMD64 x86 support was always intended Do you mean i386? x86 = { i386, x32, amd64 } No, please don’t. I use i386 as “companion architecture” to x32, only the kernel and actually memory-hungry things (qemu) are amd64 binaries on my system, and this works very well. Thanks, //mirabilos -- 18:47⎜ well channels… you see, I see everything in the same window anyway 18:48⎜ i know, you have some kind of telnet with automatic pong 18:48⎜ haha, yes :D 18:49⎜ though that's more tinyirc – sirc is more comfy
Re: Can we drop upstream Linux x32 support?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 First off I'd like to request: Please don't break my userspace! I have a number of systems running with x32-abi as native. They work well, I've no want or desire to upgrade their memory or CPUs to make keep them working as well as they do now. Let alone the hassle of having to redeploy with a completely different ABI. I've been working on getting as much userspace as I can working with x32 since it first became somewhat usable, I've sent patches upstream and pushed to encourage projects to write portable code. Sometimes that has meant just little tweaks to build systems, or sometimes disabling asm where I consider it isn't worth the effort of making it work. For other projects I've converted the asm or even in some cases got the JIT working, mozjs17 for example. So I'm both a user and a developer. Breaking support for x32 would be really bad for me, but if I'm the only one using it I suppose I can't really justify it being kept on that basis. I know CERN has sucessfully experimented with it over the years though, so I wouldn't be surprised if there are other users hiding out there. I can't help but wonder if it wouldn't make more sense to drop x86 support from long mode than x32. AMD64 x86 support was always intended as a compatibility feature, I very much doubt there are more people out there using an x86 native userspace on a 64 bit kernel than there are native x32 users. x86 support could be implemented via KVM and/or qemu-user. There is no reason to keep the extra complexity in the kernel for what is effectively an emulation layer anyway. -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEEWa1B4K0Hk12RDstE+lAa6BzrmeAFAlwQ0QQACgkQ+lAa6Bzr meBILQf9EF1GqHKfnRC7AOFnNCm0235OmH1dJJd4+6zzLWTKGAAvFF6T1F1IG3fu QTZTEok5s238BapjrvgZ5bxtMP0TGNK++vGZ8ESb6Hi+Q975duemWD8ZsSVPw7SH YcqEgmxKn28iHq/W//SUPo1kqz7D0jFCDU9dIA1wQY+AwTIzjfMPltWGrKbMbOBQ LsW+VlL7PfoEzx9sXvaMpjgINEouCvLcuTvhTRclCOO5MWqTQLdIdL9urrBGukUN 7dvKiWWAk6c/Af1W5jnLtYIijaztu3hrZ7lykFmOnwyDoeOhqzIhUkcDaLJcy7Vo Rsrb1CjzFngpbgTJeOkyC9ZGZ2CZ0g== =TCpw -END PGP SIGNATURE-
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 3:38 AM Thorsten Glaser wrote: > > Andy Lutomirski dixit: > > >That’s the thing, though: the whole generic kernel compat > >infrastructure assumes there are at most two ABIs: native and, if > >enabled and relevant, compat. x32 breaks this entirely. > > MIPS had o32, n32, n64 since like forever. o32 and n32 are practically the same, the only difference on the syscall ABI that I can see are the actual syscall numbers, and the 'struct sigcontext' definition. > ARM has old ABI, EABI and now 64-bit. arm64 intentionally did not attempt to support OABI user space because of this, and as I said the ilp32 ABI follows what MIPS n32 does using the same data structures as aarch32 (corresponding to mips o32). > >How hard would it be to have __attribute__((ilp64)), with an optional > >warning if any embedded structs are not ilp64? This plus a wrapper to > > You mean LP64. Impossible, because LP64 vs. ILP32 is not the only > difference between amd64 and x32. I think the above is what Intel's compiler does, and similar to what they do for mixing big-endian and little-endian code (!). Generally possible yes, but a lot of work, as well as error-prone and not particular appealing for the GNU toolchain IMHO. Arnd
Re: Can we drop upstream Linux x32 support?
Andy Lutomirski dixit: >That’s the thing, though: the whole generic kernel compat >infrastructure assumes there are at most two ABIs: native and, if >enabled and relevant, compat. x32 breaks this entirely. MIPS had o32, n32, n64 since like forever. ARM has old ABI, EABI and now 64-bit. Other architectures are yet to come. >IMO the real right solution would be to push the whole problem to >userspace: get an ILP32 system working with almost or entirely LP64 Is this a reflex of Linux kernel developers? ;-) I doubt that userspace is the right place for this, remember the recent glibc vs. syscalls debate. It would also need to multiply across various libcs. >How hard would it be to have __attribute__((ilp64)), with an optional >warning if any embedded structs are not ilp64? This plus a wrapper to You mean LP64. Impossible, because LP64 vs. ILP32 is not the only difference between amd64 and x32. bye, //mirabilos -- I believe no one can invent an algorithm. One just happens to hit upon it when God enlightens him. Or only God invents algorithms, we merely copy them. If you don't believe in God, just consider God as Nature if you won't deny existence. -- Coywolf Qi Hunt
Re: Can we drop upstream Linux x32 support?
> On Dec 11, 2018, at 3:35 PM, Thorsten Glaser wrote: > > Andy Lutomirski dixit: > >> What happens if someone adds a struct like: >> >> struct nasty_on_x32 { >> __kernel_long_t a; >> void * __user b; >> }; >> >> On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte >> fields. On x32, it's an 8-byte field and a 4-byte field. Now what? > > Yes, that’s indeed ugly. I understand. But don’t we already have > this problem with architectures which support multiple ABIs at the > same time? An amd64 kernel with i386 userspace comes to mind, or > the multiple MIPS ABIs. That’s the thing, though: the whole generic kernel compat infrastructure assumes there are at most two ABIs: native and, if enabled and relevant, compat. x32 breaks this entirely. > >> I'm sure we could have some magic gcc plugin or other nifty tool that >> gives us: >> >> copy_from_user(struct struct_name, kernel_ptr, user_ptr); > > Something like that might be useful. Generate call stubs, which > then call the syscall implementation with the actual user-space > struct contents as arguments. Hm, that might be too generic to > be useful. Generate macros that can read from or write specific > structures to userspace? > > I think something like this could solve other more general problems > as well, so it might be “nice to have anyway”. Of course it’s work, > and I’m not involved enough in Linux kernel programming to be able > to usefully help with it (doing too much elsewhere already). > >> actually do this work. Instead we get ad hoc fixes for each syscall, >> along the lines of preadv64v2(), which get done when somebody notices > > Yes, that’s absolutely ugly and ridiculous and all kinds of bad. > > On the other hand, from my current experience, someone (Arnd?) noticed > all the currently existing baddies for x32 already and fixed them. > > New syscalls are indeed an issue, but perhaps something generating > copyinout stubs could help. This might allow other architectures > that could do with a new ABI but have until now feared the overhead > as well. (IIRC, m68k could do with a new ABI that reserves a register > for TLS, but Geert would know. At the same time, time_t and off_t could > be bumped to 64 bit. Something like that. If changing sizes of types > shared between kernel and user spaces is not something feared…) Magic autogenerated stubs would be great. Difficult, too, given unions, multiplexers, cmsg, etc. I suppose I will see how bad it would be to split out the x32 syscall table and at least isolate the mess to some extent. IMO the real right solution would be to push the whole problem to userspace: get an ILP32 system working with almost or entirely LP64 syscalls. POSIX support might have to be a bit flexible, but still. How hard would it be to have __attribute__((ilp64)), with an optional warning if any embedded structs are not ilp64? This plus a wrapper to make sure that mmap puts everything below 4GB ought to do the trick. Or something like what arm64 is proposing where the kernel ABI has 32-bit long doesn’t seem too horrible.
Re: Can we drop upstream Linux x32 support?
On Tue, 11 Dec 2018, Shawn Rutledge wrote: > I like the idea of x32 in theory: any machine with 4GB or less of memory > for sure does not really need to waste any precious space storing 64-bit > pointers; and even if more memory is available, software that needs to > access more than 4GB per process is somewhat rare (as it should be). You can't mmap(2) files the size of which exceeds the size of the 32-bit user address space with an ILP32 ABI though, regardless of how much physical memory a given system supports or has actually installed. Also you can have a process's VM size exceed the 32-bit user address space where swap files are used, e.g. I have a system with 3GiB of DRAM installed and as much swap space available, where the amount of memory consumed by a process sometimes exceeds 4GiB -- it's enough to open a couple of web pages at once in a browser these days to get there (don't ask me what it uses all that memory for). So there are still use cases for an LP64 ABI even for systems the whole physical memory of which can be handled with 32-bit addressing. Maciej
Re: Can we drop upstream Linux x32 support?
On Wed, Dec 12, 2018 at 12:38 AM Thorsten Glaser wrote: > > Andy Lutomirski dixit: > > >What happens if someone adds a struct like: > > > >struct nasty_on_x32 { > > __kernel_long_t a; > > void * __user b; > >}; > > > >On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte > >fields. On x32, it's an 8-byte field and a 4-byte field. Now what? > > Yes, that’s indeed ugly. I understand. But don’t we already have > this problem with architectures which support multiple ABIs at the > same time? An amd64 kernel with i386 userspace comes to mind, or > the multiple MIPS ABIs. The main trouble that I see with x32 is in how it's different from all the other compat architectures. We have a number of drivers that support not only two ABIs (native and compat) but also a third one specifically for x32, and more drivers that fail on x32 because they never implemented the third case. Note that this is not a problem for MIPS n32 and aarch64ilp32, which generally stick to the regular compat ABI (each compat architecture has some minor quirks somewhere, much less so than x32 does). > >actually do this work. Instead we get ad hoc fixes for each syscall, > >along the lines of preadv64v2(), which get done when somebody notices > > Yes, that’s absolutely ugly and ridiculous and all kinds of bad. > > On the other hand, from my current experience, someone (Arnd?) noticed > all the currently existing baddies for x32 already and fixed them. I did some, and others did more, but it's hard to ever do a complete job, and even harder to prevent new bugs from creeping in. > New syscalls are indeed an issue, but perhaps something generating > copyinout stubs could help. This might allow other architectures > that could do with a new ABI but have until now feared the overhead > as well. (IIRC, m68k could do with a new ABI that reserves a register > for TLS, but Geert would know. At the same time, time_t and off_t could > be bumped to 64 bit. Something like that. If changing sizes of types > shared between kernel and user spaces is not something feared…) Creating a completely new ABI for an existing architecutre is generally something you want to avoid, as this means all existing user space has to be recompiled. All architectures support 64-bit off_t already, and soon will support 64-bit time_t as well, at this point (as of this week's linux-next) we mostly just need to assign a couple of syscall numbers for each 32-bit architectures. Arnd
Re: Can we drop upstream Linux x32 support?
On Tue, 11 Dec 2018, Thorsten Glaser wrote: > Furthermore, x32 was the first of the many *64ilp32 > architectures; I know I’ve seen amd64ilp32 and at least > one other I don’t recall. Maybe you mean n32 MIPS, complete kernel+userland support for which has been available in Linux from 2001? Maciej
Re: Can we drop upstream Linux x32 support?
Andy Lutomirski dixit: >What happens if someone adds a struct like: > >struct nasty_on_x32 { > __kernel_long_t a; > void * __user b; >}; > >On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte >fields. On x32, it's an 8-byte field and a 4-byte field. Now what? Yes, that’s indeed ugly. I understand. But don’t we already have this problem with architectures which support multiple ABIs at the same time? An amd64 kernel with i386 userspace comes to mind, or the multiple MIPS ABIs. >I'm sure we could have some magic gcc plugin or other nifty tool that >gives us: > >copy_from_user(struct struct_name, kernel_ptr, user_ptr); Something like that might be useful. Generate call stubs, which then call the syscall implementation with the actual user-space struct contents as arguments. Hm, that might be too generic to be useful. Generate macros that can read from or write specific structures to userspace? I think something like this could solve other more general problems as well, so it might be “nice to have anyway”. Of course it’s work, and I’m not involved enough in Linux kernel programming to be able to usefully help with it (doing too much elsewhere already). >actually do this work. Instead we get ad hoc fixes for each syscall, >along the lines of preadv64v2(), which get done when somebody notices Yes, that’s absolutely ugly and ridiculous and all kinds of bad. On the other hand, from my current experience, someone (Arnd?) noticed all the currently existing baddies for x32 already and fixed them. New syscalls are indeed an issue, but perhaps something generating copyinout stubs could help. This might allow other architectures that could do with a new ABI but have until now feared the overhead as well. (IIRC, m68k could do with a new ABI that reserves a register for TLS, but Geert would know. At the same time, time_t and off_t could be bumped to 64 bit. Something like that. If changing sizes of types shared between kernel and user spaces is not something feared…) Thanks for considering, //mirabilos -- „Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund, mksh auf jedem System zu installieren.“ -- XTaran auf der OpenRheinRuhr, ganz begeistert (EN: “[…]uhr.gz is a reason to install mksh on every system.”)
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 03:22:43PM -0800, Andy Lutomirski wrote: > On Tue, Dec 11, 2018 at 2:14 PM Thorsten Glaser wrote: > > I was also considering investing a nontrivial amount of > > work into porting klibc to x32, since hpa does not wish > > to do it himself. Thankfully I have only done a bit yet. > > > > Furthermore, x32 was the first of the many *64ilp32 > > architectures; I know I’ve seen amd64ilp32 and at least > > one other I don’t recall. It will have prototyped many > > of the problems users of these will run in, and I’d prefer > > to keep it (completely selfish because I don’t wish to > > have to crossgrade a whole system yet again). > > it kind of seems like arm64's lesson is "don't do it like x32". > > There's some effort going on right now to make it possible to add > syscalls without having to muck with every single architecture. I > don't really want x32 to derail that effort. I suppose we could say > that x32 stays but that it simply gets no new syscalls, but that seems > a bit lame. Unfortunately, on x86, x32 really is a third ABI that is > not compatible in a structure-memory-layout sense with the other two. > What happens if someone adds a struct like: > > struct nasty_on_x32 { > __kernel_long_t a; > void * __user b; > }; > > On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte > fields. On x32, it's an 8-byte field and a 4-byte field. Now what? > > I'm sure we could have some magic gcc plugin or other nifty tool that gives > us: > > copy_from_user(struct struct_name, kernel_ptr, user_ptr); > > where it automatically generates code for all possible ABIs to copy > over the struct and dispatches dynamically based on the current > syscall ABI, but I have trouble imagining anyone volunteering to > actually do this work. Instead we get ad hoc fixes for each syscall, > along the lines of preadv64v2(), which get done when somebody notices > a problem. For what it's worth, this is exactly what I plan to do if/when I ever get to doing my kernel with multiple/universal userspace ABI support: pure-data markup for the data structures (and syscall argument arrangements) userspace is speaking, with generic marshalling code. But I don't think it's easy to retrofit to Linux, and probably wouldn't be welcome there. Rich
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 09:59:48PM +, Thorsten Glaser wrote: > John Paul Adrian Glaubitz dixit: > > >I can't say anything about the syscall interface. However, what I do know > >is that the weird combination of a 32-bit userland with a 64-bit kernel > >interface is sometimes causing issues. For example, application code usually > > Yes, but more and more ${foo}64ilp32 architectures are popping up. > > >Additionally, x32 support in many applications is either rudimentary > > If a signal is sent that this kind of architectures will stay, some > people might be convinced to fix that. > > >It's also that the performance benefits of x32 are often eaten up by > >the fact that none of the scripted languages that I know of provide > > Non-JITted languages like yours truly’s shell do benefit from it, > though. (mksh works just fine on LP64 but its internal structures > pack massively better on ILP32, for example.) gcc also benefits massively, up to 2x or more performance improvement from ILP32, just because you can double your -jN to make. The "or more" comes from cache utilization also improving. Actually I often use i486 gcc even on 64-bit systems for this reason, and have observed this order of improvement despite the lack of registers and worse call ABI. Presumably x32 would be better, but I've avoided it because of how buggy it's been in the past. > >If x32 is eventually to be removed, we should also take care of removing > >x32 support from userland code. From the top of my head, this would at least > > I don’t think so. The patches also contain > – stuff to support 64-bit time_t on 32-bit architectures, e.g: > - bugfixes like printf("%lld", (long long)timet_value) instead > of assuming time_t fits into a long (also important for other > operating systems…) > - generally switching from generic types like long to specific > types like size_t, ptrdiff_t, etc. > - there was one more but after having written two eMails I forgot it > - oh and, of course, they lay the base for e.g. amd64ilp32 support I would not classify any of that as part of "removing x32 support", but rather as "reintroducing broken assumptions". I think "removing x32 support" was intended to mean places where there's explicitly-x32-specific code or tooling, like asm functions/fragments in libc, x32 versions of the dispatch asm in mesa, x32 code generation, etc. Obviously (at least I hope it's obvious) nonportable assumptions should not be reintroduced regardless of whether x32 is dropped. FWIW x32 is nice as a test case for catching such nonportable assumptions and getting them fixed. Rich
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 2:14 PM Thorsten Glaser wrote: > > Note: please keep me in Cc because I am not subscribed. > > Linus Torvalds dixit: > > >I'm not opposed to trying to sunset the support, but let's see who > >complains.. > > I will hereby complain. > > I’m using Debian/x32 on my main desktop at work, and do > occasionally help out with porting issues. It’s a good > way to make more out of 64-bit machines without going > all 64 bit; it’s also helped me find bugs in software. > It’s a nice architectural idea, and a way forward for > things that are constricted to 32 bits while opening > up stuff like 64-bit time_t without taking up half the > available CPU registers (while more than doubling the > number of the available CPU registers, too). Thanks for responding! I suppose the question is: are you enough of a user to justify the continued maintenance effort. > > I was also considering investing a nontrivial amount of > work into porting klibc to x32, since hpa does not wish > to do it himself. Thankfully I have only done a bit yet. > > Furthermore, x32 was the first of the many *64ilp32 > architectures; I know I’ve seen amd64ilp32 and at least > one other I don’t recall. It will have prototyped many > of the problems users of these will run in, and I’d prefer > to keep it (completely selfish because I don’t wish to > have to crossgrade a whole system yet again). it kind of seems like arm64's lesson is "don't do it like x32". There's some effort going on right now to make it possible to add syscalls without having to muck with every single architecture. I don't really want x32 to derail that effort. I suppose we could say that x32 stays but that it simply gets no new syscalls, but that seems a bit lame. Unfortunately, on x86, x32 really is a third ABI that is not compatible in a structure-memory-layout sense with the other two. What happens if someone adds a struct like: struct nasty_on_x32 { __kernel_long_t a; void * __user b; }; On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte fields. On x32, it's an 8-byte field and a 4-byte field. Now what? I'm sure we could have some magic gcc plugin or other nifty tool that gives us: copy_from_user(struct struct_name, kernel_ptr, user_ptr); where it automatically generates code for all possible ABIs to copy over the struct and dispatches dynamically based on the current syscall ABI, but I have trouble imagining anyone volunteering to actually do this work. Instead we get ad hoc fixes for each syscall, along the lines of preadv64v2(), which get done when somebody notices a problem. Linus, any advice here? --Andy
Re: Can we drop upstream Linux x32 support?
John Paul Adrian Glaubitz dixit: >I can't say anything about the syscall interface. However, what I do know >is that the weird combination of a 32-bit userland with a 64-bit kernel >interface is sometimes causing issues. For example, application code usually Yes, but more and more ${foo}64ilp32 architectures are popping up. >Additionally, x32 support in many applications is either rudimentary If a signal is sent that this kind of architectures will stay, some people might be convinced to fix that. >It's also that the performance benefits of x32 are often eaten up by >the fact that none of the scripted languages that I know of provide Non-JITted languages like yours truly’s shell do benefit from it, though. (mksh works just fine on LP64 but its internal structures pack massively better on ILP32, for example.) >If x32 is eventually to be removed, we should also take care of removing >x32 support from userland code. From the top of my head, this would at least I don’t think so. The patches also contain – stuff to support 64-bit time_t on 32-bit architectures, e.g: - bugfixes like printf("%lld", (long long)timet_value) instead of assuming time_t fits into a long (also important for other operating systems…) - generally switching from generic types like long to specific types like size_t, ptrdiff_t, etc. - there was one more but after having written two eMails I forgot it - oh and, of course, they lay the base for e.g. amd64ilp32 support bye, //mirabilos -- FWIW, I'm quite impressed with mksh interactively. I thought it was much *much* more bare bones. But it turns out it beats the living hell out of ksh93 in that respect. I'd even consider it for my daily use if I hadn't wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh
Re: Can we drop upstream Linux x32 support?
Note: please keep me in Cc because I am not subscribed. Linus Torvalds dixit: >I'm not opposed to trying to sunset the support, but let's see who complains.. I will hereby complain. I’m using Debian/x32 on my main desktop at work, and do occasionally help out with porting issues. It’s a good way to make more out of 64-bit machines without going all 64 bit; it’s also helped me find bugs in software. It’s a nice architectural idea, and a way forward for things that are constricted to 32 bits while opening up stuff like 64-bit time_t without taking up half the available CPU registers (while more than doubling the number of the available CPU registers, too). I was also considering investing a nontrivial amount of work into porting klibc to x32, since hpa does not wish to do it himself. Thankfully I have only done a bit yet. Furthermore, x32 was the first of the many *64ilp32 architectures; I know I’ve seen amd64ilp32 and at least one other I don’t recall. It will have prototyped many of the problems users of these will run in, and I’d prefer to keep it (completely selfish because I don’t wish to have to crossgrade a whole system yet again). Thanks for your consideration, //mirabilos -- When he found out that the m68k port was in a pretty bad shape, he did not, like many before him, shrug and move on; instead, he took it upon himself to start compiling things, just so he could compile his shell. How's that for dedication. -- Wouter, about my Debian/m68k revival
Re: Can we drop upstream Linux x32 support?
I like the idea of x32 in theory: any machine with 4GB or less of memory for sure does not really need to waste any precious space storing 64-bit pointers; and even if more memory is available, software that needs to access more than 4GB per process is somewhat rare (as it should be). I used a laptop every day with 4GB until recently, so was generally trying to run relatively lean software. I have played around with the Gentoo x32 variant on another machine, just to see how that would go. I haven't used it for much... but tried out some basic things; and as long as it is powered on, it functions fine as a node in an icecream cluster, too. The icecream statistics shown in icemon tell me that this machine seems to be performing faster than it should be, for its age. So I think there is some worthwhile average speedup (without being more scientific about it). As a Qt developer, I have some interest that Qt and Qt Quick ought to work well on this architecture. (When it breaks, I write up bugs like QTBUG-52658 and QTBUG-42759.) There is still no JIT support for QML, but perhaps there could be (I could try, if I ever find time to learn how the existing QML/V4 JIT works on other architectures well enough to add another one); and meanwhile, at least the interpreter works AFAIK. I was just thinking of testing this again, maybe running gentoo or debian x32 on that old 4GB laptop; so I did a quick google search, and found this thread right away. Linux supports a lot of rarely-used hardware and features; so as long as the burden of supporting x32 is not too great, I'd like to put in my vote to keep it working. And I will keep testing from time to time and do my bit to keep Qt working on this architecture, as long as I'm not wasting my time (if it's going to disappear, it would be a waste of time). I can't explain why there aren't more users, and more distros. It seems to me it would be a good choice for the compact or small-is-beautiful distros, like Puppy. (CC me on this thread if you like, since I'm not currently subscribed to LKML)
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 12:37:42PM +0100, Florian Weimer wrote: > * Catalin Marinas: > > On Tue, Dec 11, 2018 at 10:02:45AM +0100, Arnd Bergmann wrote: > >> On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski wrote: > >> > I tried to understand what's going on. As far as I can tell, most of > >> > the magic is the fact that __kernel_long_t and __kernel_ulong_t are > >> > 64-bit as seen by x32 user code. This means that a decent number of > >> > uapi structures are the same on x32 and x86_64. Syscalls that only > >> > use structures like this should route to the x86_64 entry points. But > >> > the implementation is still highly dubious -- in_compat_syscall() will > >> > be *true* in such system calls, > >> > >> I think the fundamental issue was that the intention had always been > >> to use only the 64-bit entry points for system calls, but the most > >> complex one we have -- ioctl() -- has to use the compat entry point > >> because device drivers define their own data structures using 'long' > >> and pointer members and they need translation, as well as > >> matching in_compat_syscall() checks. This in turn breaks down > >> again whenever a driver defines an ioctl command that takes > >> a __kernel_long_t or a derived type like timespec as its argument. > > > > With arm64 ILP32 we tried to avoid the ioctl() problem by having > > __kernel_long_t 32-bit, IOW mimicking the arm32 ABI (compat). The > > biggest pain point is signals where the state is completely different > > from arm32 (more, wider registers) and can't be dealt with by the compat > > layer. > > I would expect to approach this from the opposite direction: use 64-bit > types in places where the 64-bit kernel interface uses 64-bit types. > After all, not everyone who is interested in ILP32 has a companion > 32-bit architecture which could serve as a model for the application > ABI. I fully agree with you that if someone wants ILP32 for a 64-bit only architecture, they should use the 64-bit kernel interface and ensure POSIX is adjusted. In the arm64 context, both options were discussed with the libc community complaining that a partial 64-bit syscall ABI breaks POSIX while the potential users were just asking for a 32-bit ABI to run their existing software stack on ARMv8 machines without native 32-bit support (until they complete the migration to 64-bit). > (If there are conflicts with POSIX, then POSIX needs to be fixed to > support this.) This would have been nice but no-one volunteered and, more importantly, there was no conclusive argument that ARM ILP32 is better than LP64 (well, apart from a minority of benchmarks) and something that people would want to migrate to. Given that the only credible case made was about legacy code, we decided to go ahead with a (mostly) compat 32-bit ABI. -- Catalin
Re: Can we drop upstream Linux x32 support?
* Catalin Marinas: > On Tue, Dec 11, 2018 at 10:02:45AM +0100, Arnd Bergmann wrote: >> On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski wrote: >> > I tried to understand what's going on. As far as I can tell, most of >> > the magic is the fact that __kernel_long_t and __kernel_ulong_t are >> > 64-bit as seen by x32 user code. This means that a decent number of >> > uapi structures are the same on x32 and x86_64. Syscalls that only >> > use structures like this should route to the x86_64 entry points. But >> > the implementation is still highly dubious -- in_compat_syscall() will >> > be *true* in such system calls, >> >> I think the fundamental issue was that the intention had always been >> to use only the 64-bit entry points for system calls, but the most >> complex one we have -- ioctl() -- has to use the compat entry point >> because device drivers define their own data structures using 'long' >> and pointer members and they need translation, as well as >> matching in_compat_syscall() checks. This in turn breaks down >> again whenever a driver defines an ioctl command that takes >> a __kernel_long_t or a derived type like timespec as its argument. > > With arm64 ILP32 we tried to avoid the ioctl() problem by having > __kernel_long_t 32-bit, IOW mimicking the arm32 ABI (compat). The > biggest pain point is signals where the state is completely different > from arm32 (more, wider registers) and can't be dealt with by the compat > layer. I would expect to approach this from the opposite direction: use 64-bit types in places where the 64-bit kernel interface uses 64-bit types. After all, not everyone who is interested in ILP32 has a companion 32-bit architecture which could serve as a model for the application ABI. (If there are conflicts with POSIX, then POSIX needs to be fixed to support this.) Thanks, Florian
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 10:02:45AM +0100, Arnd Bergmann wrote: > On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski wrote: > > I tried to understand what's going on. As far as I can tell, most of > > the magic is the fact that __kernel_long_t and __kernel_ulong_t are > > 64-bit as seen by x32 user code. This means that a decent number of > > uapi structures are the same on x32 and x86_64. Syscalls that only > > use structures like this should route to the x86_64 entry points. But > > the implementation is still highly dubious -- in_compat_syscall() will > > be *true* in such system calls, > > I think the fundamental issue was that the intention had always been > to use only the 64-bit entry points for system calls, but the most > complex one we have -- ioctl() -- has to use the compat entry point > because device drivers define their own data structures using 'long' > and pointer members and they need translation, as well as > matching in_compat_syscall() checks. This in turn breaks down > again whenever a driver defines an ioctl command that takes > a __kernel_long_t or a derived type like timespec as its argument. With arm64 ILP32 we tried to avoid the ioctl() problem by having __kernel_long_t 32-bit, IOW mimicking the arm32 ABI (compat). The biggest pain point is signals where the state is completely different from arm32 (more, wider registers) and can't be dealt with by the compat layer. Fortunately, we haven't merge it yet as we have the same dilemma about real users and who's going to regularly test the ABI in the long run. In the meantime, watching this thread with interest ;). -- Catalin
Re: Can we drop upstream Linux x32 support?
On 12/11/18 11:37 AM, Florian Weimer wrote: > Note that OpenJDK (well, Hotspot) has its own 32-bit pointer support for > the Java heap (compressed oops), so only the native code parts (and JNI) > benefit from x32 anyway. Yeah, I was actually only talking about this single commit by me to add Zero build support for x86_64-linux-gnux32: > http://hg.openjdk.java.net/jdk/jdk/rev/9ce4a0d718c7 Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaub...@debian.org `. `' Freie Universitaet Berlin - glaub...@physik.fu-berlin.de `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: Can we drop upstream Linux x32 support?
* John Paul Adrian Glaubitz: > As for the enterprise support, this seems to be correct. I don't know > of any enterprise distribution with x32 support either. Me neither. I would expect a pure userspace port, with limitations in what ioctls you can use, and perhaps support from GCC to share more APIs with the kernel. (I do not know of any plans to work on this.) > I'm not terribly opposed to this change. I'm usually for keeping support > for things that people are using, but the maintenance is a huge burden > to upstream projects, I'm fine with letting it go. Thank you for sharing your perspective. > If x32 is eventually to be removed, we should also take care of removing > x32 support from userland code. From the top of my head, this would at least > concern: > > * OpenJDK Note that OpenJDK (well, Hotspot) has its own 32-bit pointer support for the Java heap (compressed oops), so only the native code parts (and JNI) benefit from x32 anyway. Thanks, Florian
Re: Can we drop upstream Linux x32 support?
Hello! I'm Debian's principal maintainer of the x32 port. On 12/11/18 2:23 AM, Andy Lutomirski wrote: > 1. It's not entirely clear that it has users. As far as I know, it's > supported on Gentoo and Debian, and the Debian popcon graph for x32 > has been falling off dramatically. I don't think that any enterprise > distro has ever supported x32. There are definitely some users of this port. I don't know the actual number, but I hear from users from time to time. As for the popcon curve, I wouldn't say it has dropped dramatically as it was never high in the first place. > https://popcon.debian.org/stat/sub-x32.png It seems that the highest number of recorded users was 18 and it's now down to 7. Keep in mind though that popcon participation is opt-in, so the actual number of users should be higher. According to popcon, there are also only 172331 Debian installations on x86_64: > https://popcon.debian.org/ As for the enterprise support, this seems to be correct. I don't know of any enterprise distribution with x32 support either. > 2. The way that system calls work is very strange. Most syscalls on > x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE) > entry point, and this is intentional. For example, adjtimex() uses > the native entry, not the compat entry, because x32's struct timex > matches the x86_64 layout. But a handful of syscalls have separate > entry points -- these are the syscalls starting at 512. These enter > through the COMPAT_SYSCALL_DEFINE entry points. > > The x32 syscalls that are *not* in the 512 range violate all semblance > of kernel syscall convention. In the syscall handlers, > in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry > is not invoked. This is nutty and risks breaking things when people > refactor their syscall implementations. And no one tests these > things. Similarly, if someone calls any of the syscalls below 512 but > sets bit 31 in RAX, then the native entry will be called with > in_compat_set(). > > Conversely, if you call a syscall in the 512 range with bit 31 > *clear*, then the compat entry is set with in_compat_syscall() > *clear*. This is also nutty. I can't say anything about the syscall interface. However, what I do know is that the weird combination of a 32-bit userland with a 64-bit kernel interface is sometimes causing issues. For example, application code usually expects things like time_t to be 32-bit on a 32-bit system. However, this isn't the case for x32 which is why code fails to build. Additionally, x32 support in many applications is either rudimentary or broken. For example, while LLVM has support for x32, the backend isn't really stable on this target meaning that compilers like clang or Rust are partially broken or crash. I'm not sure whether anyone is interested in fixing this. It's also that the performance benefits of x32 are often eaten up by the fact that none of the scripted languages that I know of provide a JIT that supports x32. Thus, things like Javascript are either unsupported or slow on x32. > I propose that we make CONFIG_X86_X32 depend on BROKEN for a release > or two and then remove all the code if no one complains. If anyone > wants to re-add it, IMO they're welcome to do so, but they need to do > it in a way that is maintainable. I'm not terribly opposed to this change. I'm usually for keeping support for things that people are using, but the maintenance is a huge burden to upstream projects, I'm fine with letting it go. There are other architectures in the kernel like Alpha, HPPA, M68K, PowerPC, SH and SPARC that I care much more about than x32. If x32 is eventually to be removed, we should also take care of removing x32 support from userland code. From the top of my head, this would at least concern: * OpenJDK * LLVM * gcc * glibc * Rust * binutils I can take care of these once I know about the decision regarding the kernel. Usually, it's a matter of grepping the commit history for "x32" and revert the corresponding commits. Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaub...@debian.org `. `' Freie Universitaet Berlin - glaub...@physik.fu-berlin.de `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: Can we drop upstream Linux x32 support?
On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski wrote: > On Mon, Dec 10, 2018 at 7:15 PM H.J. Lu wrote: > > On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski wrote: > Right. My question wasn't whether x32 had developers -- it was > whether it had users. If the only users are a small handful of people > who keep the toolchain and working and some people who benchmark it, > then I think the case for keeping it in upstream Linux is a bit weak. +1 > > > Conversely, if you call a syscall in the 512 range with bit 31 > > > *clear*, then the compat entry is set with in_compat_syscall() > > > *clear*. This is also nutty. > > > > This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel. > > > > I tried to understand what's going on. As far as I can tell, most of > the magic is the fact that __kernel_long_t and __kernel_ulong_t are > 64-bit as seen by x32 user code. This means that a decent number of > uapi structures are the same on x32 and x86_64. Syscalls that only > use structures like this should route to the x86_64 entry points. But > the implementation is still highly dubious -- in_compat_syscall() will > be *true* in such system calls, I think the fundamental issue was that the intention had always been to use only the 64-bit entry points for system calls, but the most complex one we have -- ioctl() -- has to use the compat entry point because device drivers define their own data structures using 'long' and pointer members and they need translation, as well as matching in_compat_syscall() checks. This in turn breaks down again whenever a driver defines an ioctl command that takes a __kernel_long_t or a derived type like timespec as its argument. > which means that, if someone changes: > ... > where one argument has x32 and x86_64 matching but the other has x32 > and x86_32 matching. > > This whole thing seems extremely fragile. It definitely is. We have lots of workarounds specifically for x32 in device drivers, but in the time_t conversion for y2038 I still found ones that had not been caught earlier, and for each y2038 conversion that someone did to a driver or syscall, we have to make sure that it doesn't break x32 in the process. Arnd
Re: Can we drop upstream Linux x32 support?
* Linus Torvalds: > Apparently the main real use case is for extreme benchmarking. It's > the only use-case where the complexity of maintaining a whole > development environment and distro is worth it, it seems. Apparently a > number of Spec submissions have been done with the x32 model. Are you sure GCC was used for that? ICC has a 32-bit pointer mode which does not change the (64-bit) kernel interface. Thanks, Florian
Re: Can we drop upstream Linux x32 support?
On Mon, Dec 10, 2018 at 05:23:39PM -0800, Andy Lutomirski wrote: > Hi all- > > I'm seriously considering sending a patch to remove x32 support from > upstream Linux. Here are some problems with it: > > 1. It's not entirely clear that it has users. As far as I know, it's > supported on Gentoo and Debian, and the Debian popcon graph for x32 > has been falling off dramatically. I don't think that any enterprise > distro has ever supported x32. > > 2. The way that system calls work is very strange. Most syscalls on > x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE) > entry point, and this is intentional. For example, adjtimex() uses > the native entry, not the compat entry, because x32's struct timex > matches the x86_64 layout. But a handful of syscalls have separate > entry points -- these are the syscalls starting at 512. These enter > through the COMPAT_SYSCALL_DEFINE entry points. > > The x32 syscalls that are *not* in the 512 range violate all semblance > of kernel syscall convention. In the syscall handlers, > in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry > is not invoked. This is nutty and risks breaking things when people > refactor their syscall implementations. And no one tests these > things. Similarly, if someone calls any of the syscalls below 512 but > sets bit 31 in RAX, then the native entry will be called with > in_compat_set(). > > Conversely, if you call a syscall in the 512 range with bit 31 > *clear*, then the compat entry is set with in_compat_syscall() > *clear*. This is also nutty. > > Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI > and and CONFIG_X86_X32, which I suspect results in incorrect builds if > the host doesn't have an x32 toolchain installed. > > I propose that we make CONFIG_X86_X32 depend on BROKEN for a release > or two and then remove all the code if no one complains. If anyone Based on the discussion we had at the beginning of the pidfd_send_signal syscall patchset I think this is a good idea. For once, the complex compat handling can make adding new syscalls that need to rely on compat types because of precedent established by older syscalls icky. > wants to re-add it, IMO they're welcome to do so, but they need to do > it in a way that is maintainable. > > --Andy
Re: Can we drop upstream Linux x32 support?
On Mon, Dec 10, 2018 at 7:15 PM H.J. Lu wrote: > > On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski wrote: > > > > Hi all- > > > > I'm seriously considering sending a patch to remove x32 support from > > upstream Linux. Here are some problems with it: > > > > 1. It's not entirely clear that it has users. As far as I know, it's > > supported on Gentoo and Debian, and the Debian popcon graph for x32 > > has been falling off dramatically. I don't think that any enterprise > > distro has ever supported x32. > > I have been posting x32 GCC results for years: > > https://gcc.gnu.org/ml/gcc-testresults/2018-12/msg01358.html Right. My question wasn't whether x32 had developers -- it was whether it had users. If the only users are a small handful of people who keep the toolchain and working and some people who benchmark it, then I think the case for keeping it in upstream Linux is a bit weak. > > > 2. The way that system calls work is very strange. Most syscalls on > > x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE) > > entry point, and this is intentional. For example, adjtimex() uses > > the native entry, not the compat entry, because x32's struct timex > > matches the x86_64 layout. But a handful of syscalls have separate > > This becomes less an issue with 64-bit time_t. > > > entry points -- these are the syscalls starting at 512. These enter > > throuh the COMPAT_SYSCALL_DEFINE entry points. > > > > The x32 syscalls that are *not* in the 512 range violate all semblance > > of kernel syscall convention. In the syscall handlers, > > in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry > > is not invoked. This is nutty and risks breaking things when people > > refactor their syscall implementations. And no one tests these > > things. Similarly, if someone calls any of the syscalls below 512 but > > sets bit 31 in RAX, then the native entry will be called with > > in_compat_set(). > > > > Conversely, if you call a syscall in the 512 range with bit 31 > > *clear*, then the compat entry is set with in_compat_syscall() > > *clear*. This is also nutty. > > This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel. > I tried to understand what's going on. As far as I can tell, most of the magic is the fact that __kernel_long_t and __kernel_ulong_t are 64-bit as seen by x32 user code. This means that a decent number of uapi structures are the same on x32 and x86_64. Syscalls that only use structures like this should route to the x86_64 entry points. But the implementation is still highly dubious -- in_compat_syscall() will be *true* in such system calls, which means that, if someone changes: SYSCALL_DEFINE1(some_func, struct some_struct __user *, ptr) { /* x32 goes here, but it's entirely non-obvious unless you read the x86 syscall table */ native impl; } COMPAT_SYSCALL_DEFINE1(some_func, struct compat_some_struct __user *, ptr) { compat impl; } to the Obviously Equivalent (tm): SYSCALL_DEFINE1(some_func, struct some_struct __user *, ptr) { struct some_struct kernel_val; if (in_compat_syscall()) { get_compat_some_struct(&kernel_val, ptr); } else { copy_from_user(&kernel_val, ptr, sizeof(struct some_struct)); } do the work; } then x32 breaks. And I don't even know how x32 is supposed to support some hypothetical syscall like this: long sys_nasty(struct adjtimex *a, struct iovec *b); where one argument has x32 and x86_64 matching but the other has x32 and x86_32 matching. This whole thing seems extremely fragile.
Re: Can we drop upstream Linux x32 support?
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski wrote: > > Hi all- > > I'm seriously considering sending a patch to remove x32 support from > upstream Linux. Here are some problems with it: > > 1. It's not entirely clear that it has users. As far as I know, it's > supported on Gentoo and Debian, and the Debian popcon graph for x32 > has been falling off dramatically. I don't think that any enterprise > distro has ever supported x32. I have been posting x32 GCC results for years: https://gcc.gnu.org/ml/gcc-testresults/2018-12/msg01358.html > 2. The way that system calls work is very strange. Most syscalls on > x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE) > entry point, and this is intentional. For example, adjtimex() uses > the native entry, not the compat entry, because x32's struct timex > matches the x86_64 layout. But a handful of syscalls have separate This becomes less an issue with 64-bit time_t. > entry points -- these are the syscalls starting at 512. These enter > throuh the COMPAT_SYSCALL_DEFINE entry points. > > The x32 syscalls that are *not* in the 512 range violate all semblance > of kernel syscall convention. In the syscall handlers, > in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry > is not invoked. This is nutty and risks breaking things when people > refactor their syscall implementations. And no one tests these > things. Similarly, if someone calls any of the syscalls below 512 but > sets bit 31 in RAX, then the native entry will be called with > in_compat_set(). > > Conversely, if you call a syscall in the 512 range with bit 31 > *clear*, then the compat entry is set with in_compat_syscall() > *clear*. This is also nutty. This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel. > Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI > and and CONFIG_X86_X32, which I suspect results in incorrect builds if > the host doesn't have an x32 toolchain installed. X86-64 binutils and GCC can be used to build x86-64 kernel with x32 support. > I propose that we make CONFIG_X86_X32 depend on BROKEN for a release > or two and then remove all the code if no one complains. If anyone > wants to re-add it, IMO they're welcome to do so, but they need to do > it in a way that is maintainable. -- H.J.
Re: Can we drop upstream Linux x32 support?
On December 10, 2018 5:40:33 PM PST, Linus Torvalds wrote: >On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski >wrote: >> >> I'm seriously considering sending a patch to remove x32 support from >> upstream Linux. Here are some problems with it: > >I talked to Arnd (I think - we were talking about all the crazy ABI's, >but maybe it was with somebody else) about exactly this in Edinburgh. > >Apparently the main real use case is for extreme benchmarking. It's >the only use-case where the complexity of maintaining a whole >development environment and distro is worth it, it seems. Apparently a >number of Spec submissions have been done with the x32 model. > >I'm not opposed to trying to sunset the support, but let's see who >complains.. > > Linus The use case aside, I need to address the technical issues in this post; some of the behaviors that Andy is pointing out area quite intentional, even if they are perhaps somewhat confusing at first glance. That being said, some were due to tradeoffs that might have been wrong. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Can we drop upstream Linux x32 support?
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski wrote: > > I'm seriously considering sending a patch to remove x32 support from > upstream Linux. Here are some problems with it: I talked to Arnd (I think - we were talking about all the crazy ABI's, but maybe it was with somebody else) about exactly this in Edinburgh. Apparently the main real use case is for extreme benchmarking. It's the only use-case where the complexity of maintaining a whole development environment and distro is worth it, it seems. Apparently a number of Spec submissions have been done with the x32 model. I'm not opposed to trying to sunset the support, but let's see who complains.. Linus