Hi On 2023-11-14, Elliott Mitchell wrote: > On Wed, Nov 15, 2023 at 05:35:11AM +0100, Stefan Lippers-Hollmann wrote: > > On 2023-11-14, Elliott Mitchell wrote: [...] > > What would be the reason for enabling x32? > > There is very little upstream buy-in for x32, when this question came > > up for Debian, it was rejected to be enabled for -among other reasons- > > concerns about the the system call ABI and its security hardening, as > > well as concerns about the long term ABI compatibility (the later of > > which probably not that relevant for OpenWrt, the former however is). > > What I see seems like x32 is still making progress on Debian. I note > musl has a x32 target. While it still isn't a release architecture for > Debian, I could see it becoming one.
I wouldn't really consider this to be very likely, if at all, it has become even less likely than it was over a decade ago. > > I do understand that this is a pet peeve among the high-density > > virtualization crowd, but anywhere else, x32 as a concept is dead (and > > never took off in the first place). > > In my view x32 has some substantial use cases. Have you ever seen `ls`, > `rm`, `grep`, `find`, or `test` needing more than 4GB of memory? Any > of these needing that much memory would suggest something was wrong > (or perhaps someone was trying to abuse them in an impressive way). For > quite a number of shell utilities being able to use more than 4GB of > memory is pointless. The counter question would be, does it hurt? Yes, the x32 proponents (mostly from the high-density virtualization crowd and, as Linus called it "extreme benchmarking" environments) will point out that pure-x32 uses slightly less disk space and slightly less RAM than pure64, but we are talking about very little in today's world. > This is speculation, but I suspect there would be substantial value in > having mixed systems where some utilities were x32 and some were amd64. Please point out what exactly those would be, especially in the context of OpenWrt. If you really want to mix x32 and amd64 binaries, you will need a full library set (binary packages - co-installable) for both x32 and x86_64 installed and in RAM - which is completely detrimental to any of the gains promised above. x32 only works, if you have (aside from the kernel itself) a pure-x32 userspace, or your losses in terms of additional disk space and RAM usage outweigh your gains by an order of magnitude. In a general purpose distribution you might ignore that by claiming that 'only' your big database management system (and its dependencies) would have to be a 64 binary (or some other -very isolated, compared to the rest of the running system- big component), but the situation for OpenWrt is different. This kind of mixed userspace would be a nightmare for complexity on the source side - and/or would require Debian-style multiarch support. Neither of which is very likely to succeed for OpenWrt (let alone for opkg). > > *If* (and imho, again purely my own irrelevant opinion, that is a > > big IF) x32 would really be deemed worthwhile for OpenWrt, it still > > doesn't make sense to enable this whole userspace ABI for the x86_64 > > kernel now, before your desired additional patches are available > > (and even then, you'd probably want a dedicated x32 subtarget instead > > of killing off the pure64 target for OpenWrt - something I'd > > personally be even less in favour of). > > Enabling the kernel support is always the first step. How does one test > userspace if the kernel won't even load the executables? In a general purpose distribution one might say that, however even there one would expect quite a lot of testing having happened before enabling a complete new/ additional ABI to userspace (duplicate to the native one), something akin to the semi-official debian-ports infrastructure, used for bootstrapping of new ports or retiring old ones that still have some form of interest. But in this case there is nothing to test in isolation, as there are no x32 binaries compatible with OpenWrt (you could only test inside a non-OpenWrt x32 chroot, but that's barely grounds for enabling x32 in OpenWrt's kernel, just in case; otherwise float emulation would be enabled for mips, to facilitate Debian chroots - spoiler, it's not). In contrast to the changes necessary (buildsystem, image generation, kernel configs, etc.) to test an OpenWrt x32 subtarget, a lazy-man's two-line patch of enabling X86_X32 && X86_X32_ABI would be trivial in comparison to anything of the rest that would be needed. Given that OpenWrt doesn't have any binary compatibility between major versions and needs to get rebuilt in total for each new revision, there is no need to stage- and phase in additions like this, add the new sub-target in its entirety (or not). [...] > On Wed, Nov 15, 2023 at 05:41:54AM +0100, Stefan Lippers-Hollmann wrote: > > On 2023-11-15, Stefan Lippers-Hollmann wrote: > > > What would be the reason for enabling x32? > > > There is very little upstream buy-in for x32, when this question came > > > up for Debian, it was rejected to be enabled for -among other reasons- > > > concerns about the the system call ABI and its security hardening, as > > > well as concerns about the long term ABI compatibility (the later of > > > which probably not that relevant for OpenWrt, the former however is). > > > > Just to add some references to this: > > - Can we drop upstream Linux x32 support? > > https://lkml.org/lkml/2018/12/10/1145 > > the whole thread is interesting and doesn't display too much > > sympathy for x32 > > Yes, and did it get dropped? No, it didn't. At least some of the issues > brought up there have been fixed. It didn't get any more traction either, rather to the contrary (disk- and RAM are growing in modern hardware, what would be the real gains here). > > - information about the never merged/ defunct x32 port proposed for > > Debian > > https://wiki.debian.org/X32Port > > There are pointers to many bugs which were found and fixed. There don't > appear to be that many unresolved ones. Unfortunately I don't know > whether it is likely to be part of the next formal release. > I also note even the most extreme of the OpenWRT virtual machine > suggestions doesn't suggest more than 4GB of memory. and as show cased, a pure64 OpenWrt can get away with ~16 MB disk space (using a sparse image, rather than the ~121 MB raw disk image) and easily gets away with ~96 MB RAM with its default package set - how much do you think x32 would improve that (we're talking about x86_64 hardware here, not legacy i386 systems)? Is it worth the effort depicted above? (and yes, I've tested both with 23.05.0 on qemu; 12 MB qcow2 sparse image, 96 MB RAM (64 MB throws OOM)). Even the runtime performance improvements should be negligible, all the routing happens in kernelspace (for the most part), which would remain x86_64 - any speed improvements on the userspace would be (apart from being minimal in isolation) completely irrelevant to routing performance. dnsmasq, ls, cp, rm and friends being 5% faster or slower doesn't really matter (and around 5% is the most even its proponents claim). > I don't know the future, but enabling kernel support is the first step. If you wanted to add it to Debian (with working a multi-arch implementation and organically grown repositories (which aren't rebuild aside from sourceful uploads or targeted binNMUs), you would be right - for OpenWrt, no, there really is no need to enable it, before you have the rest of the x32 subtarget ready to be merged (and I can't imagine any reason to enable it for the current OpenWrt x86_64 subtarget at all). For OpenWrt, I could imagine only two approaches to this: - make the x86_64 subtarget x32-only - add a new x32-only subtarget and leave x86_64 as it is Neither would really be 'sensible', but the only workable approach would imho be the later, even if the intention was (I hope not to witness that) to kill off the old x86_64 target. Yes, as laid out before, there isn't any strict need for OpenWrt to remain binary compatible between releases, however this would seriously mess up base vs. ports for (at least) a few weeks (broken packages, package relations, implicit expectations by build recipes throughout the tree) - and anyone doing this in one step (archive wide), without prior testing, would be insane, so the later step of adding a new subtarget in parallel to the old one would be necessary for multiple months (if not at least a stable release cycle shipping both) either way. So I really don't see any reason to enable x32 for the x86_64 subtarget, there's nothing to be gained, just major disadvantages. While I'm not a proponent for a pure-x32 subtarget either (at all), this would be the only workable approach to introduce it. Regards Stefan Lippers-Hollmann _______________________________________________ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel