Re: Release status of i386 for Bullseye and long term support for 3 years?

[Paul Wise]

On Tue, Jan 5, 2021 at 10:24 PM Lou Poppler wrote:

> I would like to help.
...
> Please suggest any debian lists or IRC channels or webpages I should look at,
> or other steps to make myself useful.  Thanks.

The general page for how to help Debian is here:

https://www.debian.org/intro/help

More specifically, you could:

Join the #debian-cd, #debian-boot, #debian-buildd, #debian-ftp,
#debian-ports (introduce yourself here at minimum), #debian-release
and #debian-toolchain IRC channels (and maybe #debian-devel). There
isn't really a mailing list for i386, so maybe use debian-devel or
debian-amd64.

Help test the installer:

https://wiki.debian.org/Teams/DebianCD/ReleaseTesting

Install and enable popularity-contest on your i386 machines and submit
to the Linux hardware database. Encourage other i386 users to do the
same.

https://popcon.debian.org/
https://wiki.debian.org/Hardware/Database
https://linux-hardware.org/

Create install wiki pages for the hardware you own:

https://wiki.debian.org/InstallingDebianOn

Help maintain the installer and documentation:

https://wiki.debian.org/DebianInstaller
https://www.debian.org/releases/stable/i386/

Create a new i386 page in the Ports area on the wiki based on the template:

https://wiki.debian.org/Ports
https://wiki.debian.org/PortTemplate

Maintain the other documentation about the i386 port:

https://www.debian.org/ports/i386/
https://wiki.debian.org/i386

Create a scheme to tag i386-specific bugs (maybe user debian-devel,
usertag i386) and maintain the installer usertags for debian-boot and
the port usertags. Work on fixing the bugs tagged i386.

https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=debian-b...@lists.debian.org=i386
https://wiki.debian.org/Teams/Debbugs/ArchitectureTags

Work on increasing the amount of software that builds and works on i386:

https://udd.debian.org/cgi-bin/ftbfs.cgi?arch=i386
https://buildd.debian.org/status/architecture.php?a=i386
https://tests.reproducible-builds.org/debian/testing/index_suite_i386_stats.html
https://ci.debian.net/status/

Follow #debian and #debian-next and help out any i386 users asking
questions (although there aren't (m)any in practice).

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Lou Poppler]

On Wed, 2020-12-30 at 14:27 +, Andrew M.A. Cater wrote:
> On Mon, Dec 07, 2020 at 07:55:11PM +, Andrew M.A. Cater wrote:
> > Dear release team
> > 
> > There seems to be only one maintainer.
> > 
> 
> Still true as far as I can see - others have stepped up to test i386 
> executables but no more developers.
> 
> > Is i386 going to be supportable for the next 3 1/2 years and buildable for
> > that long (given that almost all machines are now 64 bit capable and we're
> > having to build some packages on amd64 for i386 - per ballombe)?

I would like to help.

I have one "Pentium III (Katmai)" based Gateway2000 machine, with debian_etch,
384MB ram (max possible I think), ethernet, USB1.6, DVD-RW, 3.5" FDD, IDE disks;
and one "VIA C7-M" Pentium M based HP 2133 netbook, Jessie installed, 512MB ram,
ethernet, USB2.0, 4GB SSD, 32GB SD-card, "Express Card/54" slot, WiFi+Bluetooth;
(and one working 486, running an ancient 2.0.30 kernel I compiled myself; plus
some more modern machines we actually use here).

I volunteer using those pentiums for testing and/or building.  I have pretty
good internet speeds with pretty large monthly limits here.  Both pentiums can
run Buster live systems (needing swapspace for DEs), and can be re-installed
with any desired releases.  Various external USB disks are available.

I also volunteer some amount of my own time for this cause -- I have somewhat
rusty but hopefully still serviceable experience coding, reviewing code,
building, integrating, testing, debugging, troubleshooting.

Please suggest any debian lists or IRC channels or webpages I should look at,
or other steps to make myself useful.  Thanks.

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Andrew M.A. Cater]

On Mon, Dec 07, 2020 at 07:55:11PM +, Andrew M.A. Cater wrote:
> Dear release team
> 
> There seems to be only one maintainer.
> 

Still true as far as I can see - others have stepped up to test i386 
executables but no more developers.

> Is i386 going to be supportable for the next 3 1/2 years and buildable for
> that long (given that almost all machines are now 64 bit capable and we're
> having to build some packages on amd64 for i386 - per ballombe)?
> 

>From the discussion: we have some choices:

** Continue building i386 as now

** Build a smaller subsystem to facilitate running old i386 binaries

** Build a system using amd64 kernel and 32 bit userland

** Abandon i386 to VMs and emulation

Security support may well be a problem in any event.

> Also asking because this came up over the weekend when working with the Debian
> Images team - no one has real UEFI hardware for i386 and it's becoming harder
> and hader to justify spending too much time on testing of the images as fewer
> and fewer machines can benefit from them.
> 

Most i386 only hardware is obsolescent: there are some numbers of 64 bit 
processor 32 bit firmware, limited memory netbooks out there limited to 4G 
memory

Numbers from popcon may include VMS and emulation.

Some good people on debian-user have been kind enough to test this on real 
i686 hardware: live CDs and the Calamares installer have problems 
working on low memory machines.

> What are your thoughts, collectively? [I did ask one of you as an individual
> but he suggested respectfully and correctly that I should ask you all - thanks
> to him for the polite response].
> 

Have I outlined the alternatives above correctly from my reading of the list?

Are we any closer to a resolution or suggestion of how best to move forward?

> All the very best to you all and with thanks, as ever, for all your work
> 
> Andy C.
> 
And a happy new year to all as well :)

Andy C

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Christian Kastner]

On 15.12.20 01:55, Russ Allbery wrote:
> Increasingly most of the people who work on Debian don't have i386
> hardware lying around, particularly i386 hardware that requires an i386
> kernel or that exercises the full range of older boot processes.  If you
> do, testing and reporting good bugs would probably be helpful.

FWIW, most people probably have amd64 hardware around, and can therefore
use KVM-accelerated i386 emulation using QEMU. That emulation can be
configured with a fine grain, down to CPU models/features.

And until at least a few years ago, that emulation was quite realistic.
For my bachelor's thesis, I looked into portability of binaries, and I
used autopkgtest + QEMU to hunt for bugs where the buildd environment
affected the build (for example, by picking up CPU flags of the buildd
machine).

I found #781998 like that (i386 binary assuming SSE is present), and
confirmed it on real hardware.

So, technically, I think i386 is quite easy to test. To me, even easier
than arm64, where I need to get useful hardware first.

Using QEMU, it's trivial to build packages for i386 on amd64 (using
sbuild, or the qemu-sbuild-tools wrapper I'm dabbling on, which will be
absorbed by sbuild soon), and autopkgtest using QEMU has been trivial
since forever.

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Paul Wise]

On Mon, Dec 14, 2020 at 11:36 PM Adrian Bunk wrote:

> A bigger worry for i386 would be the availability of microcode updates

This is also a big problem for amd64, since only the newest
generations of Intel processors get BIOS/UEFI and or microcode
updates, so lots of amd64 users (including myself) do not have
microcode updates.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Russ Allbery]

Calum McConnell  writes:

> A very fair point, and quite equitably put.  If I was remotely
> comfortable tweaking kernels, or used a 32 bit machine regularly, I
> would be more comfortable volunteering.  As it is, I have only really
> learned to maintain packages in the past few months, and I feel nowhere
> near confident enough about my future to make a three-year commitment.

It sounds like from Adrien's message that one of the missing pieces is
more people testing d-i regularly on i386 hardware to confirm that it
works properly.  That's something that doesn't require much kernel
tweaking, and may be a quick way to help.

Increasingly most of the people who work on Debian don't have i386
hardware lying around, particularly i386 hardware that requires an i386
kernel or that exercises the full range of older boot processes.  If you
do, testing and reporting good bugs would probably be helpful.

It sounds like there are a fair number of people want the i386
architecture to survive, which is great.  That probably means the
resources are there.  One of the things a porter does is coordinate the
effort, so people who are willing to invest time into that coordination
can help even if they don't think they can fix tricky kernel bugs.

> But I would like to say that, while it is a significant workload, it
> isn't one that isn't being done.  There is still only one porter, it's
> true, but that's also currently the case for ppc64el, and s390x has no
> confirmed porters.

My intuition is also that i386, although becoming less popular, was
starting from such a huge install base that the resources are probably out
there somewhere.

> Further, unless "sudden death of most porters" can be added to the list
> of bad events of 2020, I feel confident in saying that there are still
> probably some more people who simply haven't gotten around to confirming
> that they can be a porter.

I agree.  Most of my point is just that they should do that.  :)  Now's
the time.

> While I agree that i386 kernel support should be phased out, and might
> even need to be dropped altogether, I strongly disagree with the
> original premise of this thread, that all i386 support should be dropped
> for Bullseye.

I may be able to reassure you a bit there.  Someone pointing out that we
don't have enough confirmed resources for a port happens semi-regularly,
and the usual outcome is that enough resources step forward.  We're not
very eager to drop things that people want to support.  The point is to
prod people into stepping forward and volunteering for the things they
care about.

What's perhaps more significant is that i386 is now getting to the point
where it requires such prodding, instead of being an assumed default
architecture.  That means that the folks who care about it should probably
start thinking about building more organization and structure around the
work, recruiting people, building a task list, and so forth, instead of
just assuming "oh, everything will work on i386, it always has."
Volunteering to do that sort of coordination is helpful even if you aren't
debugging FTBFS problems.

-- 
Russ Allbery (r...@debian.org)  

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Adrian Bunk]

On Mon, Dec 14, 2020 at 02:54:37PM -0800, Russ Allbery wrote:
> 
> The quantity of hardware is useful data, but I think this is also a place
> where it's important to stress the specific problem that Debian has,
> namely that we need people to do the work.
>...

The list of Debian release architecture that did have a non-zero number 
of porters committed by the bullseye porter roll call deadline is 
exactly the list of release architectures not supported by Ubuntu.

Among the architectures also supported by Ubuntu,
none had a porter committed for bullseye.

cu
Adrian

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Calum McConnell]

On Mon, 2020-12-14 at 14:54 -0800, Russ Allbery wrote:
> Calum McConnell  writes:
> > The point I'm making is that i386 processors are still incredibly
> > common, and we shouldn't abandon their users.
> 
> Not abandoning users is a powerful motivating force, but it still has to
> succeed in motivating people.  Debian can't make a decision on the basis
> of not abandoning users.  We have to make a decision on the basis that
> someone is volunteering to do the work.  Perhaps they're volunteering to
> do that work so that we're not abandoning users, and that's great, but
> that additional step is important.
> 
> I think it's therefore useful in this sort of thread to be very clear
> whether your conclusion is "and therefore I am volunteering to do the
> work to keep i386 alive" or whether your conclusion is "and therefore I
> am asking other people to volunteer to keep i386 alive," and be aware
> that the latter may not be successful because volunteer time is a
> limited resource and there are many worthy things that we could all be
> working on to make the lives of users better.

A very fair point, and quite equitably put.  If I was remotely comfortable
tweaking kernels, or used a 32 bit machine regularly, I would be more
comfortable volunteering.  As it is, I have only really learned to
maintain packages in the past few months, and I feel nowhere near
confident enough about my future to make a three-year commitment.

But I would like to say that, while it is a significant workload, it isn't
one that isn't being done.  There is still only one porter, it's true, but
that's also currently the case for ppc64el, and s390x has no confirmed
porters.  In fact, no architecture has any more than 2 porters. Plus, in
other areas i386 is in a better position than most: it has more archive
coverage than any other (non-amd64) port, and still has good upstream
support. 

Further, unless "sudden death of most porters" can be added to the list of
bad events of 2020, I feel confident in saying that there are still
probably some more people who simply haven't gotten around to confirming
that they can be a porter.  Every port but arm64 has less than half the
porters that it had for Buster, and many have a third or a fourth the
people. So while it's true that I am asking others to give up their time,
without backing it up with my own commitment as Johannes has, I feel that
it is a request that will be met.  

> The reason why I'm focusing on the kernel is because the upstream kernel
> developers have been signaling rather strongly for a while that i386 is
> a semi-deprecated architecture that you should avoid running if you can,
> and the amount of resources and attention that it is getting are
> steadily dropping.  Maybe the resources and attention it gets are still
> something we consider "good enough" (although we're already at the point
> where if you care about kernel security, you should put serious effort
> into getting onto the amd64 kernel even if you keep an i386 userspace),
> but at some point it seems likely they will no longer be.  That means it
> may be time to push our users a bit harder to switch to the amd64 kernel
> if they can.

I agree, but I don't think we are yet at that point where dropping support
should be an issue.  If the debate was whether i386 should be maintained
as an LTS architecture [1], I would be more willing to agree.  Perhaps for
Bookworm, a reasonable compromise would be to drop security support for
i386 kernels.  But that discussion is at least a year down the road. While
the kernel upstream may be discouraging i386 users, it is still (mostly)
supported by them for the time being: as long as that remains the case, I
don't think we should drop the i386 kernel.

While I agree that i386 kernel support should be phased out, and might
even need to be dropped altogether, I strongly disagree with the original
premise of this thread, that all i386 support should be dropped for
Bullseye.  There is still a massive library of software that is only
available as 32-bit: indeed, in my experience it has only been in the past
few years that 64-bit has been the default.  While keeping old binaries
running isn't the responsibility of Debian, I do think that, after 20
years of recommending i386 as the most compatible target for code, we
should try to support them at least a little longer.

[1]: I mean a true LTS, not just the three years mentioned in the original
subject



signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Russ Allbery (2020-12-14 23:54:37)
> > The point I'm making is that i386 processors are still incredibly common,
> > and we shouldn't abandon their users.
> 
> Not abandoning users is a powerful motivating force, but it still has to
> succeed in motivating people.  Debian can't make a decision on the basis
> of not abandoning users.  We have to make a decision on the basis that
> someone is volunteering to do the work.  Perhaps they're volunteering to
> do that work so that we're not abandoning users, and that's great, but
> that additional step is important.
> 
> I think it's therefore useful in this sort of thread to be very clear
> whether your conclusion is "and therefore I am volunteering to do the work
> to keep i386 alive" or whether your conclusion is "and therefore I am
> asking other people to volunteer to keep i386 alive," and be aware that
> the latter may not be successful because volunteer time is a limited
> resource and there are many worthy things that we could all be working on
> to make the lives of users better.
> 
> If we can confirm that the volunteer resources are there, we can ask what
> they need from the rest of the project to be successful.

I cannot donate my time (I'm also lacking the skill) but I'm willing to put my
money somewhere if it means to keep i386 alive for longer. I privately own
perfectly working old hardware that I would hate to send to the landfill just
because software support is ending.  Should i386 be discontinued I should
probably only keep using the hardware in a separate network disconnected from
the internet but that would make the hardware much less useful.

If somebody could direct me to organizations or people who just need financial
support to keep i386 alive, that would be great. In the light of a planet with
limited resources, I think it's worth my money to help keeping old hardware
around and avoid the waste of resources and energy that manufacturing new
equipment incurs.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Adrian Bunk]

On Mon, Dec 14, 2020 at 01:22:11PM +0100, Ben Hutchings wrote:
> On Sun, 2020-12-13 at 01:53 -0800, Steve Langasek wrote:
> [...]
> > While the ongoing
> > costs of maintaining a full port were a consideration, of equal concern was
> > the fact that we believed we would not be able to provide security support
> > for the architecture as a whole at par with other architectures, due to,
> > among other things, lack of adequate support from the upstream
> > kernel/toolchain community.  I'm not sure if i386 has caught up and now has
> > adequate mitigation for Spectre etc, but it definitely wasn't available on
> > an equivalent timeline as amd64.
> 
> I agree that kernel security support for i386 is seriously lacking.
> 
> The Spectre mitigations were actually available for both x86
> architectures at the same time, but the initial Meltdown mitigation was
> amd64-specific and was not extended to i386 until Linux 4.19.  The
> implementation used in stable kernel branches (KAISER) was sufficiently
> different from that used upstream, that i386 support has not been added
> to it.

If using Spectre/Meltdown as metric, how does kernel security support 
for architectures like arm64 or ppc64el compare to kernel security 
support for i386?

When it comes to security support, i386 often has the benefit that code 
is shared with amd64 so fixes are available early (like for Spectre).

I am not saying that there was no problem on i386, but if this was meant 
to register a security concern for release architectures we have to look 
at all architectures.

> As a result, stretch:i386 is still vulnerable when running the default
> (4.9-based) kernel.

A bigger worry for i386 would be the availability of microcode updates 
for Spectre, but this has little practical impact as long as noone cares 
enough about Spectre to start a GR that would allow us to not leave our 
amd64 users vulnerable by default even in bullseye.

> Ben.

cu
Adrian

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Russ Allbery]

Calum McConnell  writes:

> As I showed in my (slightly over dramatic, very over-long) email this
> morning, there are more people with i386 kernels than there are total
> users of every other release architecture.  Even if you only look at
> non-pae kernels, its still about double the total installs for any two
> other release architectures.

The quantity of hardware is useful data, but I think this is also a place
where it's important to stress the specific problem that Debian has,
namely that we need people to do the work.  That's both packaging work
inside Debian and enough upstream work that the software we're packaging
remains supportable on i386.

> The point I'm making is that i386 processors are still incredibly
> common, and we shouldn't abandon their users.

Not abandoning users is a powerful motivating force, but it still has to
succeed in motivating people.  Debian can't make a decision on the basis
of not abandoning users.  We have to make a decision on the basis that
someone is volunteering to do the work.  Perhaps they're volunteering to
do that work so that we're not abandoning users, and that's great, but
that additional step is important.

I think it's therefore useful in this sort of thread to be very clear
whether your conclusion is "and therefore I am volunteering to do the work
to keep i386 alive" or whether your conclusion is "and therefore I am
asking other people to volunteer to keep i386 alive," and be aware that
the latter may not be successful because volunteer time is a limited
resource and there are many worthy things that we could all be working on
to make the lives of users better.

If we can confirm that the volunteer resources are there, we can ask what
they need from the rest of the project to be successful.

The reason why I'm focusing on the kernel is because the upstream kernel
developers have been signaling rather strongly for a while that i386 is a
semi-deprecated architecture that you should avoid running if you can, and
the amount of resources and attention that it is getting are steadily
dropping.  Maybe the resources and attention it gets are still something
we consider "good enough" (although we're already at the point where if
you care about kernel security, you should put serious effort into getting
onto the amd64 kernel even if you keep an i386 userspace), but at some
point it seems likely they will no longer be.  That means it may be time
to push our users a bit harder to switch to the amd64 kernel if they can.

-- 
Russ Allbery (r...@debian.org)  

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Calum McConnell]

On Mon, 2020-12-14 at 10:02 -0800, Russ Allbery wrote:

> One possible intermediate option shy of dropping the i386 architecture
> would be to drop the i386 kernel and instead help all i386 installs
> switch
> to the amd64 kernel while still running i386 binaries.  (That said, this
> will obviously not work on actual i386 hardware, and I don't know if it
> has other issues that I personally happened not to notice.)

As I showed in my (slightly over dramatic, very over-long) email this
morning, there are more people with i386 kernels than there are total
users of every other release architecture.  Even if you only look at non-
pae kernels, its still about double the total installs for any two other
release architectures.

Since (AFAIK) there is a substantial speed penalty to installing a non-pae
kernel on a -pae processor, I don't see there being a lot of users running
that.  Further, because installations of old distributions still report
popcon information, I can also tell you that the i486, which is even older
than the i586 that was dropped in Stretch, and was deprecated in 2014,
still has more installs than all but 3 of the release architectures:
armel, armhf, amd64, and i386.  

Keep in mind, too, that these are wildly unfair comparisons: I'm comparing
"people with x specific kernel package" to "people with some package from
that architecture".  The point I'm making is that i386 processors are
still incredibly common, and we shouldn't abandon their users.




signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Russ Allbery]

Ben Hutchings  writes:

> I agree that kernel security support for i386 is seriously lacking.

> The Spectre mitigations were actually available for both x86
> architectures at the same time, but the initial Meltdown mitigation was
> amd64-specific and was not extended to i386 until Linux 4.19.  The
> implementation used in stable kernel branches (KAISER) was sufficiently
> different from that used upstream, that i386 support has not been added
> to it.

> As a result, stretch:i386 is still vulnerable when running the default
> (4.9-based) kernel.

It may be worth separating the kernel from the rest of the distribution.
Switching an existing i386 deploy to use the amd64 kernel is fairly easy.
I did that on my legacy i386 installations quite some time ago, and thus
am running a kernel with proper upstream security support.  It's far
easier and less intimidating than crossgrading the rest of the system to
amd64.

One possible intermediate option shy of dropping the i386 architecture
would be to drop the i386 kernel and instead help all i386 installs switch
to the amd64 kernel while still running i386 binaries.  (That said, this
will obviously not work on actual i386 hardware, and I don't know if it
has other issues that I personally happened not to notice.)

-- 
Russ Allbery (r...@debian.org)  

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Ben Hutchings]

On Sun, 2020-12-13 at 01:53 -0800, Steve Langasek wrote:
[...]
> While the ongoing
> costs of maintaining a full port were a consideration, of equal concern was
> the fact that we believed we would not be able to provide security support
> for the architecture as a whole at par with other architectures, due to,
> among other things, lack of adequate support from the upstream
> kernel/toolchain community.  I'm not sure if i386 has caught up and now has
> adequate mitigation for Spectre etc, but it definitely wasn't available on
> an equivalent timeline as amd64.

I agree that kernel security support for i386 is seriously lacking.

The Spectre mitigations were actually available for both x86
architectures at the same time, but the initial Meltdown mitigation was
amd64-specific and was not extended to i386 until Linux 4.19.  The
implementation used in stable kernel branches (KAISER) was sufficiently
different from that used upstream, that i386 support has not been added
to it.

As a result, stretch:i386 is still vulnerable when running the default
(4.9-based) kernel.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.


signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Steve Langasek]

On Sat, Dec 12, 2020 at 06:09:02PM +0200, Adrian Bunk wrote:
> > Ubuntu have chosen to support the first use-case, and only the first
> > use-case. They longer ship a complete, bootable i386 operating system;
> > instead, they have an i386 second-class-citizen architecture that
> > is sufficient to provide graphics drivers and other shared libraries
> > for legacy 32-bit proprietary binaries.
> >...

> Ubuntu has a business-minded focus, which is fair enough.
> But Debian should not blindly follow whatever Canonical
> does with Ubuntu for business reasons.[3]

> It does make sense for Debian to differenciate by providing support for 
> communities whose hardware is not or no longer supported by Ubuntu.

It's obviously entirely appropriate for Debian to make its own decision here
regarding what they want to support, but FTR the dropping of i386 was
largely not a "business" focused decision for Ubuntu.  While the ongoing
costs of maintaining a full port were a consideration, of equal concern was
the fact that we believed we would not be able to provide security support
for the architecture as a whole at par with other architectures, due to,
among other things, lack of adequate support from the upstream
kernel/toolchain community.  I'm not sure if i386 has caught up and now has
adequate mitigation for Spectre etc, but it definitely wasn't available on
an equivalent timeline as amd64.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Calum McConnell]

Hi all,
As someone who runs amd64/i386 multiarch, this statement from Adrian:


> i386 hardware is so numerous and widely spread, that every tiny fraction
> of i386 users might be more users than half of our release architectures
> combined. It is not even clear whether this is just an exaggeration or 
> might be literally true:

intrested me.  I wondered just how many there were.  Popcon lists 17281
people with i386 installations, but I bet that includes those who (like
me) installed multiarch.  So I grep'ed through the popcon output a bit,
looking for kernel packages.  I figure that, if you have an i386 kernel
pacakge, you don't belong in the first group of people.

Obviously you all can easily replicate this, and this only applies to
users with popularity-contest installed, but here are my results:

For a baseline, there are 181,863 amd64 users who are regularally sending
popcon reports.  Of those, 171,916 have the linux-image-amd64 package
installed.  I assume the remaining 5.4 percent are selecting what kernel
package they are running manually, or perhaps are in a VM.

The 13th most popular linux-image package is linux-image-686-pae, at
12,736 installs.  It places ahead of every single 5.x kernel, indicating
that there are more people running i386 (with some extensions) than there
are running Testing or Unstable.  

Continuing down the list, the standard linux-image-686 package (no PAE)
has 877 popcon installs.  None of the other release archetecures have
appeared yet: which isn't supprising, since every other popcon archetecure
has a combined total of 1636 installs, the largest being armhf at 636
installs.  I assume these people are the ones who would lose support:
while some of them may have PAE capable computers, I don't think it's a
significant fraction.

Clearly, I have already proved Adrian's point: I can say, with certainty,
that there are an order of magnitude more people with i386 kernels (and
thus presumably i386 hardware) than there are for every other non-amd64
release archetecture combined.  Further, there are more people with old
i386 hardware than there are for any other arch.  My point is that we
would lose less people if we dropped all ARM support than if we dropped
the oldest supported i386 kernel[1].

But lets keep going!  See, we haven't seen any arm kernal images yet, so
who knows if they even exist?  Remember, the ARM archectures are the
biggest ones after i386.

Next up, we hit linux-image-586, with 403 installs.  That means there are
403 people who were unable to upgrade to stretch, but are still running
Debian and popcon.  That's presumably the lower limit for the number
Adrian referenced as people who were upset with the increase in baseline,
since again, all of those 403 people have used their 586 machine in the
past month.

Continuing down, we see linux-image-486, 310 installs.  That's still more
installations than arm64's total popcon.  It's also been unsupported since
2014, but hey.   

Then we hit linux-image-marvell, which (as I understand it) is one of the
arm versions.  At 225 installs, its not terribly impressive.  Its also the
first non-amd64/i386 kernel that I hit on this list, and where I stop. 
This is supported as a first-class Debian citizen: and yet, the now
dropped 486 still has more installations.

Of course, the pace of technology marches on, and the 586 is an ancient
chip.  We were right to end support for it: it's not like any modern
software would run well on such a processor.  But there is still a large
section of the debian userbase using the older 686 versions.  Adrian is
right to say that ending support for them isn't right.

Wall of text meticulously analyzing the output of two commands aside, this
was a bit fun to make!  Now I'm off to bed in my bed: thanks for reading! 

Calum M

[1]: Okay, that's not strictly true: the total number of people reporting
packages from each of the arm architectures is 1256.  However, that
involves three seperate sub-archetecures, and I am willing to bet there
are a fair number of multi-arch arm users.  But for strict correctness,
pretend I said "all armhf and arm64 support", since those two total to
only 10 more than the subset of i386 in question.


signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[peter green]

   Then there was the short netbook boom, but AFAIR some early ones
   had 64bit CPUs but 32bit-only firmware.

My memory is that at the height of the boom the dominant processors
were the N270 and N280, which are 32-bit only. By the time 64-bit
netbook processors showed up the boom was on the decline.


There are at least two more:


5. People running Debian on virtual machines.

You can run an i386 VM with vmware or virtualbox with no special
hardware support. An x86-64 VM on the other hand requires VT-x
(or the AMD equivilent). While processor support for this is
the norm nowadays it's still often disabled by default
which can be a pain if you need to get IT support to access
bios setup on a machine.

i386 hardware is so numerous and widely spread, that every tiny fraction 
of i386 users might be more users than half of our release architectures 
combined. It is not even clear whether this is just an exaggeration or 
might be literally true:


i386 still gives 17281 popcon submissions, that is about
a tenth of amd64, but it's also over 10 times the next highest port

Now that probably doesn't reflect true usage, in particular
users who install using images tend to miss out on the popcon
question, but I still suspect that i386 is up there in the top
few most used architectures.

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Simon McVittie]

On Sat, 12 Dec 2020 at 18:09:02 +0200, Adrian Bunk wrote:
> 3. Computers that do support MMX and SSE2, but do not support 64bit.

Right, that's basically the second use-case I mentioned, but moving the
boundary for what we do and don't support to be more modern. We could
put the boundary anywhere we want, with higher baselines letting us make
more assumptions but excluding more old hardware.

> 4. People who wrongly installed i386 on amd64-capable hardware.

You're right that this doesn't match either of the use-cases I gave. If
this one is important to us, then that's a reason to keep a complete
bootable i386 system (with bootloaders and kernels), but we could move
its baseline as high as amd64's.

> Ubuntu has a business-minded focus, which is fair enough.
> But Debian should not blindly follow whatever Canonical
> does with Ubuntu for business reasons.[3]
> [3] neither should Debian blindly do the opposite

I agree - we need to weigh up the same advantages and disadvantages
that Canonical/Ubuntu did, but we might come to a different conclusion
because our priorities (and infrastructure) are different.

smcv

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Adrian Bunk]

On Tue, Dec 08, 2020 at 05:46:26PM +, Simon McVittie wrote:
>...
> I think it's necessary to consider what the purpose of the i386 port is,
> and set expectations and an appropriate baseline based on that.
> 
> I see two possible use-cases for i386:
> 
> 1. It's a compatibility layer for legacy 32-bit binaries on x86_64 CPUs:
>in particular, proprietary 32-bit binaries that we cannot recompile,
>32-bit builds of Wine, and the dependency stacks for those
> 
> 2. It's something you can genuinely run on older (or more-embedded)
>32-bit x86 CPUs that do not support x86_64 (down to some arbitrary
>baseline, currently i686 without MMX or SSE)

There are at least two more:

3. Computers that do support MMX and SSE2, but do not support 64bit.
   AFAIR the last 32bit-only CPU from Intel was released in 2007.[1]
   Then there was the short netbook boom, but AFAIR some early ones
   had 64bit CPUs but 32bit-only firmware.
   Surprisingly many netbooks are still in use even in first-world 
   countries.[2]

4. People who wrongly installed i386 on amd64-capable hardware.
   The existing userbase with this setup is large, and even though
   crossgrades are now finally possible it is better to wait until
   there are fewer users in this setup and more potential crossgrade 
   issues resolved.

> Ubuntu have chosen to support the first use-case, and only the first
> use-case. They longer ship a complete, bootable i386 operating system;
> instead, they have an i386 second-class-citizen architecture that
> is sufficient to provide graphics drivers and other shared libraries
> for legacy 32-bit proprietary binaries.
>...

Ubuntu has a business-minded focus, which is fair enough.
But Debian should not blindly follow whatever Canonical
does with Ubuntu for business reasons.[3]

It does make sense for Debian to differenciate by providing support for 
communities whose hardware is not or no longer supported by Ubuntu.

>...
> we could raise the
> baseline to include those and stop patching packages to avoid using them,
> which would remove i386's major oddity when compared with other
> architectures (i387 extended-precision floating point).

While this specific oddity is unique to the i386 port,
it is worth mentioning that every port has its own oddities.

> Also, if we are only supporting i386 for my first use-case, then I think
> we should seriously consider waiving the archive-completeness metric
> for i386: if big packages like web browsers and Libreoffice can't be
> compiled on 32-bit, then so be it. We only need to be able to compile
> a library stack, plus enough programs to be able to build and test that
> library stack on virtual or real hardware.

There is also a cost associated with having to modify packages for 
handling such port-specific omissions.

One current example for missing archive-completeness would be s390x,
and I guess you are aware how much pain its headless nature has caused 
in some places.

> If the second use-case is supported, then we are going to need an i386
> porter team analogous to any other architecture porting team, that can
> take responsibility for choosing an achievable baseline for the oldest
> i386 CPU that they intend to test and support, triaging i386-specific
> bugs, and providing patches where necessary to keep packages below that
> baseline (which will require an increasing amount of work over time if
> they choose a baseline that is suitable for historical x86 CPUs, since
> increasingly many upstreams refuse to support the i387 FPU). I don't
> think it's reasonable any more to expect individual package maintainers
> to understand the finer points of the historical x86 instruction set.

This is correct.

This is exactly porter work, and this is what I have already been doing 
for years for i386.

A large part of porter work is being a one-trick pony, often submitting
the same fix for many packages. For i386 this is usually some variant of
ifneq (,$(filter $(DEB_HOST_ARCH), i386))
export DEB_CXXFLAGS_MAINT_APPEND += -ffloat-store
endif

> One factor that makes the second use-case difficult to support is
> that even if developers still have old 32-bit x86 hardware, it's very
> likely to be considerably newer than our current baseline. For example,
> mainstream Intel and AMD 32-bit CPUs gained SSE support around 20 years
> ago (Pentium III and Athlon XP). I know there are somewhat newer x86
> embedded CPUs with lesser capabilities, but most developers would have
> to have deliberately chosen to obtain those, rather than having access
> to old machines just because they haven't disposed of them yet.

It is not realistic to expect porters to have hardware matching exactly 
the port baseline.

How many people do have hardware matching exactly our amd64 baseline?

Does hardware matching exactly our amd64 baseline even exist?

> I don't think it's realistic to drop i386 completely, and I want to
> be able to continue to run legacy 32-bit games; so if an i386 porter

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Simon McVittie]

(Please cc me, I'm not subscribed.)

Andrew M.A. Cater wrote:
> Having participated in the current discussion about 32 bit releases and
> lifetimes in Linux Weekly News (lwn.net) - what's the status of i386 for the
> lifetime of Bullseye?
>
> There seems to be only one maintainer.
>
> Is i386 going to be supportable for the next 3 1/2 years and buildable for
> that long (given that almost all machines are now 64 bit capable and we're
> having to build some packages on amd64 for i386 - per ballombe)?

I think it's necessary to consider what the purpose of the i386 port is,
and set expectations and an appropriate baseline based on that.

I see two possible use-cases for i386:

1. It's a compatibility layer for legacy 32-bit binaries on x86_64 CPUs:
   in particular, proprietary 32-bit binaries that we cannot recompile,
   32-bit builds of Wine, and the dependency stacks for those

2. It's something you can genuinely run on older (or more-embedded)
   32-bit x86 CPUs that do not support x86_64 (down to some arbitrary
   baseline, currently i686 without MMX or SSE)

Ubuntu have chosen to support the first use-case, and only the first
use-case. They longer ship a complete, bootable i386 operating system;
instead, they have an i386 second-class-citizen architecture that
is sufficient to provide graphics drivers and other shared libraries
for legacy 32-bit proprietary binaries. Debian infrastructure doesn't
currently support second-class-citizen, multiarch-only architectures that
are not independently bootable, but until we do, we could approximate
this by booting an i686 bootloader, kernel and library stack on x86_64
bare metal (as long as it supports BIOS boot), or more likely in practice,
on x86_64 virtual machines - which in practice is how we do i386 buildds
and other infrastructure already. The next step towards what Ubuntu do
would be to require 32-bit i386 user-space to be run on an x86_64 kernel,
similar to the way the s390 port used to work (if I remember correctly).

If it's only the first use-case that is supported, then our current
baseline for i386 is unnecessarily pessimistic, and indeed harmful to
performance and consistency with other architectures: all x86_64 CPUs are
required to offer MMX, SSE and other nice things, so we could raise the
baseline to include those and stop patching packages to avoid using them,
which would remove i386's major oddity when compared with other
architectures (i387 extended-precision floating point).

Also, if we are only supporting i386 for my first use-case, then I think
we should seriously consider waiving the archive-completeness metric
for i386: if big packages like web browsers and Libreoffice can't be
compiled on 32-bit, then so be it. We only need to be able to compile
a library stack, plus enough programs to be able to build and test that
library stack on virtual or real hardware.

If the second use-case is supported, then we are going to need an i386
porter team analogous to any other architecture porting team, that can
take responsibility for choosing an achievable baseline for the oldest
i386 CPU that they intend to test and support, triaging i386-specific
bugs, and providing patches where necessary to keep packages below that
baseline (which will require an increasing amount of work over time if
they choose a baseline that is suitable for historical x86 CPUs, since
increasingly many upstreams refuse to support the i387 FPU). I don't
think it's reasonable any more to expect individual package maintainers
to understand the finer points of the historical x86 instruction set.

One factor that makes the second use-case difficult to support is
that even if developers still have old 32-bit x86 hardware, it's very
likely to be considerably newer than our current baseline. For example,
mainstream Intel and AMD 32-bit CPUs gained SSE support around 20 years
ago (Pentium III and Athlon XP). I know there are somewhat newer x86
embedded CPUs with lesser capabilities, but most developers would have
to have deliberately chosen to obtain those, rather than having access
to old machines just because they haven't disposed of them yet.

I don't think it's realistic to drop i386 completely, and I want to
be able to continue to run legacy 32-bit games; so if an i386 porter
team doesn't materialize otherwise, I'd be willing to help to maintain a
version of i386 that is intended to be a compatibility library stack for
x86_64 machines (similar in scope to what Ubuntu does). However, I am not
willing to spend my time on making packages use i387 in preference to SSE.

smcv

Release status of i386 for Bullseye and long term support for 3 years?

[Andrew M.A. Cater]

Dear release team

Having participated in the current discussion about 32 bit releases and 
lifetimes in Linux Weekly News (lwn.net) - what's the status of i386 for the
lifetime of Bullseye?

There seems to be only one maintainer.

Is i386 going to be supportable for the next 3 1/2 years and buildable for
that long (given that almost all machines are now 64 bit capable and we're
having to build some packages on amd64 for i386 - per ballombe)?

Also asking because this came up over the weekend when working with the Debian
Images team - no one has real UEFI hardware for i386 and it's becoming harder
and hader to justify spending too much time on testing of the images as fewer
and fewer machines can benefit from them.

What are your thoughts, collectively? [I did ask one of you as an individual
but he suggested respectfully and correctly that I should ask you all - thanks
to him for the polite response].

All the very best to you all and with thanks, as ever, for all your work

Andy C.