Re: Upcoming changes to Debian Linux kernel packages

[Bastian Blank]

On Thu, Oct 05, 2023 at 07:59:54AM -0600, Sam Hartman wrote:
> I think that's what you mean by the first-level error.
> If not, I'm still confused.
> In the second level error case you are talking about is:

No, the first level is always: but the new kernel does not work.
The second is: I need to upgrade external modules.

> I think what you are saying is that
> 
> 1) the current system is fragile: sometimes you want a kernel headers
> that is not available and sometimes you have version skew between the
> kernel headers and kernel even though you have both installed.
> 
> 2) In your system, fewer things are possible, but the combination that
> is possible is more likely to work.

Yes.

> And I think people's response is that
> they care enough about some of the things you are breaking that they are
> willing to accept the fragility.

For now it looks like a better solution is to just create more meta
packages and accept that they become uninstallable from time to time.

In the future we might want to split off the modules into it's own
package anyway.  That will then allow
- a different image package containing prebuilt UKI,
- a different modules package to replace the special cloud flavours.

Regards,
Bastian

-- 
Without facts, the decision cannot be made logically.  You must rely on
your human intuition.
-- Spock, "Assignment: Earth", stardate unknown

Re: Upcoming changes to Debian Linux kernel packages

[Russ Allbery]

Sam Hartman  writes:

> B) They might already have headers installed.  Imagine someone who
> installs headers at the same time they install the kernel.  Unless they
> managed to upgrade the same version of their kernel without also
> upgrading their headers, they will still have headers.  They can still
> build modules against that kernel, either because they installed a new
> dkms package, or because one of their dkms packages upgraded.

I am also really confused by this discussion and don't entirely understand
the motivation for the proposed change to kernel headers, but isn't the
situation Sam describes above the normal way Linux kernel headers work and
have worked for years?  Kernels come with headers matching the same
version, if you want headers for external modules you install both
packages at the same time via one mechanism or another, and you only
remove the kernel and headers when you're pretty sure you will never use
that kernel again.

When I was using external modules heavily, I routinely kept three or four
old kernels and their corresponding headers installed at the same time so
that I could easily downgrade if I ran into regressions without having to
track down old packages that may have been removed from the archive.  This
feels like a normal and somewhat obvious Debian systems administration
thing to do to me.

I realize in the new signing regime every new kernel would have a separate
headers package (as opposed to today where the kernel and headers are
updated in place with the same package name if there is no ABI change),
but to me this doesn't feel like a significant difference for users.  I
haven't been paying close attention, so maybe I'm wrong, but I feel like
most kernel package updates have been ABI updates anyway, particularly in
stable.

-- 
Russ Allbery (r...@debian.org)  

Re: Upcoming changes to Debian Linux kernel packages

[Sam Hartman]

> "Bastian" == Bastian Blank  writes:

Bastian> The same as now: nowhere, because those packages have been
Bastian> removed from the archive already.

Bastian> And sadly you did not answer the question why a second
Bastian> degree error must not be worse then a worked around first
Bastian> degree error?

I'll admit that I find that phrasing difficult enough that I had to read
it a couple of times and I'm still not sure I've got it.

Let me see if I understand what you are saying.

1)  kernel headers will be removed from the archive.  So people complain
if they have an old kernel and wish to get kernel headers for it, but
those headers have been removed.

2) If the kernel header version changes but the kernel header package
name does not change  (version changed but not uname -r in the new
scheme?), you can have what you think is the right kernel headers but
they do not work with the binaries you have installed.

3) you can run into trouble between testing and backports.

I think that's what you mean by the first-level error.
If not, I'm still confused.

In the second level error case you are talking about is:

1) there is a regression in the kernel

2) someone needs kernel headers for an old kernel they want to downgrade
to.

I don't actually see how this is a second level error.
It appears to be a different first-level error (a regression), and  the
downgrade appears to follow naturally from that.

You point out that someone may not be able to get kernel headers for the
downgraded kernel.

a) They might.  In the window between a new kernel being introduced into
unstable and the same kernel being introduced into testing  there is an
old kernel available with kernel headers.
Similarly if someone needs to downgrade as far as stable, there is a
kernel with headers available in stable.

B) They might already have headers installed.  Imagine someone who
installs headers at the same time they install the kernel.
Unless they managed to upgrade the same version of their kernel without
also upgrading their headers, they will still have headers.
They can still build modules against that kernel, either because they
installed a new dkms package, or because one of their dkms packages
upgraded.

I think what you are saying is that

1) the current system is fragile: sometimes you want a kernel headers
that is not available and sometimes you have version skew between the
kernel headers and kernel even though you have both installed.

2) In your system, fewer things are possible, but the combination that
is possible is more likely to work.

And I think people's response is that
they care enough about some of the things you are breaking that they are
willing to accept the fragility.


Bastian> -- Each kiss is as the first.  -- Miramanee, Kirk's wife,
Bastian> "The Paradise Syndrome", stardate 4842.6

Re: Upcoming changes to Debian Linux kernel packages

[Bastian Blank]

On Tue, Oct 03, 2023 at 03:00:53PM -0500, Robert Nelson wrote:
> On Tue, Oct 3, 2023 at 2:54 PM Adrian Bunk  wrote:
> > How will the user get the headers matching this previously-used kernel
> > that are required until we provide a kernel with the regression fixed?

The same as now: nowhere, because those packages have been removed from
the archive already.

And sadly you did not answer the question why a second degree error must
not be worse then a worked around first degree error?

> I know there is a lot of history behind the linux-headers package in
> debian.  However since 5.2 there is a kernel config option, which
> allows you to build the kernel headers as a module (built-in or
> external)..
> https://cateee.net/lkddb/web-lkddb/IKHEADERS.html
> As long as this was enabled (ignore bugs/regressions), users can go
> back and forth on kernel versions as they wish.

If it would be so easy.  This would include all the generated things of
the build.  But it still needs all the static headers, all the support
binaries and scripts (shipped as linux-kbuild-*), which also change with
every version.

Regards,
Bastian

-- 
Each kiss is as the first.
-- Miramanee, Kirk's wife, "The Paradise Syndrome",
   stardate 4842.6

Re: Upcoming changes to Debian Linux kernel packages

[Bastian Blank]

Hi Andreas

On Tue, Oct 03, 2023 at 11:58:29PM +0200, Andreas Beckmann wrote:
> That should solve the problem where several source packages need to be
> updated together.

The problem does not come from multiple source packages that need to be
updated together.  Instead it comes from the way Debian does signing of
secure boot components.  After the linux packages got built and accepted,
an automatic process takes the output and produces a new source package
that is uploaded, built and accepted.  So the signed stuff will always
come later (between hours or days in normal cases, but esp for backports
even more then a week later).

Regards,
Bastian

-- 
Insults are effective only where emotion is present.
-- Spock, "Who Mourns for Adonais?"  stardate 3468.1

Re: Upcoming changes to Debian Linux kernel packages

[Andreas Beckmann]

On 03/10/2023 19.30, Bastian Blank wrote:

thread.  Or freak out because meta packages remain uninstallable in
backports for days.

...

plus gcc or we change how backports works.


If uninstallable packages in backports are a problem, perhaps backports 
needs something like britney to migrate packages from an uploading area 
to the publishing area (once a package set is installable, no need for 
delays). That should solve the problem where several source packages 
need to be updated together. (It would also solve problems with 
unsatisfiable dependencies where the backporter-uploaded binaries were 
built by accident in sid instead of stable, requiring a binNMU before 
migration.)


(but this should be discussed elsewhere)


Andreas

Re: Upcoming changes to Debian Linux kernel packages

[Robert Nelson]

On Tue, Oct 3, 2023 at 2:54 PM Adrian Bunk  wrote:
>
> On Tue, Oct 03, 2023 at 07:30:49PM +0200, Bastian Blank wrote:
> >...
> > The core problem is that people assume they can get headers matching the
> > currently running kernel, without upgrading first, see also the parallel
> > thread.
> >...
>
> If the new kernel has a regression that affects the user, the user
> usually has no good choice other than downgrading to the previously-used
> kernel and continue using it until a fixed kernel is available.
>
> How will the user get the headers matching this previously-used kernel
> that are required until we provide a kernel with the regression fixed?
>

I know there is a lot of history behind the linux-headers package in
debian.  However since 5.2 there is a kernel config option, which
allows you to build the kernel headers as a module (built-in or
external)..

https://cateee.net/lkddb/web-lkddb/IKHEADERS.html

As long as this was enabled (ignore bugs/regressions), users can go
back and forth on kernel versions as they wish.

Regards,

-- 
Robert Nelson
https://rcn-ee.com/

Re: Upcoming changes to Debian Linux kernel packages

[Adrian Bunk]

On Tue, Oct 03, 2023 at 07:30:49PM +0200, Bastian Blank wrote:
>...
> The core problem is that people assume they can get headers matching the
> currently running kernel, without upgrading first, see also the parallel
> thread.
>...

If the new kernel has a regression that affects the user, the user 
usually has no good choice other than downgrading to the previously-used 
kernel and continue using it until a fixed kernel is available.

How will the user get the headers matching this previously-used kernel
that are required until we provide a kernel with the regression fixed?

> Regards,
> Bastian

cu
Adrian

Re: Upcoming changes to Debian Linux kernel packages

[Bastian Blank]

Hi Sam

On Tue, Oct 03, 2023 at 08:31:57AM -0600, Sam Hartman wrote:
> I still think it would help if you would work more on articulating what
> problem you are trying to solve with the linux-headers versioning
> change.  I have read multiple versions of this proposal, and your
> follow-ups, and I still do not understand what is prompting the
> linux-headers change.

The core problem is that people assume they can get headers matching the
currently running kernel, without upgrading first, see also the parallel
thread.  Or freak out because meta packages remain uninstallable in
backports for days.

With this plan you can only get the correct ones by using something
I think like:

| apt satisfy 'linux-headers (= $(uname -r))'

There is somewhere again a maybe possible plan to get meta packages in
place that actually support the case of always providing the headers to
the installed (not running!) kernel.

Then we need to use the same versioning anyway again.  In the end I
don't really care, but we need then a way to fix the version skew
between the different source package for the kernel.  Aka either redo
larger parts of the linux package (which can never fix it completely),
plus gcc or we change how backports works.

> My intuition mirrors others in the conversation that it is problematic
> to support multiple kernel versions without also supporting multiple
> header versions.

Usually you try to guard against one error.  Noone claimed that we can't
work with one error.  All the other conversations already have to argue
with two errors at the same time.  When should we stop then?

Regards,
Bastian

-- 
Deflector shields just came on, Captain.

Re: Upcoming changes to Debian Linux kernel packages

[Sam Hartman]

> "Bastian" == Bastian Blank  writes:

Bastian> On Mon, Sep 25, 2023 at 04:35:08AM +0200, Andreas Beckmann wrote:
>> On 25/09/2023 00.50, Bastian Blank wrote:
>> > Already built modules remain until someone deletes it.  So you
>> can also > switch back to the still installed older kernel
>> version and it will have > the still working module available.
>> Assume I have Linux 6.6 and a third-party gpu driver module
>> installed (so there are dkms and the Linux 6.6 headers as well)
>> and everything is working fine.  Then I upgrade the system, which
>> brings Linux 6.7 (along linux-image-6.6 which is kept installed)
>> and a new version of the gpu driver (which adds support for
>> 6.7). So the old gpu module for 6.6 gets removed and a new one is
>> built for 6.7 only (since there are only 6.7 headers now).

Bastian> Ah, here lays the missconception.  No, the 6.6 ones are not
Bastian> removed.  Why should they be?  The system knows it can't
Bastian> rebuild them.

Bastian> If the current implementation would remove them, it is a
Bastian> problem there, not in the concept.

I still think it would help if you would work more on articulating what
problem you are trying to solve with the linux-headers versioning
change.  I have read multiple versions of this proposal, and your
follow-ups, and I still do not understand what is prompting the
linux-headers change.

My intuition mirrors others in the conversation that it is problematic
to support multiple kernel versions without also supporting multiple
header versions.

Re: Upcoming changes to Debian Linux kernel packages

[Bastian Blank]

On Mon, Sep 25, 2023 at 04:35:08AM +0200, Andreas Beckmann wrote:
> On 25/09/2023 00.50, Bastian Blank wrote:
> > Already built modules remain until someone deletes it.  So you can also
> > switch back to the still installed older kernel version and it will have
> > the still working module available.
> Assume I have Linux 6.6 and a third-party gpu driver module installed (so
> there are dkms and the Linux 6.6 headers as well) and everything is working
> fine.
> Then I upgrade the system, which brings Linux 6.7 (along linux-image-6.6
> which is kept installed) and a new version of the gpu driver (which adds
> support for 6.7). So the old gpu module for 6.6 gets removed and a new one
> is built for 6.7 only (since there are only 6.7 headers now).

Ah, here lays the missconception.  No, the 6.6 ones are not removed.  Why
should they be?  The system knows it can't rebuild them.

If the current implementation would remove them, it is a problem there,
not in the concept.

Bastian

-- 
Superior ability breeds superior ambition.
-- Spock, "Space Seed", stardate 3141.9

Re: Upcoming changes to Debian Linux kernel packages

[M. Zhou]

On Mon, 2023-09-25 at 04:35 +0200, Andreas Beckmann wrote:
> On 25/09/2023 00.50, Bastian Blank wrote:
> > Already built modules remain until someone deletes it.  So you can
> > also
> > switch back to the still installed older kernel version and it will
> > have
> > the still working module available.
> 
> This is what I expect not to work.
> 
> Assume I have Linux 6.6 and a third-party gpu driver module installed
> (so there are dkms and the Linux 6.6 headers as well) and everything
> is 
> working fine.
> Then I upgrade the system, which brings Linux 6.7 (along linux-image-
> 6.6 
> which is kept installed) and a new version of the gpu driver (which
> adds 
> support for 6.7). So the old gpu module for 6.6 gets removed and a
> new 
> one is built for 6.7 only (since there are only 6.7 headers now).
> Unfortunately 6.7 breaks some exotic in-tree driver (which I
> desperately 
> need), so I need to go back to 6.6. Oops, there is no gpu driver
> module 
> any more. Recovery now needs manual intervention.

Same concern here. We cannot pose strong assumption on the user's
upgrade path. The said scenario may happen for any dkms package
when the newer kernel version is not supported.

> I'm not sure which class of bugs you are trying to solve with this 
> proposed unversioned linux-headers change. IMO the current scheme of 
> linux-headers-$version-$abi-$flavor matching 
> linux-image-$version-$abi-$flavor works well. But perhaps something 
> could be improved on the metapackage side. Ideally a user should
> install 
> either meta-linux-image-without-headers-$flavor OR 
> meta-linux-image-with-headers-$flavor (and ideally installing dkms 
> should "automatically switch" to the with-headers variant, not sure
> how 
> this could be done). The current scheme of having to install 
> linux-image-$flavor AND linux-headers-$flavor is a bit tricky.
> I'm open to implement improvements on the dkms side.

I could not understand the benefit of it neither. Apart from the dkms
part, the user-customized kernel packages cannot be omitted as well.

For instance, if I build a customized kernel from debian's kernel
source, using `make bindeb-pkg`, I get those:

linux-headers-6.5.3_6.5.3-2_amd64.deb
linux-image-6.5.3_6.5.3-2_amd64.deb
linux-libc-dev_6.5.3-2_amd64.deb

Currently they are well integrated into the system, and IIRC dkms
also works for them. If versioning is gone, how to make it
compatible with user's local kernel package? There must be two
copies of kernel headers in the system in this case because we
cannot remove user's local customized headers on our own.

Then the design still has to support multi version co-existence.

Re: Upcoming changes to Debian Linux kernel packages

[Andreas Beckmann]

On 25/09/2023 00.50, Bastian Blank wrote:

Already built modules remain until someone deletes it.  So you can also
switch back to the still installed older kernel version and it will have
the still working module available.


This is what I expect not to work.

Assume I have Linux 6.6 and a third-party gpu driver module installed 
(so there are dkms and the Linux 6.6 headers as well) and everything is 
working fine.
Then I upgrade the system, which brings Linux 6.7 (along linux-image-6.6 
which is kept installed) and a new version of the gpu driver (which adds 
support for 6.7). So the old gpu module for 6.6 gets removed and a new 
one is built for 6.7 only (since there are only 6.7 headers now).
Unfortunately 6.7 breaks some exotic in-tree driver (which I desperately 
need), so I need to go back to 6.6. Oops, there is no gpu driver module 
any more. Recovery now needs manual intervention.


I'm not sure which class of bugs you are trying to solve with this 
proposed unversioned linux-headers change. IMO the current scheme of 
linux-headers-$version-$abi-$flavor matching 
linux-image-$version-$abi-$flavor works well. But perhaps something 
could be improved on the metapackage side. Ideally a user should install 
either meta-linux-image-without-headers-$flavor OR 
meta-linux-image-with-headers-$flavor (and ideally installing dkms 
should "automatically switch" to the with-headers variant, not sure how 
this could be done). The current scheme of having to install 
linux-image-$flavor AND linux-headers-$flavor is a bit tricky.

I'm open to implement improvements on the dkms side.

Andreas

PS: the proposed "more versioning in the linux-image packages" will 
solve some rare dkms issues where modules didn't get rebuilt after 
linux-headers-* was upgraded but $(uname -r) didn't change

Re: Upcoming changes to Debian Linux kernel packages

[Bastian Blank]

Hi Andreas

On Sun, Sep 24, 2023 at 11:10:36PM +0200, Andreas Beckmann wrote:
> On 24/09/2023 15.01, Bastian Blank wrote:
> > ## Kernel modules will be signed with an ephemeral key
> > 
> > The modules will not longer be signed using the Secure Boot CA like the
> > EFI kernel image itself.  Instead a key will be created during the build
> > and thrown away after.
> 
> Do I correctly assume that change only affects the modules shipped by the
> linux-image packages and not third-party modules built with dkms?

Yes.  Nothing calls for changes to MOK keys, which are used by dkms.

> > ## Header and tool packages will not longer contain version
> 
> > This means that only headers of one single version can be available on
> > the system at one time.  This might be a bit inconvinient for dkms, as
> > it can't longer build modules for multiple versions.
> 
> That sounds problematic in case of third party modules. If it is possible to
> have multiple linux-image-* packages installed, but only headers for one of
> them, the third-party modules will only be available for one of the kernel
> versions for sure (maybe there are still old module builds available, but no
> guarantee especially after the third-party module got updated). This will
> make switching between different kernel versions difficult to impossible,
> e.g. it may be hard to go back to a working older kernel version in case the
> new one does not work properly (or the third-party module cannot be built or
> does not work for the new version).

Already built modules remain until someone deletes it.  So you can also
switch back to the still installed older kernel version and it will have
the still working module available.

Yes, you would not be able to build new modules for the older kernel
until you also install the matching headers.

> Regarding getting the correct linux-header-* packages installed for the
> installed linux-image-* packages:
> Maybe linux-image-* could have
>   Recommends: linux-headers-* | no-linux-headers
> s.t. the correct linux-headers-* are installed by default (installation of
> recommends is enabled by default) for all installed linux-image-* packages.
> no-linux-headers would be an opt-out package that can be installed manually
> if someone does not want to get linux-headers-* installed at all. It should
> never be installed automatically.

Nack.  I actually thought about that.  But third-party modules are too
much a special configuration to do that and pay the 50MiB or so penalty
for each system.  Also this still have the version skew problem between
linux and linux-signed-*, so will be unreliable.

> For dkms it is hard recommend the correct linux-header-* package, right now
> we have
>   Recommends: linux-headers-generic | linux-headers-686-pae |
> linux-headers-amd64 | linux-headers
> which does not really work for the non-default kernel flavor, e.g. the
> -cloud or -i386 kernel. So some improvement on the kernel side would be nice
> here.

I thought about adding a versioned provides with the complete kernel
release string as version, so something like
| Provides: linux-headers (= $(uname -r))

This can then be installed via apt-get and the correct version as long
as the package is available.  This however can't be done via
dependencies, because it is dynamic.  So dkms would need to actively
make sure it got the correct package, if they are still reachable at
all.

Bastian

-- 
We have found all life forms in the galaxy are capable of superior
development.
-- Kirk, "The Gamesters of Triskelion", stardate 3211.7

Re: Upcoming changes to Debian Linux kernel packages

[Andreas Beckmann]

On 24/09/2023 15.01, Bastian Blank wrote:

## Kernel modules will be signed with an ephemeral key

The modules will not longer be signed using the Secure Boot CA like the
EFI kernel image itself.  Instead a key will be created during the build
and thrown away after.


Do I correctly assume that change only affects the modules shipped by 
the linux-image packages and not third-party modules built with dkms?



## Header and tool packages will not longer contain version



This means that only headers of one single version can be available on
the system at one time.  This might be a bit inconvinient for dkms, as
it can't longer build modules for multiple versions.


That sounds problematic in case of third party modules. If it is 
possible to have multiple linux-image-* packages installed, but only 
headers for one of them, the third-party modules will only be available 
for one of the kernel versions for sure (maybe there are still old 
module builds available, but no guarantee especially after the 
third-party module got updated). This will make switching between 
different kernel versions difficult to impossible, e.g. it may be hard 
to go back to a working older kernel version in case the new one does 
not work properly (or the third-party module cannot be built or does not 
work for the new version).



Regarding getting the correct linux-header-* packages installed for the 
installed linux-image-* packages:

Maybe linux-image-* could have
  Recommends: linux-headers-* | no-linux-headers
s.t. the correct linux-headers-* are installed by default (installation 
of recommends is enabled by default) for all installed linux-image-* 
packages. no-linux-headers would be an opt-out package that can be 
installed manually if someone does not want to get linux-headers-* 
installed at all. It should never be installed automatically.


For dkms it is hard recommend the correct linux-header-* package, right 
now we have
  Recommends: linux-headers-generic | linux-headers-686-pae | 
linux-headers-amd64 | linux-headers
which does not really work for the non-default kernel flavor, e.g. the 
-cloud or -i386 kernel. So some improvement on the kernel side would be 
nice here.



Andreas