Re: Firmware GR result - what happens next?

[Michael Stone]

On Fri, Oct 14, 2022 at 10:52:01AM +0200, Santiago Ruano Rincón wrote:

5. transitional packages along with a helper package (that fails or
success during install) to prompt the user so they add non-free-firmware
section when needed.

Is there any reason why you are not considering 5.?


The danger we're trying to avoid is that a system with a working 
"something" (say, networking) gets upgraded, user reboots (or machine 
crashes, or there's a power failure, etc, etc.), the working "something" 
is now a not-working "something", and fixing it is really hard for the 
user who has no idea what happened and maybe doesn't have a network or a 
console or whatever any more. A package that fails during install will 
prevent the upgrade from completing, but will leave things in an 
in-between state until some action is taken, the upgrade restarted, and 
the upgrade manages to finish successfully. What happens if the 
reboot/crash/powercycle/etc happens during that in-between state? How do 
you make a firmware helper package that reliably prevents a kernel 
installation when the kernel doesn't have any dependencies on the 
firmware package, and also doesn't yank out the old working firmware, 
etc. I'm sure you can make the install explode, but making it reliably 
explode at just the right time seems harder. I guess this could all 
work, but I'm seeing a lot of potential for partial installs/failures 
with this approach and I suspect this would require transition code in a 
number of packages' preinsts, not a discrete "helper package".

Re: Firmware GR result - what happens next?

[Steve McIntyre]

Hi Pascal,

On Fri, Oct 14, 2022 at 09:57:37PM +0200, Pascal Hambourg wrote:
>On 14/10/2022 at 21:49, Holger Wansing wrote:
>> 
>> Pascal Hambourg  wrote (Fri, 14 Oct 2022 21:31:31 
>> +0200):
>> > > > > As far as I understand it, they can be packaged once their
>> > > > > redistributability is clarified?
>> > > 
>> > > The only way for the installer to make use of such firmware is, if the
>> > > user provides the firmware files on a removable device, like an USB 
>> > > stick.
>> > 
>> > This is my point. If Debian does not provide all firmware for drivers
>> > which are included into the installer, then support for extra firmware
>> > medium is still useful and should not be removed.
>> 
>> For this specific case (use of firmware files, which are not available in
>> Debian) there is no need for a special installer medium.
>
>I did not mention the need for any special installer medium, only the need
>for the installer to support extra firmware medium.
>
>> The usual installer also has the capability to make use of firmware
>> provided via USB.
>
>It was suggested in this thread to remove this feature.

ACK, that was our thought. Thanks for pointing out the issue here,
it's appreciated!

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"... the premise [is] that privacy is about hiding a wrong. It's not.
 Privacy is an inherent human right, and a requirement for maintaining
 the human condition with dignity and respect."
  -- Bruce Schneier

Re: Firmware GR result - what happens next?

[Bernd Zeimetz]

On Sun, 2022-10-02 at 12:26 -0700, Steve Langasek wrote:
> 
> I heartily endorse ubuntu-release-upgrader, it has been useful in addressing
> uncounted upgrade issues over the years and I think something like this
> would be a nice addition to Debian as well.  Two caveats:

That thing is actually one of the main reasons I'm not using Ubuntu.

I expect properly maintained and upgradable packages, and not a hacky thing.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: Firmware GR result - what happens next?

[Holger Wansing]

Hi,

Pascal Hambourg  wrote (Fri, 14 Oct 2022 21:57:37 
+0200):
> I did not mention the need for any special installer medium, only the 
> need for the installer to support extra firmware medium.
> 
> > The usual installer also has the capability to make use of firmware
> > provided via USB.
> 
> It was suggested in this thread to remove this feature.

Hmm, ok. 
You talked about an "extra firmware medium".
I mixed that up with the "extra installer images with firmware included".

Sorry for the noise

Holger


-- 
Holger Wansing 
PGP-Fingerprint: 496A C6E8 1442 4B34 8508  3529 59F1 87CA 156E B076

Re: Firmware GR result - what happens next?

[Pascal Hambourg]

On 14/10/2022 at 21:49, Holger Wansing wrote:


Pascal Hambourg  wrote (Fri, 14 Oct 2022 21:31:31 
+0200):

As far as I understand it, they can be packaged once their
redistributability is clarified?


The only way for the installer to make use of such firmware is, if the
user provides the firmware files on a removable device, like an USB stick.


This is my point. If Debian does not provide all firmware for drivers
which are included into the installer, then support for extra firmware
medium is still useful and should not be removed.


For this specific case (use of firmware files, which are not available in
Debian) there is no need for a special installer medium.


I did not mention the need for any special installer medium, only the 
need for the installer to support extra firmware medium.



The usual installer also has the capability to make use of firmware
provided via USB.


It was suggested in this thread to remove this feature.

Re: Firmware GR result - what happens next?

[Cyril Brulebois]

Holger Wansing  (2022-10-14):
> Pascal Hambourg  wrote (Fri, 14 Oct 2022
> > This is my point. If Debian does not provide all firmware for
> > drivers which are included into the installer, then support for
> > extra firmware medium is still useful and should not be removed.
> 
> For this specific case (use of firmware files, which are not available
> in Debian) there is no need for a special installer medium.  The usual
> installer also has the capability to make use of firmware provided via
> USB.

That's really Pascal's point: we talked about maybe removing that.

(It's hard to get right for end-users.)


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Holger Wansing]

Hi,

Pascal Hambourg  wrote (Fri, 14 Oct 2022 21:31:31 
+0200):
> >>> As far as I understand it, they can be packaged once their
> >>> redistributability is clarified?
> > 
> > The only way for the installer to make use of such firmware is, if the
> > user provides the firmware files on a removable device, like an USB stick.
> 
> This is my point. If Debian does not provide all firmware for drivers 
> which are included into the installer, then support for extra firmware 
> medium is still useful and should not be removed.

For this specific case (use of firmware files, which are not available in 
Debian) there is no need for a special installer medium.
The usual installer also has the capability to make use of firmware
provided via USB.


Holger



-- 
Holger Wansing 
PGP-Fingerprint: 496A C6E8 1442 4B34 8508  3529 59F1 87CA 156E B076

Re: Firmware GR result - what happens next?

[Pascal Hambourg]

On 14/10/2022 at 20:54, Holger Wansing wrote:


Pascal Hambourg  wrote (Fri, 14 Oct 2022 19:50:11 
+0200):

On 13/10/2022 at 10:13, Cyril Brulebois wrote:

Pascal Hambourg  (2022-10-13):

What about firmware not available in Debian packages ? For example
Prims54 wireless adapters using p54pci and p54usb drivers included
in nic-wireless-modules-*-di need firmware unavailable in Debian
packages.


As far as I understand it, they can be packaged once their
redistributability is clarified?


The only way for the installer to make use of such firmware is, if the
user provides the firmware files on a removable device, like an USB stick.


This is my point. If Debian does not provide all firmware for drivers 
which are included into the installer, then support for extra firmware 
medium is still useful and should not be removed.

Re: Firmware GR result - what happens next?

[Holger Wansing]

Hi,

Pascal Hambourg  wrote (Fri, 14 Oct 2022 19:50:11 
+0200):
> On 13/10/2022 at 10:13, Cyril Brulebois wrote:
> > Pascal Hambourg  (2022-10-13):
> >> What about firmware not available in Debian packages ? For example
> >> Prims54 wireless adapters using p54pci and p54usb drivers included
> >> in nic-wireless-modules-*-di need firmware unavailable in Debian
> >> packages.
> > 
> > As far as I understand it, they can be packaged once their
> > redistributability is clarified?

The only way for the installer to make use of such firmware is, if the 
user provides the firmware files on a removable device, like an USB stick.
This is documented here:
https://d-i.debian.org/manual/en.amd64/ch06s04.html

How and where to get the files, is not where Debian can help with in this case.


Holger


-- 
Holger Wansing 
PGP-Fingerprint: 496A C6E8 1442 4B34 8508  3529 59F1 87CA 156E B076

Re: Firmware GR result - what happens next?

[Pascal Hambourg]

On 13/10/2022 at 10:13, Cyril Brulebois wrote:

Pascal Hambourg  (2022-10-13):

What about firmware not available in Debian packages ? For example
Prims54 wireless adapters using p54pci and p54usb drivers included
in nic-wireless-modules-*-di need firmware unavailable in Debian
packages.


As far as I understand it, they can be packaged once their
redistributability is clarified?


What if some firmware is not redistributable in the end ?

The above example is for old hardware and the download URLs are 
mentioned in the Debian wiki, so I assumed that the reason why it was 
not packaged was because it was not redistributable. But maybe I was 
wrong and nobody just cared.


Also, I assume that packages such as firmware-b43legacy-installer and 
firmware-b43-installer which download Broadcom drivers and extract the 
firmware from them would not need to exist if the firmware was known to 
be redistributable. But I may be wrong again.

Re: Firmware GR result - what happens next?

[David Kalnischkies]

On Fri, Oct 14, 2022 at 01:32:41PM +0800, Paul Wise wrote:
> On Thu, 2022-10-13 at 17:35 +0200, Julien Cristau wrote:
> > I'd prefer if we could make things work vs making things fail,
> > however loudly.
> 
> There seem to be a few ways to deal with this transition:

(quotes reordered in Pauls preference order mentioned at the end)

> 4. Keep all non-free-firmware packages in non-free too. This would be
> backwards compatible, but may expose bugs in dak, debian-cd, apt and
> other tools, so IIRC this has been vetoed by the archive and CD teams.
> This also wouldn't result in users without non-free getting any of the
> non-free firmware, which maybe we want since it is the new default?

libapt or probably most user-facing client are no concern in this as
packages coming from multiple sources is normal. If you have e.g.
unstable and testing in your sources.list it is happening all the time,
or you have multiple mirrors, … only by the virtue of having packages
installed and somewhere online a client needs a way to figure out if the
version installed is the same as (one of) the versions online and which
online sources talk about the same version (naively, you would think
version number is enough, but libapt actually goes beyond that).

I assume through that in servers and other tools who are not usually
exposed to packages in multiple versions or multiple sources in general
have a harder time with this, so I can understand them being reluctant.

There are also a lot of tools… most user-facing clients will be based
on libapt or at least directly inspired by it, but if clients like
(c)debootstrap expect a package name to be no longer unique in their
world view (after all, they don't even do multi-sources like -updates
and -security …) I honestly don't know and fear the worst.


It is also that if we decide that, we basically decide that it will stay
that way forever. Its also an (arguably tiny) waste of diskspace and
such for everyone who has both components configured.

It is the only solution treating everyone equal though. All the others
talk only about sources.list with no idea if and how e.g. debootstrap
needs to understand that non-free-firmware is a thing now. Does it?


> 3. Add some code to apt to download non-free-firmware when non-free is
> available in the sources and the downloaded Release files. This would
> solve the issue for Debian and all other derivatives too, if they
> decide to add a new non-free-firmware component too. This might not
> be accepted by apt developers as it is kind of a hack to special-case
> Debian component semantics in apt, although maybe a component mapping
> config option would be accepted. This might result in extra Debian
> support requests when users notice the new component in `apt update`.
> This might not work for users of tools not based on apt, like cupt?
> This wouldn't result in users without non-free getting any non-free
> firmware though, which maybe we want since it is the new default?

The problem with this within libapt is that libapt wants to look at the
sources.list entry and produce a list of files which could possibly
belong to this entry. The Release file is just one of those files.
It is consulted later to remove entries from the file list (e.g. in an
'update'), but files aren't added at that stage.

Think of 'apt update --print-uris': What would that print if it would
need to look at the Release first? If you look closely, you will notice
e.g. a line talking about 'binary-all/Packages'. The Release file will
later tell apt to not download it for Debian. Similar, depending on your
locale environment apt talks about downloading Translation files here
which don't exist or perhaps even of Contents files which are in an
other location than in reality for some distros (hello Ubuntu).

So, if we wanted that, I could hardcode in apt that it should look for
non-free-firmware, too, if it sees non-free as a component configured
and decide later on not to download that component if it doesn't exist –
on the upside (depending on your view) that isn't Debian specific.
In practice it would probably turn out to be a medium sized patch as
so far apt doesn't have the concept of an implicit component, but it
shouldn't be rocket science and easily done ahead of the freeze.
It would probably be too big for a backport through and even if very
unlikely in practice a behaviour change as suddenly grabbing an existing
non-free-firmware component and upgrades from it after an unattended
upgrade in an unattended upgrade is… that rules out availability of
the bookworm-version of firmware packages in the upgrade to bookworm.
They would be installed only with the first 'update && upgrade' cycle
on the upgraded bookworm system, potentially unattended. Most other
"solutions" have the same "problem" – I would hope that for firmware
packages it isn't really one in practice as they should be very light on
(rev-)dependencies and demands upon them.

We are potentially burning one 

Re: Firmware GR result - what happens next?

[Holger Levsen]

On Fri, Oct 14, 2022 at 01:32:41PM +0800, Paul Wise wrote:
> 4. Keep all non-free-firmware packages in non-free too. This would be
> backwards compatible, but may expose bugs in dak, debian-cd, apt and
> other tools, so IIRC this has been vetoed by the archive and CD teams.
> This also wouldn't result in users without non-free getting any of the
> non-free firmware, which maybe we want since it is the new default?
> 
> Personally I would choose 4 first, I expect any potential issues could
> be easily fixed before the freeze. 

but what would you then propose after the release, where we would have
exactly the same situation as we would have now. carry this on forever?


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

When this virus is over, I still want some of you all to stay away from me.


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Marvin Renich]

* Paul Wise  [221014 01:35]:
> On Thu, 2022-10-13 at 17:35 +0200, Julien Cristau wrote:
> 
> > I'd prefer if we could make things work vs making things fail,
> > however loudly.
> 
> There seem to be a few ways to deal with this transition:
> 
> 2. Add some code somewhere to automatically modify the apt sources,
> somehow ensure that code is run by all Debian users and hope that other
> automated processes (like ansible/puppet) don't overwrite those changes
> and hope that users aren't storing apt sources config in packages,
> which would mean conffile prompts after the modification happens.

Actually, I think this would work really well.  Do not modify any
existing sources.list; drop a new file in sources.list.d.  This file can
have a default name that includes "firmware-nonfree" so is highly
unlikely to exist, but if it does, add "-NNN" suffix with the smallest
numeric NNN that does not exist.

Of course, the file would only be added if the current sources do not
include that component, which would _really_ decrease the probability of
a file name conflict to be about the same as the probability of the
earth being destroyed by an asteroid.

...Marvin

Re: Firmware GR result - what happens next?

[Peter Pentchev]

On Fri, Oct 14, 2022 at 10:52:01AM +0200, Santiago Ruano Rincón wrote:
> El 14/10/22 a las 13:32, Paul Wise escribió:
> > On Thu, 2022-10-13 at 17:35 +0200, Julien Cristau wrote:
> > 
> > > I'd prefer if we could make things work vs making things fail,
> > > however loudly.
> > 
> > There seem to be a few ways to deal with this transition:
> > 
> > 1. Document it in the release notes and let users handle it. This means
> > lots of users won't get security updates for firmware (which are mostly
> > only issued for x86 CPU microcode), since lots of folks won't read the
> > release notes. This also means lots of support requests when users
> > can't find the firmware package they wanted.
> > 
> > 2. Add some code somewhere to automatically modify the apt sources,
> > somehow ensure that code is run by all Debian users and hope that other
> > automated processes (like ansible/puppet) don't overwrite those changes
> > and hope that users aren't storing apt sources config in packages,
> > which would mean conffile prompts after the modification happens.
> > 
> > 3. Add some code to apt to download non-free-firmware when non-free is
> > available in the sources and the downloaded Release files. This would
> > solve the issue for Debian and all other derivatives too, if they
> > decide to add a new non-free-firmware component too. This might not
> > be accepted by apt developers as it is kind of a hack to special-case
> > Debian component semantics in apt, although maybe a component mapping
> > config option would be accepted. This might result in extra Debian
> > support requests when users notice the new component in `apt update`. 
> > This might not work for users of tools not based on apt, like cupt?
> > This wouldn't result in users without non-free getting any non-free
> > firmware though, which maybe we want since it is the new default?
> > 
> > 4. Keep all non-free-firmware packages in non-free too. This would be
> > backwards compatible, but may expose bugs in dak, debian-cd, apt and
> > other tools, so IIRC this has been vetoed by the archive and CD teams.
> > This also wouldn't result in users without non-free getting any of the
> > non-free firmware, which maybe we want since it is the new default?
> > 
> > Personally I would choose 4 first, I expect any potential issues could
> > be easily fixed before the freeze. Next I would choose 3. Next I would
> > choose 1 because I think /etc belongs to the sysadmin not packages.
> 
> 5. transitional packages along with a helper package (that fails or
> success during install) to prompt the user so they add non-free-firmware
> section when needed.
> 
> Is there any reason why you are not considering 5.?

Do you mean having packages with the same names, but different versions
(even if only Debian revisions) and totally different contents, and also
built from different source packages, in different sections of the same
suite in the archive?... I'm... I'm not sure this would work... I think that
others already expressed some doubts as to how dak would handle that.

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Santiago Ruano Rincón]

El 14/10/22 a las 13:32, Paul Wise escribió:
> On Thu, 2022-10-13 at 17:35 +0200, Julien Cristau wrote:
> 
> > I'd prefer if we could make things work vs making things fail,
> > however loudly.
> 
> There seem to be a few ways to deal with this transition:
> 
> 1. Document it in the release notes and let users handle it. This means
> lots of users won't get security updates for firmware (which are mostly
> only issued for x86 CPU microcode), since lots of folks won't read the
> release notes. This also means lots of support requests when users
> can't find the firmware package they wanted.
> 
> 2. Add some code somewhere to automatically modify the apt sources,
> somehow ensure that code is run by all Debian users and hope that other
> automated processes (like ansible/puppet) don't overwrite those changes
> and hope that users aren't storing apt sources config in packages,
> which would mean conffile prompts after the modification happens.
> 
> 3. Add some code to apt to download non-free-firmware when non-free is
> available in the sources and the downloaded Release files. This would
> solve the issue for Debian and all other derivatives too, if they
> decide to add a new non-free-firmware component too. This might not
> be accepted by apt developers as it is kind of a hack to special-case
> Debian component semantics in apt, although maybe a component mapping
> config option would be accepted. This might result in extra Debian
> support requests when users notice the new component in `apt update`. 
> This might not work for users of tools not based on apt, like cupt?
> This wouldn't result in users without non-free getting any non-free
> firmware though, which maybe we want since it is the new default?
> 
> 4. Keep all non-free-firmware packages in non-free too. This would be
> backwards compatible, but may expose bugs in dak, debian-cd, apt and
> other tools, so IIRC this has been vetoed by the archive and CD teams.
> This also wouldn't result in users without non-free getting any of the
> non-free firmware, which maybe we want since it is the new default?
> 
> Personally I would choose 4 first, I expect any potential issues could
> be easily fixed before the freeze. Next I would choose 3. Next I would
> choose 1 because I think /etc belongs to the sysadmin not packages.

5. transitional packages along with a helper package (that fails or
success during install) to prompt the user so they add non-free-firmware
section when needed.

Is there any reason why you are not considering 5.?

Cheers!

 -- Santiago


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Paul Wise]

On Thu, 2022-10-13 at 17:35 +0200, Julien Cristau wrote:

> I'd prefer if we could make things work vs making things fail,
> however loudly.

There seem to be a few ways to deal with this transition:

1. Document it in the release notes and let users handle it. This means
lots of users won't get security updates for firmware (which are mostly
only issued for x86 CPU microcode), since lots of folks won't read the
release notes. This also means lots of support requests when users
can't find the firmware package they wanted.

2. Add some code somewhere to automatically modify the apt sources,
somehow ensure that code is run by all Debian users and hope that other
automated processes (like ansible/puppet) don't overwrite those changes
and hope that users aren't storing apt sources config in packages,
which would mean conffile prompts after the modification happens.

3. Add some code to apt to download non-free-firmware when non-free is
available in the sources and the downloaded Release files. This would
solve the issue for Debian and all other derivatives too, if they
decide to add a new non-free-firmware component too. This might not
be accepted by apt developers as it is kind of a hack to special-case
Debian component semantics in apt, although maybe a component mapping
config option would be accepted. This might result in extra Debian
support requests when users notice the new component in `apt update`. 
This might not work for users of tools not based on apt, like cupt?
This wouldn't result in users without non-free getting any non-free
firmware though, which maybe we want since it is the new default?

4. Keep all non-free-firmware packages in non-free too. This would be
backwards compatible, but may expose bugs in dak, debian-cd, apt and
other tools, so IIRC this has been vetoed by the archive and CD teams.
This also wouldn't result in users without non-free getting any of the
non-free firmware, which maybe we want since it is the new default?

Personally I would choose 4 first, I expect any potential issues could
be easily fixed before the freeze. Next I would choose 3. Next I would
choose 1 because I think /etc belongs to the sysadmin not packages.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Firmware GR result - what happens next?

[Wouter Verhelst]

On Thu, Oct 13, 2022 at 04:13:57PM +0100, Steve McIntyre wrote:
> On Thu, Oct 13, 2022 at 05:08:57PM +0200, Julien Cristau wrote:
> >On Thu, Oct 06, 2022 at 05:13:22PM +0200, Tobias Frost wrote:
> >> Maybe and idea would to do something like isa-support does for e.g 
> >> sseX-support
> >> on CPUs that does not have that feature: It fails on installation with an 
> >> debconf message, IIRC.
> >> So that would allow something like "new package" | 
> >> "you-need-to-enable-nonfree-firmware-reminder-package"
> >> 
> >Failing on installation is a terrible user experience, let's not, pretty
> >please.
> 
> It's not great, no. Do you have a better suggestion for making sure
> people update sources.list?

We can display a debconf error (which debconf tries really really hard
to show to the user) and then succeed?

Alternatively, the package could install an apt hook that nags the user
every time they run "apt update" or equivalent, and that turns silent if
the updated firmware packages are installed (because of the difference
between "purge" and "remove").

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

Re: Firmware GR result - what happens next?

[Tobias Frost]

On Thu, Oct 13, 2022 at 05:08:57PM +0200, Julien Cristau wrote:
> On Thu, Oct 06, 2022 at 05:13:22PM +0200, Tobias Frost wrote:
> > Maybe and idea would to do something like isa-support does for e.g 
> > sseX-support
> > on CPUs that does not have that feature: It fails on installation with an 
> > debconf message, IIRC.
> > So that would allow something like "new package" | 
> > "you-need-to-enable-nonfree-firmware-reminder-package"
> > 
> Failing on installation is a terrible user experience, let's not, pretty
> please.

I'd prefer failing loudly to failing silently.
 
> Cheers,
> Julien

Re: Firmware GR result - what happens next?

[Julien Cristau]

On Thu, Oct 13, 2022 at 05:17:59PM +0200, Tobias Frost wrote:
> On Thu, Oct 13, 2022 at 05:08:57PM +0200, Julien Cristau wrote:
> > On Thu, Oct 06, 2022 at 05:13:22PM +0200, Tobias Frost wrote:
> > > Maybe and idea would to do something like isa-support does for e.g 
> > > sseX-support
> > > on CPUs that does not have that feature: It fails on installation with an 
> > > debconf message, IIRC.
> > > So that would allow something like "new package" | 
> > > "you-need-to-enable-nonfree-firmware-reminder-package"
> > > 
> > Failing on installation is a terrible user experience, let's not, pretty
> > please.
> 
> I'd prefer failing loudly to failing silently.
>  
I'd prefer if we could make things work vs making things fail, however loudly.

Cheers,
Julien

Re: Firmware GR result - what happens next?

[Steve McIntyre]

On Thu, Oct 13, 2022 at 05:08:57PM +0200, Julien Cristau wrote:
>On Thu, Oct 06, 2022 at 05:13:22PM +0200, Tobias Frost wrote:
>> Maybe and idea would to do something like isa-support does for e.g 
>> sseX-support
>> on CPUs that does not have that feature: It fails on installation with an 
>> debconf message, IIRC.
>> So that would allow something like "new package" | 
>> "you-need-to-enable-nonfree-firmware-reminder-package"
>> 
>Failing on installation is a terrible user experience, let's not, pretty
>please.

It's not great, no. Do you have a better suggestion for making sure
people update sources.list?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Since phone messaging became popular, the young generation has lost the
 ability to read or write anything that is longer than one hundred and sixty
 characters."  -- Ignatios Souvatzis

Re: Firmware GR result - what happens next?

[Julien Cristau]

On Thu, Oct 06, 2022 at 05:13:22PM +0200, Tobias Frost wrote:
> Maybe and idea would to do something like isa-support does for e.g 
> sseX-support
> on CPUs that does not have that feature: It fails on installation with an 
> debconf message, IIRC.
> So that would allow something like "new package" | 
> "you-need-to-enable-nonfree-firmware-reminder-package"
> 
Failing on installation is a terrible user experience, let's not, pretty
please.

Cheers,
Julien

Re: Firmware GR result - what happens next?

[Cyril Brulebois]

Pascal Hambourg  (2022-10-13):
> What about firmware not available in Debian packages ? For example
> Prims54 wireless adapters using p54pci and p54usb drivers included
> in nic-wireless-modules-*-di need firmware unavailable in Debian
> packages.

As far as I understand it, they can be packaged once their
redistributability is clarified?


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Pascal Hambourg]

Hello,

On 08/10/2022 at 18:37, Steve McIntyre wrote:


   * Let's drop the question/support for an extra firmware medium -
 it's not useful any more. We're going to have firmware available
 directly.


What about firmware not available in Debian packages ? For example 
Prims54 wireless adapters using p54pci and p54usb drivers included in 
nic-wireless-modules-*-di need firmware unavailable in Debian packages.

Re: Firmware GR result - what happens next?

[Steve McIntyre]

Hey Jonathan!

On Sun, Oct 09, 2022 at 09:30:55AM +0200, Jonathan Carter (highvoltage) wrote:
>
>On 2022/10/08 18:37, Steve McIntyre wrote:
>>* Is PXE over wifi a thing? Never seen it...
>
>I've been poking around firmware setups of new laptops, and I'm intrigued by
>a new option that at least all the new Dell laptops have, called httpboot. It
>seems that you can just provide a URL to an EFI stub and then it would boot
>that (it's then up to whatever you do in that stub to boot something that can
>set up further networking etc), but it seems like a great thing to support in
>some future Debian release if possible, since you wouldn't even have to write
>as much as a boot USB disk in order to install your system.
>
>I'm not at all sure about where all of it's limitations are, and the
>documentation I could find on Dell's website are minimal, but if I have
>access to such a machine again I'll poke around and experiment a bit and
>write some notes for this list.

Nod. As you ACKed in IRC later, I did mention this in my mail... :-P

It's something that we should definitely be looking at, agreed.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Arguing that you don't care about the right to privacy because you have
 nothing to hide is no different than saying you don't care about free
 speech because you have nothing to say."
   -- Edward Snowden

Re: Firmware GR result - what happens next?

[Jonathan Carter (highvoltage)]

Hi Steve

On 2022/10/08 18:37, Steve McIntyre wrote:

   * Is PXE over wifi a thing? Never seen it...


I've been poking around firmware setups of new laptops, and I'm 
intrigued by a new option that at least all the new Dell laptops have, 
called httpboot. It seems that you can just provide a URL to an EFI stub 
and then it would boot that (it's then up to whatever you do in that 
stub to boot something that can set up further networking etc), but it 
seems like a great thing to support in some future Debian release if 
possible, since you wouldn't even have to write as much as a boot USB 
disk in order to install your system.


I'm not at all sure about where all of it's limitations are, and the 
documentation I could find on Dell's website are minimal, but if I have 
access to such a machine again I'll poke around and experiment a bit and 
write some notes for this list.


-Jonathan

Re: Firmware GR result - what happens next?

[Steve McIntyre]

Hey again folks!

On Sun, Oct 02, 2022 at 03:27:36PM +0100, Steve McIntyre wrote:

...

>We have quite a few things to do now, ideally before the freeze for
>Debian 12 (bookworm), due January 2023 [2]. This list of work items is
>almost definitely not complete, and Cyril and I are aiming to get
>together this week and do more detailed planning for the d-i
>pieces. Off the top of my head I can think of the following:

Cyril and I have just had that planning meeting, and here are the
notes. We're *not* trying to solve every problem here, just working
out the next steps needed for d-i and debian-cd in particular.

Debian firmware changes for d-i and installer images - notes


Steve and Cyril, 2022-10-08

Netboot and firmware - what do we have to do?
-

  * PXE over wired ethernet should just work as-is?
  * Is PXE over wifi a thing? Never seen it...
  * If we need any more than simple wired, we're not going to worry
about it for bookworm
  * This will likely be a problem for audio firmware
* Is netboot a common use case for partially-sighted people?
* Not going to focus on this here just *yet*, maybe we could work
  on it as a later option
* Option to maybe load firmware via PXE/tftp process at early
  kernel startup? Would need to investigate...
  * HTTP(S) boot maybe?

Updating the sources.list of existing systems on upgrade


  * **It's not a problem for d-i / debian-cd, we're not dealing with this here**
  * automation is not likely to happen, we don't do it already
  * **not** something we're trying to solve here
  * having packages in both n-f and n-f-f seems like not a good solution
* worries about dak and others - we've never done this before
  * Maybe move things to n-f-f and leave a transitional package in n-f?
* not sure about this either
  * Let's punt on this decision for now, we can look at it again later if we 
have to

Detection of needed firmware


  * We already had a solution in bullseye which was good enough, let's
keep with that?
* We're trying to solve the 99% case, and we have that now AFAIK?
  * Worry about bonded network handling?
* Maybe tweak interface handling to not take down one half of a
  virtual interface
* Cyril plans to work on VLAN & bonding topics anyway; easy to
  hook it up/down.
  * We *think* things are working ok, but we're not 100%
confident. We've not had complaints, but we don't know if that
just means we don't have users!
  * Let's ask for testing to double-check that the new
firmware-included images work OK - bookworm d-i alpha 2 should be
ready with the nff changes?
* Cyril will test with 2 “d-i laptops” (same as bullseye).

What process should we follow for firmware during d-i?
--

  * Several processes where we may ask for firmware: audio, network,
disks, etc.
  * Do we ask about loading fw at each stage? Or just at the end?
  * Three options via kernel command line? Cyril can take it.
* Allow firmware by default, then just before finish-install we
  add a new screen listing firmware needed, details, write to disk
  on the installed system. Cyril can take this.
  * Maybe: if no firmware is needed then give a "congratulations!"
message. Cyril will not take this.
* Deny all firmware - don't attempt to load things at all, if the
  system is broken then so be it
* Confirm - at each point display a prompt to the user before
  loading fw
* We could **maybe** add support for changing the choice during
  the installation, but let's keep it simple for now. Probably add a
  question in expert mode?
  * What do we do with **free* firmware here? Should we **always**
allow it?
* What happens when we have both non-free and free implementations
  of firmware for given hardware?
* At some point we'll have to make a choice of which to use by
  default, or allow for overriding on the kernel cmd line or
  something.
  * How do we handle sources.list changes (during the course of the
installation process, not on installed systems)? We might need to
enable/disable n-f-f at various times, for both the main archive
and the security archive. Cyril takes it.
  * Even with fw available, we *might* get prompts where some modules
are unclear, or where we may not have the exact suggested firmware
available.
* Don't panic about this too much; we might add blacklists for
  known-awkward modules later. Not a priority here (yet). We
  already have one case implemented: iwl-debug-yoyo.bin (iwlwifi).
  * Let's drop the question/support for an extra firmware medium -
it's not useful any more. We're going to have firmware available
directly.
  * d-i should look in 

Re: Firmware GR result - what happens next?

[Tobias Frost]

On Thu, Oct 06, 2022 at 05:03:20PM +0200, Julien Cristau wrote:
> On Thu, Oct  6, 2022 at 15:45:25 +0100, Steve McIntyre wrote:
> 
> > On Wed, Oct 05, 2022 at 10:11:27PM +0200, Julien Cristau wrote:
> > >On Sun, Oct 02, 2022 at 08:21:31PM +0100, Steve McIntyre wrote:
> > >> On Sun, Oct 02, 2022 at 11:08:47AM -0400, Michael Stone wrote:
> > >> >On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
> > >> >> On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
> > >> >> > What's the plan for upgraded systems with an existing 
> > >> >> > /etc/apt/sources.list.
> > >> >> > Will the new n-f-f section added on upgrades automatically(if 
> > >> >> > non-free was
> > >> >> > enabled before)?
> > >> >> 
> > >> >> So this is the one bit that I don't think we currently have a good
> > >> >> answer for. We've never had a specific script to run on upgrades (like
> > >> >> Ubuntu do), so this kind of potentially breaking change doesn't really
> > >> >> have an obvious place to be fixed.
> > >> >
> > >> >Is there a reason to not continue to make the packages available in 
> > >> >non-free?
> > >> >I don't see a reason to force any change on existing systems.
> > >> 
> > >> Two things:
> > >> 
> > >>  1. I'm worried what bugs we might expose by having packages be in two
> > >> components at once.
> > >>  2. I really don't like the idea of leaving two different
> > >> configurations in the wild; it'll confuse people and is more
> > >> likely to cause issues in the future IMHO.
> > >> 
> > >> Plus, as Shengjing Zhu points out: we already expect people to manage
> > >> the sources.list anyway on upgrades.
> > >> 
> > >I think in the absence of a release upgrade script (which I very much
> > >doubt will happen, and be tested, and we can rely will be used, for
> > >bookworm), Michael's suggestion seems like a reasonable way forward.  I
> > >imagine we'll need to patch dak to allow that, but it seems like it
> > >should be tractable?
> > 
> > I'm also worried what effect this will have on other tools that have
> > to grok the archive (mirror tools, debian-cd, etc.). I'm not going to
> > try and veto having things in more than one component, but (ugh!) I
> > really think it's ugly. Actually, I think I'd much prefer Santiago's
> > idea:
> > 
> > > Couldn't we handle this via transitional firware* non-free packages,
> > > that depend on bookworm non-free-firmware packages?
> > 
> > We'd need to add some transitional binary packages for the small
> > number of n-f-f source packages. That way people would get errors from
> > apt if they don't read our warnings and update. Maybe this is a way
> > forward?
> > 
> I don't think that will work well, the packages will likely just be held
> at the old version if the new ones are uninstallable because the new
> component isn't enabled.

Maybe and idea would to do something like isa-support does for e.g sseX-support
on CPUs that does not have that feature: It fails on installation with an 
debconf message, IIRC.
So that would allow something like "new package" | 
"you-need-to-enable-nonfree-firmware-reminder-package"

> Cheers,
> Julien
> 

Re: Firmware GR result - what happens next?

[Julien Cristau]

On Thu, Oct  6, 2022 at 15:45:25 +0100, Steve McIntyre wrote:

> On Wed, Oct 05, 2022 at 10:11:27PM +0200, Julien Cristau wrote:
> >On Sun, Oct 02, 2022 at 08:21:31PM +0100, Steve McIntyre wrote:
> >> On Sun, Oct 02, 2022 at 11:08:47AM -0400, Michael Stone wrote:
> >> >On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
> >> >> On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
> >> >> > What's the plan for upgraded systems with an existing 
> >> >> > /etc/apt/sources.list.
> >> >> > Will the new n-f-f section added on upgrades automatically(if 
> >> >> > non-free was
> >> >> > enabled before)?
> >> >> 
> >> >> So this is the one bit that I don't think we currently have a good
> >> >> answer for. We've never had a specific script to run on upgrades (like
> >> >> Ubuntu do), so this kind of potentially breaking change doesn't really
> >> >> have an obvious place to be fixed.
> >> >
> >> >Is there a reason to not continue to make the packages available in 
> >> >non-free?
> >> >I don't see a reason to force any change on existing systems.
> >> 
> >> Two things:
> >> 
> >>  1. I'm worried what bugs we might expose by having packages be in two
> >> components at once.
> >>  2. I really don't like the idea of leaving two different
> >> configurations in the wild; it'll confuse people and is more
> >> likely to cause issues in the future IMHO.
> >> 
> >> Plus, as Shengjing Zhu points out: we already expect people to manage
> >> the sources.list anyway on upgrades.
> >> 
> >I think in the absence of a release upgrade script (which I very much
> >doubt will happen, and be tested, and we can rely will be used, for
> >bookworm), Michael's suggestion seems like a reasonable way forward.  I
> >imagine we'll need to patch dak to allow that, but it seems like it
> >should be tractable?
> 
> I'm also worried what effect this will have on other tools that have
> to grok the archive (mirror tools, debian-cd, etc.). I'm not going to
> try and veto having things in more than one component, but (ugh!) I
> really think it's ugly. Actually, I think I'd much prefer Santiago's
> idea:
> 
> > Couldn't we handle this via transitional firware* non-free packages,
> > that depend on bookworm non-free-firmware packages?
> 
> We'd need to add some transitional binary packages for the small
> number of n-f-f source packages. That way people would get errors from
> apt if they don't read our warnings and update. Maybe this is a way
> forward?
> 
I don't think that will work well, the packages will likely just be held
at the old version if the new ones are uninstallable because the new
component isn't enabled.

Cheers,
Julien

Re: Firmware GR result - what happens next?

[Steve McIntyre]

On Wed, Oct 05, 2022 at 10:11:27PM +0200, Julien Cristau wrote:
>On Sun, Oct 02, 2022 at 08:21:31PM +0100, Steve McIntyre wrote:
>> On Sun, Oct 02, 2022 at 11:08:47AM -0400, Michael Stone wrote:
>> >On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
>> >> On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
>> >> > What's the plan for upgraded systems with an existing 
>> >> > /etc/apt/sources.list.
>> >> > Will the new n-f-f section added on upgrades automatically(if non-free 
>> >> > was
>> >> > enabled before)?
>> >> 
>> >> So this is the one bit that I don't think we currently have a good
>> >> answer for. We've never had a specific script to run on upgrades (like
>> >> Ubuntu do), so this kind of potentially breaking change doesn't really
>> >> have an obvious place to be fixed.
>> >
>> >Is there a reason to not continue to make the packages available in 
>> >non-free?
>> >I don't see a reason to force any change on existing systems.
>> 
>> Two things:
>> 
>>  1. I'm worried what bugs we might expose by having packages be in two
>> components at once.
>>  2. I really don't like the idea of leaving two different
>> configurations in the wild; it'll confuse people and is more
>> likely to cause issues in the future IMHO.
>> 
>> Plus, as Shengjing Zhu points out: we already expect people to manage
>> the sources.list anyway on upgrades.
>> 
>I think in the absence of a release upgrade script (which I very much
>doubt will happen, and be tested, and we can rely will be used, for
>bookworm), Michael's suggestion seems like a reasonable way forward.  I
>imagine we'll need to patch dak to allow that, but it seems like it
>should be tractable?

I'm also worried what effect this will have on other tools that have
to grok the archive (mirror tools, debian-cd, etc.). I'm not going to
try and veto having things in more than one component, but (ugh!) I
really think it's ugly. Actually, I think I'd much prefer Santiago's
idea:

> Couldn't we handle this via transitional firware* non-free packages,
> that depend on bookworm non-free-firmware packages?

We'd need to add some transitional binary packages for the small
number of n-f-f source packages. That way people would get errors from
apt if they don't read our warnings and update. Maybe this is a way
forward?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.

Re: Firmware GR result - what happens next?

[Julien Cristau]

On Sun, Oct 02, 2022 at 08:21:31PM +0100, Steve McIntyre wrote:
> On Sun, Oct 02, 2022 at 11:08:47AM -0400, Michael Stone wrote:
> >On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
> >> On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
> >> > What's the plan for upgraded systems with an existing 
> >> > /etc/apt/sources.list.
> >> > Will the new n-f-f section added on upgrades automatically(if non-free 
> >> > was
> >> > enabled before)?
> >> 
> >> So this is the one bit that I don't think we currently have a good
> >> answer for. We've never had a specific script to run on upgrades (like
> >> Ubuntu do), so this kind of potentially breaking change doesn't really
> >> have an obvious place to be fixed.
> >
> >Is there a reason to not continue to make the packages available in non-free?
> >I don't see a reason to force any change on existing systems.
> 
> Two things:
> 
>  1. I'm worried what bugs we might expose by having packages be in two
> components at once.
>  2. I really don't like the idea of leaving two different
> configurations in the wild; it'll confuse people and is more
> likely to cause issues in the future IMHO.
> 
> Plus, as Shengjing Zhu points out: we already expect people to manage
> the sources.list anyway on upgrades.
> 
I think in the absence of a release upgrade script (which I very much
doubt will happen, and be tested, and we can rely will be used, for
bookworm), Michael's suggestion seems like a reasonable way forward.  I
imagine we'll need to patch dak to allow that, but it seems like it
should be tractable?

Cheers,
Julien

Re: Firmware GR result - what happens next?

[Thomas Goirand]

On 10/3/22 02:23, Charles Plessy wrote:

Le Mon, Oct 03, 2022 at 12:33:20AM +0200, Mattia Rizzolo a écrit :


I can live with an APT hook warning me if I have non-free but not
non-free-firmware, but I would prefer to even do without that.


In addition, how about distributing the firmware in both 'non-free' and
'non-free-firmware' at the same time for a release or two ?


How about being smarter than this? :)

Cheers,

Thomas Goirand (zigo)

Re: Firmware GR result - what happens next?

[Michael Stone]

On Sun, Oct 02, 2022 at 08:21:31PM +0100, Steve McIntyre wrote:

Plus, as Shengjing Zhu points out: we already expect people to manage
the sources.list anyway on upgrades.


We also try to avoid silent install problems that might or might not 
result in a system that doesn't boot properly.

Re: Firmware GR result - what happens next?

[Julian Andres Klode]

On Sun, Oct 02, 2022 at 12:26:29PM -0700, Steve Langasek wrote:
> On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
> > >What's the plan for upgraded systems with an existing 
> > >/etc/apt/sources.list.
> > >Will the new n-f-f section added on upgrades automatically(if non-free was
> > >enabled before)?
> 
> > So this is the one bit that I don't think we currently have a good
> > answer for. We've never had a specific script to run on upgrades (like
> > Ubuntu do), so this kind of potentially breaking change doesn't really
> > have an obvious place to be fixed.
> 
> > Obviously we'll need to mention this in the release notes for
> > bookworm. Should we maybe talk about adding an upgrade helper tool?

We have already talked about apt release-upgrade for a couple of times,
but no time to implement it yet and the design needs to be talked
through how the target release can hook into apt to provide quirks.

We want apt release-upgrade too

1. rewrite sources for you if you want to
2. upgrade the system in three steps equivalent to

apt safe-upgrade --without-new-pkgs
apt safe-upgrade --with-new-pkg
apt full-upgrade

3. Hooks to customize the dependency solving (adding/removing packages),
   and potentially arbitrary quirks.

Where things get awkard is that (1) we'd like declarative hooks where
possible, a deb822 file of hooks, but (2) we also need to perhaps add
new logic.

So there was an idea to build a binary package that ships a module that
can be loaded at runtime by apt. So apt would install that package first
before it can upgrade. Or you can make it be a shell script and have
hooks of Type: Shell. Or just add Depends to release file that requires
users to install a newer version of apt before they can use the
repository and then ship that in (old)stable-updates with the additional
types of hooks.


> 
> I heartily endorse ubuntu-release-upgrader, it has been useful in addressing
> uncounted upgrade issues over the years and I think something like this
> would be a nice addition to Debian as well.  Two caveats:
> 
>  - Despite this being the sanctioned upgrade path in Ubuntu for over a
>decade, every single cycle we get bug reports from users who have run
>into issues because they have bypassed it and done the manual sed
>/etc/apt/sources.list && apt dist-upgrade.  So in Debian where this has
>been the norm for /two/ decades, I would not expect this to substantially
>reduce the error rate in the first release where such a mechanism is
>introduced.  (After all, whether telling users to use a new upgrader tool
>or telling them to manually add a component to sources.list, they will
>have to read the release notes to know about it!)
> 
>  - There are always some users that end up with buggy systems after upgrade
>despite using the supported interface because they upgrade to the devel
>release, and the release-upgrader is still under development up until
>release so they miss out on quirks being applied - and there is no
>interface for users to replay the quirks that they missed out on.  Don't
>repeat the same design mistake.

That makes sense. So the idea for apt release-upgrade is to have a list
of quirks and each quirk can then have a test to see if it ran already.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Lennart Sorensen]

On Mon, Oct 03, 2022 at 02:47:33PM +0200, Pascal Hambourg wrote:
> Not even replace "stable/updates" with "stable-security" during the upgrade
> from buster to bullseye ?

Hmm I don't recall but I suppose it just wasn't very memorable to do it.
At least it would have given an error fetching the list if you didn't
do it.  Not adding a new entry on the other hand will not.

-- 
Len Sorensen

Re: Firmware GR result - what happens next?

[Pascal Hambourg]

On 03/10/2022 at 01:00, Lennart Sorensen wrote:

On Sun, Oct 02, 2022 at 08:21:31PM +0100, Steve McIntyre wrote:


Plus, as Shengjing Zhu points out: we already expect people to manage
the sources.list anyway on upgrades.


People that just have 'stable' in their sources.list haven't had to
do anything.  I can't think of ever having had to add anything, only
change the release name.


Not even replace "stable/updates" with "stable-security" during the 
upgrade from buster to bullseye ?

Re: Firmware GR result - what happens next?

[Cyril Brulebois]

Steve McIntyre  (2022-10-02):
>   + ftpsync (?)

I don't think that's needed. Using buster's and more recently bullseye's
version, I have this locally:

drwxr-xr-x 4 mirror mirror 4096 Jul 19 04:16 
/srv/mirrors/debian/dists/bookworm/non-free-firmware/by-hash/

which matches when dak's config got updated with that extra component.

The git tree has a bin/archive_release that lists components explicitly
but it doesn't end up in any binary packages (there's a single ftpsync,
and it's not shipped there). Furthermore the 'non-free' string doesn't
appear in any of the files below debian/ftpsync/ after a build, so the
package doesn't look like it needs an update for that (even though a lot
of work happened in master since the last tag/release).


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Cyril Brulebois]

Colin Watson  (2022-10-03):
> Done in debmirror 1:2.37.  I guess we need to cherry-pick this to
> bullseye too?  I know bullseye doesn't have non-free-firmware (which
> is fine, the new debmirror doesn't object), but most people running
> mirrors probably run stable rather than testing.

Thanks for patching and verifying the extra component's presence on a
per-suite basis isn't an issue, that was on my to-do list.

And yes, it looks like bullseye-proposed-updates material to me; esp.
given that tiny diff…


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Colin Watson]

On Sun, Oct 02, 2022 at 03:27:36PM +0100, Steve McIntyre wrote:
> * Check/add support for the non-free-firmware section in various
>   places:
>   + debmirror (?)

Done in debmirror 1:2.37.  I guess we need to cherry-pick this to
bullseye too?  I know bullseye doesn't have non-free-firmware (which is
fine, the new debmirror doesn't object), but most people running mirrors
probably run stable rather than testing.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Firmware GR result - what happens next?

[Charles Plessy]

Le Mon, Oct 03, 2022 at 12:33:20AM +0200, Mattia Rizzolo a écrit :
> 
> I can live with an APT hook warning me if I have non-free but not
> non-free-firmware, but I would prefer to even do without that.

In addition, how about distributing the firmware in both 'non-free' and
'non-free-firmware' at the same time for a release or two ?

Cheers,

-- 
Charles Plessy Nagahama, Yomitan, Okinawa, Japan
Tooting from work,   https://mastodon.technology/@charles_plessy
Tooting from home, https://framapiaf.org/@charles_plessy

Re: Firmware GR result - what happens next?

[Lennart Sorensen]

On Sun, Oct 02, 2022 at 08:21:31PM +0100, Steve McIntyre wrote:
> Two things:
> 
>  1. I'm worried what bugs we might expose by having packages be in two
> components at once.
>  2. I really don't like the idea of leaving two different
> configurations in the wild; it'll confuse people and is more
> likely to cause issues in the future IMHO.
> 
> Plus, as Shengjing Zhu points out: we already expect people to manage
> the sources.list anyway on upgrades.

People that just have 'stable' in their sources.list haven't had to
do anything.  I can't think of ever having had to add anything, only
change the release name.

This will get missed and it will get missed a lot.

-- 
Len Sorensen

Re: Firmware GR result - what happens next?

[Mattia Rizzolo]

On Mon, Oct 03, 2022 at 01:18:55AM +0300, Dmitry Baryshkov wrote:
> вс, 2 окт. 2022 г. в 22:36, Steve Langasek :
> > > So this is the one bit that I don't think we currently have a good
> > > answer for. We've never had a specific script to run on upgrades (like
> > > Ubuntu do), so this kind of potentially breaking change doesn't really
> > > have an obvious place to be fixed.
> >
> > > Obviously we'll need to mention this in the release notes for
> > > bookworm. Should we maybe talk about adding an upgrade helper tool?
> >
> > I heartily endorse ubuntu-release-upgrader, it has been useful in addressing
> > uncounted upgrade issues over the years and I think something like this
> > would be a nice addition to Debian as well.  Two caveats:
> 
> I'd kindly ask against additional upgrade scripts. It is too easy to
> miss them, especially if one has been using Debian for ages with bare
> apt-get update && apt-get dist-upgrade.
> Moreover such a script will not help people that are already using testing.
> 
> For the past few decades, updating the setup was always a job of the
> package scripts.

This is getting OT in this thread, but I agree with the many that are
against such upgrading script.

I consider even the need for such thing a bug, as each package should
take care of its own quirks.

Besides, if something wants to mess with my system configuration, that's
an even greater bug for me (something that IME ubuntu has been doing
more and more over the last years).


I can live with an APT hook warning me if I have non-free but not
non-free-firmware, but I would prefer to even do without that.

For stable→stable updates there are the release notes for this tiny
change.
For testing/unstable users, they should really read d-d-a, and this
change be announced there (if it hasn't already).

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Dmitry Baryshkov]

Hello,

вс, 2 окт. 2022 г. в 22:36, Steve Langasek :
>
> On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
> > >What's the plan for upgraded systems with an existing 
> > >/etc/apt/sources.list.
> > >Will the new n-f-f section added on upgrades automatically(if non-free was
> > >enabled before)?
>
> > So this is the one bit that I don't think we currently have a good
> > answer for. We've never had a specific script to run on upgrades (like
> > Ubuntu do), so this kind of potentially breaking change doesn't really
> > have an obvious place to be fixed.
>
> > Obviously we'll need to mention this in the release notes for
> > bookworm. Should we maybe talk about adding an upgrade helper tool?
>
> I heartily endorse ubuntu-release-upgrader, it has been useful in addressing
> uncounted upgrade issues over the years and I think something like this
> would be a nice addition to Debian as well.  Two caveats:

I'd kindly ask against additional upgrade scripts. It is too easy to
miss them, especially if one has been using Debian for ages with bare
apt-get update && apt-get dist-upgrade.
Moreover such a script will not help people that are already using testing.

For the past few decades, updating the setup was always a job of the
package scripts. Thus we potentially can have an addon in the apt's
postinstall script that will check if the user is running bookworm,
the non-free repo is enabled and non-free-firmware is not. In such
case postinst can present user with a choice of ignoring n-f-f,
attempting automatic '/bookworm/s/non-free/non-free
non-free-firmware/' or letting user to fix setup. This way the user
will be present with such choice whichever path he uses to update to
bookworm.


-- 
With best wishes
Dmitry

Re: Firmware GR result - what happens next?

[Thomas Goirand]

Hi Steve,

On 10/2/22 21:26, Steve Langasek wrote:

I heartily endorse ubuntu-release-upgrader, it has been useful in addressing
uncounted upgrade issues over the years and I think something like this
would be a nice addition to Debian as well.  Two caveats:

  - Despite this being the sanctioned upgrade path in Ubuntu for over a
decade, every single cycle we get bug reports from users who have run
into issues because they have bypassed it and done the manual sed
/etc/apt/sources.list && apt dist-upgrade.  So in Debian where this has
been the norm for /two/ decades, I would not expect this to substantially
reduce the error rate in the first release where such a mechanism is
introduced.  (After all, whether telling users to use a new upgrader tool
or telling them to manually add a component to sources.list, they will
have to read the release notes to know about it!)

  - There are always some users that end up with buggy systems after upgrade
despite using the supported interface because they upgrade to the devel
release, and the release-upgrader is still under development up until
release so they miss out on quirks being applied - and there is no
interface for users to replay the quirks that they missed out on.  Don't
repeat the same design mistake.


I very much dislike the Ubuntu approach, but not only because of the 
above. Also because this approach forgets the fact that we also maintain 
2 rolling-updates distro (testing and unstable).



In the absence of a release-upgrader, the only way I see to automate this on
upgrade would be to handle it in the maintainer scripts of either base-files
(which I don't think the base-files maintainer would like) or apt.


If the base-files maintainer (ie: Santiago Vila) doesn't like doing 
things like this in "his" package, maybe we could have base-file >> 12.2 
depend on another package (called non-free-upgrade?) that would do the 
work instead. We could even have only base-file to depend on that 
package for a single release (ie: only for the lifetime of bookworm, and 
we get rid of the package after the release).


I think that's an even better approach than having this done in 
base-files itself.


Your thoughts?

Cheers,

Thomas Goirand (zigo)

Re: Firmware GR result - what happens next?

[Steve Langasek]

On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
> >What's the plan for upgraded systems with an existing /etc/apt/sources.list.
> >Will the new n-f-f section added on upgrades automatically(if non-free was
> >enabled before)?

> So this is the one bit that I don't think we currently have a good
> answer for. We've never had a specific script to run on upgrades (like
> Ubuntu do), so this kind of potentially breaking change doesn't really
> have an obvious place to be fixed.

> Obviously we'll need to mention this in the release notes for
> bookworm. Should we maybe talk about adding an upgrade helper tool?

I heartily endorse ubuntu-release-upgrader, it has been useful in addressing
uncounted upgrade issues over the years and I think something like this
would be a nice addition to Debian as well.  Two caveats:

 - Despite this being the sanctioned upgrade path in Ubuntu for over a
   decade, every single cycle we get bug reports from users who have run
   into issues because they have bypassed it and done the manual sed
   /etc/apt/sources.list && apt dist-upgrade.  So in Debian where this has
   been the norm for /two/ decades, I would not expect this to substantially
   reduce the error rate in the first release where such a mechanism is
   introduced.  (After all, whether telling users to use a new upgrader tool
   or telling them to manually add a component to sources.list, they will
   have to read the release notes to know about it!)

 - There are always some users that end up with buggy systems after upgrade
   despite using the supported interface because they upgrade to the devel
   release, and the release-upgrader is still under development up until
   release so they miss out on quirks being applied - and there is no
   interface for users to replay the quirks that they missed out on.  Don't
   repeat the same design mistake.

In the absence of a release-upgrader, the only way I see to automate this on
upgrade would be to handle it in the maintainer scripts of either base-files
(which I don't think the base-files maintainer would like) or apt.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Steve McIntyre]

On Sun, Oct 02, 2022 at 11:08:47AM -0400, Michael Stone wrote:
>On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:
>> On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
>> > What's the plan for upgraded systems with an existing 
>> > /etc/apt/sources.list.
>> > Will the new n-f-f section added on upgrades automatically(if non-free was
>> > enabled before)?
>> 
>> So this is the one bit that I don't think we currently have a good
>> answer for. We've never had a specific script to run on upgrades (like
>> Ubuntu do), so this kind of potentially breaking change doesn't really
>> have an obvious place to be fixed.
>
>Is there a reason to not continue to make the packages available in non-free?
>I don't see a reason to force any change on existing systems.

Two things:

 1. I'm worried what bugs we might expose by having packages be in two
components at once.
 2. I really don't like the idea of leaving two different
configurations in the wild; it'll confuse people and is more
likely to cause issues in the future IMHO.

Plus, as Shengjing Zhu points out: we already expect people to manage
the sources.list anyway on upgrades.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
You raise the blade, you make the change... You re-arrange me 'til I'm sane...

Re: Firmware GR result - what happens next?

[Steve McIntyre]

On Sun, Oct 02, 2022 at 05:31:16PM +0200, Cyril Brulebois wrote:
>Steve McIntyre  (2022-10-02):
>> * Extra d-i code to inform users about what firmware blobs have been
>>   loaded and the matching non-free-firmware packages. Plus information
>>   about the hardware involved. Maybe a new d-i module / udeb for this?
>>   Exact details here still TBD. Probably the biggest individual piece
>>   of work here.
>
>Not just blobs that were loaded, but also those which might get loaded
>in the installed system (since we don't have all modules around), see
>hw-detect.post-base-installer.d/50install-firmware in hw-detect (udevadm
>vs. modalias stuff).

ACK.

>> * Tweaks to add the non-free-firmware section in the apt-setup module
>>   if desired/needed.
>> 
>> * An extra boot option (a debconf variable) to disable loading extra
>>   firmware automatically, then exposed as an extra option through the
>>   isolinux and GRUB menus. d-i "expert mode" can also be used to tweak
>>   this behaviour later, except (obviously) for things like audio
>>   firmware that *must* be loaded early if they're going to be useful
>>   at all.
>
>The option/variable looks easy enough (once we decide what to call it)…
>Not sure what would be best to expose it to users: bootloader menus
>already have many options (text vs.  graphical, normal vs. rescue,
>accessibility settings, etc.), and don't get internationalization
>support anyway. On the flip side, having to go through a full expert
>installation is very nice.
>
>Maybe start by documenting the option (probably installation guide plus
>release notes), and of course implementing it; then see if/how we expose
>it?

Yup. Very much in that order... :-)

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
You lock the door
And throw away the key
There's someone in my head but it's not me 

Re: Firmware GR result - what happens next?

[John Scott]

> I think it's ironic
Apologies, on second thought this was poor wording. It's not ironic,
merely an oversight. We all believe in the success of free software, and
I don't mean to question anyone's values or allegiance for wanting to
serve users by tackling the most evident problems.


signature.asc
Description: This is a digitally signed message part

Re: Firmware GR result - what happens next?

[John Scott]

I concede I'm biased as its maintainer, but I think it's ironic that
non-free firmware is about to have better support than the flagship
libre wireless firmware. I'm referring to open-ath9k-htc-firmware, which
if you're not familiar, is the firmware for the most prominent USB
wireless adapters that work with exclusively free software, including
all of the recent Respects Your Freedom-certified ones.

I specifically have two grievances:
 * Unlike all other free firmware, firmware-ath9k-htc is not installed
on systems by default. The only reason it's segregated is because it 
gets built from source, unlike the other free firmware.
 * The ath9k_htc firmware should also be in the installer so users can
install over Wi-Fi. Fortunately the live images already include it, so I
usually recommend users wanting to install over Wi-Fi use that.

Note that downstream distros that are more free software-centric, like
Trisquel and PureOS, have already addressed the foregoing by choosing to
install firmware-ath9k-htc by default as a downstream change.

I'm not a new Debian Contributor; I know how things work around here.
But it would be nice if I could get some help from folks who already
know the ins and outs of the Debian Installer, and I thought this thread
would be a good opportunity to bring awareness to these issues affecting
firmware-ath9k-htc.

P.S. I'm aware that open-ath9k-htc-firmware may be FTBFS right now, I
plan to have a fix shortly.

P.P.S. I plan to split carl9170fw (the other 802.11n USB wireless
firmware) out into its own source package in the style of firmware-
ath9k-htc so we can build it from source too. Even though it's already
in firmware-linux-free, carl9170 currently doesn't work with the
installer either.

Thanks for your consideration.


signature.asc
Description: This is a digitally signed message part

Re: Firmware GR result - what happens next?

[Shengjing Zhu]

On Sun, Oct 2, 2022 at 10:53 PM Steve McIntyre  wrote:
>
> On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
> >
> >Hi Steve,
> >
> >thanks for the update!
> >
> >Am 02.10.22 um 16:27 schrieb Steve McIntyre:
> >
> >> * Tweaks to add the non-free-firmware section in the apt-setup module
> >>if desired/needed.
> >
> >...
> >
> >> If you think I've missed anything here, please let me and Cyril know -
> >> the best place would be the mailing list
> >> (debian-boot@lists.debian.org).
> >
> >What's the plan for upgraded systems with an existing /etc/apt/sources.list.
> >Will the new n-f-f section added on upgrades automatically(if non-free was
> >enabled before)?
>
> So this is the one bit that I don't think we currently have a good
> answer for. We've never had a specific script to run on upgrades (like
> Ubuntu do), so this kind of potentially breaking change doesn't really
> have an obvious place to be fixed.
>
> Obviously we'll need to mention this in the release notes for
> bookworm. Should we maybe talk about adding an upgrade helper tool?

For upgrading, people already need to edit their source list to change
the suite name, why would it hurt to add one more manual step to
change the section name?

-- 
Shengjing Zhu

Re: Firmware GR result - what happens next?

[Jeremy Bicha]

On Sun, Oct 2, 2022 at 10:53 AM Steve McIntyre  wrote:
> On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
> >What's the plan for upgraded systems with an existing /etc/apt/sources.list.
> >Will the new n-f-f section added on upgrades automatically(if non-free was
> >enabled before)?
>
> So this is the one bit that I don't think we currently have a good
> answer for. We've never had a specific script to run on upgrades (like
> Ubuntu do), so this kind of potentially breaking change doesn't really
> have an obvious place to be fixed.

There is /etc/apt/sources.list.d/ . One idea is that you could have a
package install a non-free-firmware apt source line there without
needing to touch anyone's primary /etc/apt/sources.list

I'm guessing things that change apt sources for mirrors would need to
be able to handle that file, but maybe they needed to handle
sources.list.d/ anyway.

Thank you,
Jeremy Bicha

Re: Firmware GR result - what happens next?

[Cyril Brulebois]

Steve McIntyre  (2022-10-02):
> * Extra d-i code to inform users about what firmware blobs have been
>   loaded and the matching non-free-firmware packages. Plus information
>   about the hardware involved. Maybe a new d-i module / udeb for this?
>   Exact details here still TBD. Probably the biggest individual piece
>   of work here.

Not just blobs that were loaded, but also those which might get loaded
in the installed system (since we don't have all modules around), see
hw-detect.post-base-installer.d/50install-firmware in hw-detect (udevadm
vs. modalias stuff).

> * Tweaks to add the non-free-firmware section in the apt-setup module
>   if desired/needed.
> 
> * An extra boot option (a debconf variable) to disable loading extra
>   firmware automatically, then exposed as an extra option through the
>   isolinux and GRUB menus. d-i "expert mode" can also be used to tweak
>   this behaviour later, except (obviously) for things like audio
>   firmware that *must* be loaded early if they're going to be useful
>   at all.

The option/variable looks easy enough (once we decide what to call it)…
Not sure what would be best to expose it to users: bootloader menus
already have many options (text vs.  graphical, normal vs. rescue,
accessibility settings, etc.), and don't get internationalization
support anyway. On the flip side, having to go through a full expert
installation is very nice.

Maybe start by documenting the option (probably installation guide plus
release notes), and of course implementing it; then see if/how we expose
it?


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Firmware GR result - what happens next?

[Michael Stone]

On Sun, Oct 02, 2022 at 03:53:00PM +0100, Steve McIntyre wrote:

On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:

What's the plan for upgraded systems with an existing /etc/apt/sources.list.
Will the new n-f-f section added on upgrades automatically(if non-free was
enabled before)?


So this is the one bit that I don't think we currently have a good
answer for. We've never had a specific script to run on upgrades (like
Ubuntu do), so this kind of potentially breaking change doesn't really
have an obvious place to be fixed.


Is there a reason to not continue to make the packages available in 
non-free? I don't see a reason to force any change on existing systems.

Re: Firmware GR result - what happens next?

[Steve McIntyre]

On Sun, Oct 02, 2022 at 04:43:47PM +0200, Michael Biebl wrote:
>
>Hi Steve,
>
>thanks for the update!
>
>Am 02.10.22 um 16:27 schrieb Steve McIntyre:
>
>> * Tweaks to add the non-free-firmware section in the apt-setup module
>>if desired/needed.
>
>...
>
>> If you think I've missed anything here, please let me and Cyril know -
>> the best place would be the mailing list
>> (debian-boot@lists.debian.org).
>
>What's the plan for upgraded systems with an existing /etc/apt/sources.list.
>Will the new n-f-f section added on upgrades automatically(if non-free was
>enabled before)?

So this is the one bit that I don't think we currently have a good
answer for. We've never had a specific script to run on upgrades (like
Ubuntu do), so this kind of potentially breaking change doesn't really
have an obvious place to be fixed.

Obviously we'll need to mention this in the release notes for
bookworm. Should we maybe talk about adding an upgrade helper tool?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Into the distance, a ribbon of black
Stretched to the point of no turning back

Re: Firmware GR result - what happens next?

[Michael Biebl]

Hi Steve,

thanks for the update!

Am 02.10.22 um 16:27 schrieb Steve McIntyre:


* Tweaks to add the non-free-firmware section in the apt-setup module
   if desired/needed.


...


If you think I've missed anything here, please let me and Cyril know -
the best place would be the mailing list
(debian-boot@lists.debian.org).


What's the plan for upgraded systems with an existing 
/etc/apt/sources.list. Will the new n-f-f section added on upgrades 
automatically(if non-free was enabled before)?


Regards,
Michael


OpenPGP_signature
Description: OpenPGP digital signature