Bug#954794: New packages must not declare themselves Essential

[Jonathan Nieder]

Hi,

Sean Whitton wrote:
> On Mon 16 Nov 2020 at 04:12AM +01, Guillem Jover wrote:
> > On Sat, 2020-11-07 at 13:30:13 -0700, Sean Whitton wrote:

>>> Could I ask you to explain your wanting to reduce the Essential set for
>>> the sake of small installation size in more detail, including some
>>> numbers, please?  It would be good to get to the bottom of Bill's worry
>>> about this change, but in addition, I would like to see a stronger
>>> positive case.
>>
>> I'm not sure about Josh, but I think the main reasons for wanting to
>> reduce the essential set are:
>>
>>   - Making chroots/containers slimmer, which can have a substantial
>> impact when needing lots of them, where even few MiB can make a
>> difference.
>>   - Making bootstrapping (build and installation) in general easier,
>> even though for the former these packages also need to then
>> be ideally removed from the build-essential set too.
>
> Thank you for this, but I was hoping for some more specifics.  For
> example, what are some examples of large Essential: yes packages that
> might actually, in practice, be removable?

See https://wiki.debian.org/Proposals/EssentialOnDiet: an example is
e2fsprogs, which is ~2.1 MiB.  (You might not consider that large, but
when you multiply that by "every Debian installation everywhere", it
adds up.)

https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=nonessentiale2fsprogs;users=helm...@debian.org
shows that people are already adding explicit dependencies on it,
which means that
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954794;msg=111 is
the de facto policy / what people believe policy to say.  (Which is a
surprise to me, but it's a useful signal.)

Thanks,
Jonathan

Bug#954794: New packages must not declare themselves Essential

[Sean Whitton]

Hello,

On Mon 16 Nov 2020 at 04:12AM +01, Guillem Jover wrote:

> On Sat, 2020-11-07 at 13:30:13 -0700, Sean Whitton wrote:
>> Could I ask you to explain your wanting to reduce the Essential set for
>> the sake of small installation size in more detail, including some
>> numbers, please?  It would be good to get to the bottom of Bill's worry
>> about this change, but in addition, I would like to see a stronger
>> positive case.
>
> I'm not sure about Josh, but I think the main reasons for wanting to
> reduce the essential set are:
>
>   - Making chroots/containers slimmer, which can have a substantial
> impact when needing lots of them, where even few MiB can make a
> difference.
>   - Making bootstrapping (build and installation) in general easier,
> even though for the former these packages also need to then
> be ideally removed from the build-essential set too.

Thank you for this, but I was hoping for some more specifics.  For
example, what are some examples of large Essential: yes packages that
might actually, in practice, be removable?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#954794: New packages must not declare themselves Essential

[Guillem Jover]

On Sat, 2020-11-07 at 13:30:13 -0700, Sean Whitton wrote:
> Could I ask you to explain your wanting to reduce the Essential set for
> the sake of small installation size in more detail, including some
> numbers, please?  It would be good to get to the bottom of Bill's worry
> about this change, but in addition, I would like to see a stronger
> positive case.

I'm not sure about Josh, but I think the main reasons for wanting to
reduce the essential set are:

  - Making chroots/containers slimmer, which can have a substantial
impact when needing lots of them, where even few MiB can make a
difference.
  - Making bootstrapping (build and installation) in general easier,
even though for the former these packages also need to then
be ideally removed from the build-essential set too.

Then there are these:

  - Making the packaging system less opaque, as the full implications
of Essential:yes do not seem generally clear to all maintainers?
  - Making it easier to use alternative implementations for some of
these packages, as then it's easier to know what depends on what,
instead of having to unconditionally provide the entire interface
surface for anything.
  - Making it easier to handle some multiarch problems, as then
dependencies can be explicitly marked as needed.

Thanks,
Guillem

Bug#954794: New packages must not declare themselves Essential

[Guillem Jover]

On Wed, 2020-09-30 at 18:34:06 -0700, Jonathan Nieder wrote:
> Josh Triplett wrote:
> > Jonathan Nieder wrote:
> > > Josh Triplett wrote:
> > > > This change does not propose eliminating the concept of Essential, nor
> > > > does it propose that any specific package become non-Essential.
> > >
> > > I think I'd be more supportive of this change if it did.  Freezing the
> > > current essential set in time feels oddly unpragmatic.  If we had a
> > > plan, even one that would take a while, to shrink the essential set,
> > > then it would be more likely to feel worth the cognitive load.

I'm in full support of trying to reduce the essential set, and this is
something we have already been trying to do for a while now:

  https://wiki.debian.org/Proposals/EssentialOnDiet
  https://wiki.debian.org/BusterPriorityRequalification

I also do not see the need to disallow adding new stuff to it if need
be. I think there's enough resistance in place that proposing that on
debian-devel would be met with quite some skepticism, *but* in case it
is needed, it would seem odd to do it anyway, even if policy disallows
it.

> > Long-term, I'd like to see that happen. But I'm a huge fan of
> > incremental steps; defining the problem as "eliminate Essential" makes
> > it both difficult enough and controversial enough to make it unlikely to
> > happen at all. Right now, the first step is "let's not let the problem
> > get any worse, and let's ensure that any new package that might have
> > otherwise used Essential must instead get packages to add a dependency".

> Even so, some *rough* consensus on the plan is very useful for helping
> people evaluate that first step.  It also gives people confidence that
> it can still move forward even if some people involved no longer have
> time to carry it forward themselves.  In this example, I think we
> agree about the ideal first (freeze the current Essential set) and
> last (eliminate Essential) steps but we don't have a clear picture of
> what happens in between.  That makes it hard to be comfortable with
> the first step because there's no reason to be confident that we'll
> get any further than that.

While the idea of completely getting rid of the Essential:yes concept
has been floated around in the bootstrapping and multiarch circles,
and seems like an interesting idea, I think it currently has too many
problems to be entertained.

Several of the Essential:yes properties [P] would need to be preserved
somehow for the packaging system to be able to operate as of now. Making
that marking implicit (some kind of lore) seems worse, than the current
explicit one, because the Essential field does not get involved much in
the package system operations anyway, it's more of a contract to be
honored by *maintainers*! Having to add Pre-Depends to tons of packages,
would probably substantially strain our dependency resolvers. Maintainer
scripts in at least the essential set are another problematic area,
so they would need to be reduced substantially, or eliminated
completely [B][D].

 [P] 
 [B] 
 [D] 

> If we don't have a rough plan, then the current state already seems
> okay: whenever someone proposes adding new Essential functionality, we
> can discuss it on debian-devel and the most likely outcome is that it
> gets rejected.

I think there are some rough plans for some of the problematic pieces,
but otherwise this seems rather premature, and I concur that the
current state seems fine.

> Some next steps that would make me more comfortable with an explicit
> freeze on Essential:
> 
> - document how to mark a package as "not safe in normal circumstances
>   to uninstall" (such as apt and systemd-sysv).  Is "Priority:
>   important" enough or should one use more?  As a package maintainer,
>   how do I mark a package as something that a user would never want to
>   deinstall in normal circumstances?

This is already supported with the Protected (old Important) fields,
and was implemented precisely to make it possible to split the
essential set into its different facets.

  

> - some rough idea about how to find undeclared dependencies.  Do we
>   do an NMU to add currently Essential packages as Pre-Depends on all
>   packages and then unmark them as Essential and let package
>   maintainers sort it out?  Is there some autodetection we can do
>   using autopkgtest?

Not all packages need Pre-Depends on currently Essential:yes packages,
but this would require careful consideration. But as mentioned above,
having to increase the amount of Pre-Depends relationships could make
our dependency resolvers and upgrades unhappy.

Trying to detect dependencies is going to be tricky, given that most
of the essential set are callable interfaces, which need to be checked

Bug#954794: New packages must not declare themselves Essential

[Guillem Jover]

On Sun, 2020-10-18 at 11:43:18 +0200, Bill Allombert wrote:
> On Thu, Oct 15, 2020 at 11:56:19AM -0700, Jonathan Nieder wrote:
> > More specifically, it's the right first three steps.
> > 
> > https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
> > currently says
> > 
> > Packages are not required to declare any dependencies they
> > have on other packages which are marked Essential (see below),
> > and should not do so unless they depend on a particular
> > version of that package.[4]
> > 
> > [4] [...] If packages add unnecessary dependencies on packages
> > in this set, the chances that there will be an unresolvable
> > dependency loop caused by forcing these Essential packages to
> > be configured first before they need to be is greatly
> > increased.
> > 
> > I'd propose that as a first step we change that to
> > 
> > Packages are not required to declare any dependencies they
> > have on other packages which are marked Essential (see below),
> > but are permitted to do so even if they do not depend on a
> > particular version of that package.[4]
> 
> This is very dangerous with respect to upgrade between stable releases.
> The issue is at the time a package is made for a stable release, the
> state of Debian and Essential: yes packages is not known. It is
> unrealistic to expect Debian to plan so far in advance.
> Requiring changes to Essential packages to take into account spurious
> dependencies is too fragile.

This could refer to concerns with two opposite situations, I guess:

  - When adding new Essential:yes marks, with omitted dependencies (but
that is already a problem with the current text). In that case
the dependencies on the new Essential:yes should only be dropped
after the next stable release, or packages might assume the
presence of functionality that might not be there.
  - When removing old Essential:yes marks, with adding "excess"
dependencies, which could possibly introduce dependency loops,
that could break ordering assumptions in maintainer script
execution. This can be problematic if the requirements for the
behavior of an Essential:yes package for the affected package are
dropped before the next stable release (work even when unpacked,
Pre-Depends usage, etc), but otherwise it should be fine I guess.

Thanks,
Guillem

Bug#954794: New packages must not declare themselves Essential

[Sean Whitton]

control: retitle -1 Permit packages to declare dependencies on Essential 
packages

Hello Josh,

On Sat 17 Oct 2020 at 04:49PM -07, Josh Triplett wrote:

> On Thu, Oct 15, 2020 at 11:56:19AM -0700, Jonathan Nieder wrote:
>>
>> More specifically, it's the right first three steps.
>>
>> https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
>> currently says
>>
>>  Packages are not required to declare any dependencies they
>>  have on other packages which are marked Essential (see below),
>>  and should not do so unless they depend on a particular
>>  version of that package.[4]
>>
>>  [4] [...] If packages add unnecessary dependencies on packages
>>  in this set, the chances that there will be an unresolvable
>>  dependency loop caused by forcing these Essential packages to
>>  be configured first before they need to be is greatly
>>  increased.
>>
>> I'd propose that as a first step we change that to
>>
>>  Packages are not required to declare any dependencies they
>>  have on other packages which are marked Essential (see below),
>>  but are permitted to do so even if they do not depend on a
>>  particular version of that package.[4]
>>
>>  [4] Functionality is rarely ever removed from the Essential
>>  set, but packages have been removed from the Essential set
>>  when the functionality moved to a different package. So when
>>  depending on these packages for such functionality, it is
>>  recommended to depend on an appropriate virtual package
>>  instead.
>>
>>  Future versions of Debian policy may encourage and then
>>  require including explicit dependencies even for essential
>>  functionality. This is helpful to users putting together
>>  ultra-minimal systems (though Debian does not support omitting
>>  Essential functionality out of the box), and it means less
>>  work is required in case the moment comes to remove some
>>  functionality from the Essential set in the future.
>>
>> (The next two steps would be to change that from "are permitted" to
>> "should", and then to "must".)
>>
>> What do you think?
>
> That sounds great to me. This would mean that there'd be no policy
> violation involved if someone wanted to send out patches working towards
> making a package non-Essential. It's a good incremental step.

Could I ask you to explain your wanting to reduce the Essential set for
the sake of small installation size in more detail, including some
numbers, please?  It would be good to get to the bottom of Bill's worry
about this change, but in addition, I would like to see a stronger
positive case.

As someone who is not concerned about small installation size, I rather
enjoy how the functionality of the Essential set isn't likely to be
reduced any time soon.  As an example, I understand that RedHat systems
aren't guaranteed to have Perl on them anymore; I appreciate how when
I'm working with Debian systems, I know I'm not going to have to spend
time improving my knowledge of awk, and anyway having to use a tool
which I think is worse.

I don't mean to suggest that this usecase of mine is decisive, but it
illustrates that the benefits of keeping Essential as it is are clear,
whereas the benefits of reducing Essential are, currently, vague.  Could
we actually decrease installation size in a way that actually benefits
actually existing usecases if we were to start trying to reduce
Essential?  I would like to see evidence of that.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#954794: New packages must not declare themselves Essential

[Sam Hartman]

> "Bill" == Bill Allombert  writes:

>> I'd propose that as a first step we change that to
>> 
>> Packages are not required to declare any dependencies they have
>> on other packages which are marked Essential (see below), but are
>> permitted to do so even if they do not depend on a particular
>> version of that package.[4]

Bill> This is very dangerous with respect to upgrade between stable
Bill> releases.  The issue is at the time a package is made for a
Bill> stable release, the state of Debian and Essential: yes
Bill> packages is not known. It is unrealistic to expect Debian to
Bill> plan so far in advance.  Requiring changes to Essential
Bill> packages to take into account spurious dependencies is too
Bill> fragile.

I'm sorry, but I consider myself reasonably knowledgable about package
dependencies and Debian--not admittedly as knowledgable as you, but
knowledgable enough that I ought to be able to follow this discussion.
And after spending a couple minutes thinking about the above I don't
understand what you are getting at.

So, please explain in enough detail that we are able to understand and
evaluate your concern.

Bug#954794: New packages must not declare themselves Essential

[Bill Allombert]

On Thu, Oct 15, 2020 at 11:56:19AM -0700, Jonathan Nieder wrote:
> Javier Serrano Polo wrote:
> > On Wed, 30 Sep 2020 18:34:06 -0700 Jonathan Nieder 
> > wrote:
> 
> >> Even so, some *rough* consensus on the plan is very useful for
> >> helping people evaluate that first step.
> >
> > Here is a rough plan:
> >
> >1. Policy: Packages should declare all their dependencies, even
> >   essential ones.
> 
> I agree: this is the right first step.
> 
> More specifically, it's the right first three steps.
> 
> https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
> currently says
> 
>   Packages are not required to declare any dependencies they
>   have on other packages which are marked Essential (see below),
>   and should not do so unless they depend on a particular
>   version of that package.[4]
> 
>   [4] [...] If packages add unnecessary dependencies on packages
>   in this set, the chances that there will be an unresolvable
>   dependency loop caused by forcing these Essential packages to
>   be configured first before they need to be is greatly
>   increased.
> 
> I'd propose that as a first step we change that to
> 
>   Packages are not required to declare any dependencies they
>   have on other packages which are marked Essential (see below),
>   but are permitted to do so even if they do not depend on a
>   particular version of that package.[4]

This is very dangerous with respect to upgrade between stable releases.
The issue is at the time a package is made for a stable release, the
state of Debian and Essential: yes packages is not known. It is
unrealistic to expect Debian to plan so far in advance.
Requiring changes to Essential packages to take into account spurious
dependencies is too fragile.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#954794: New packages must not declare themselves Essential

[Josh Triplett]

On Thu, Oct 15, 2020 at 11:56:19AM -0700, Jonathan Nieder wrote:
> Javier Serrano Polo wrote:
> > On Wed, 30 Sep 2020 18:34:06 -0700 Jonathan Nieder 
> > wrote:
> 
> >> Even so, some *rough* consensus on the plan is very useful for
> >> helping people evaluate that first step.
> >
> > Here is a rough plan:
> >
> >1. Policy: Packages should declare all their dependencies, even
> >   essential ones.
> 
> I agree: this is the right first step.
> 
> More specifically, it's the right first three steps.
> 
> https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
> currently says
> 
>   Packages are not required to declare any dependencies they
>   have on other packages which are marked Essential (see below),
>   and should not do so unless they depend on a particular
>   version of that package.[4]
> 
>   [4] [...] If packages add unnecessary dependencies on packages
>   in this set, the chances that there will be an unresolvable
>   dependency loop caused by forcing these Essential packages to
>   be configured first before they need to be is greatly
>   increased.
> 
> I'd propose that as a first step we change that to
> 
>   Packages are not required to declare any dependencies they
>   have on other packages which are marked Essential (see below),
>   but are permitted to do so even if they do not depend on a
>   particular version of that package.[4]
> 
>   [4] Functionality is rarely ever removed from the Essential
>   set, but packages have been removed from the Essential set
>   when the functionality moved to a different package. So when
>   depending on these packages for such functionality, it is
>   recommended to depend on an appropriate virtual package
>   instead.
> 
>   Future versions of Debian policy may encourage and then
>   require including explicit dependencies even for essential
>   functionality. This is helpful to users putting together
>   ultra-minimal systems (though Debian does not support omitting
>   Essential functionality out of the box), and it means less
>   work is required in case the moment comes to remove some
>   functionality from the Essential set in the future.
> 
> (The next two steps would be to change that from "are permitted" to
> "should", and then to "must".)
> 
> What do you think?

That sounds great to me. This would mean that there'd be no policy
violation involved if someone wanted to send out patches working towards
making a package non-Essential. It's a good incremental step.

Bug#954794: New packages must not declare themselves Essential

[Jonathan Nieder]

Javier Serrano Polo wrote:
> On Wed, 30 Sep 2020 18:34:06 -0700 Jonathan Nieder 
> wrote:

>> Even so, some *rough* consensus on the plan is very useful for
>> helping people evaluate that first step.
>
> Here is a rough plan:
>
>1. Policy: Packages should declare all their dependencies, even
>   essential ones.

I agree: this is the right first step.

More specifically, it's the right first three steps.

https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
currently says

Packages are not required to declare any dependencies they
have on other packages which are marked Essential (see below),
and should not do so unless they depend on a particular
version of that package.[4]

[4] [...] If packages add unnecessary dependencies on packages
in this set, the chances that there will be an unresolvable
dependency loop caused by forcing these Essential packages to
be configured first before they need to be is greatly
increased.

I'd propose that as a first step we change that to

Packages are not required to declare any dependencies they
have on other packages which are marked Essential (see below),
but are permitted to do so even if they do not depend on a
particular version of that package.[4]

[4] Functionality is rarely ever removed from the Essential
set, but packages have been removed from the Essential set
when the functionality moved to a different package. So when
depending on these packages for such functionality, it is
recommended to depend on an appropriate virtual package
instead.

Future versions of Debian policy may encourage and then
require including explicit dependencies even for essential
functionality. This is helpful to users putting together
ultra-minimal systems (though Debian does not support omitting
Essential functionality out of the box), and it means less
work is required in case the moment comes to remove some
functionality from the Essential set in the future.

(The next two steps would be to change that from "are permitted" to
"should", and then to "must".)

What do you think?

Thanks,
Jonathan

[4] "Essential is needed in part to avoid unresolvable dependency loops
on upgrade. If packages add unnecessary dependencies on packages in
this set, the chances that there will be an unresolvable dependency
loop caused by forcing these Essential packages to be configured first
before they need to be is greatly increased. It also increases the
chances that frontends will be unable to calculate an upgrade path,
even if one exists.

"Also, functionality is rarely ever removed from the Essential set, but
packages have been removed from the Essential set when the
functionality moved to a different package. So depending on these
packages just in case they stop being essential does way more harm
than good."

Bug#954794: New packages must not declare themselves Essential

[Javier Serrano Polo]

On Wed, 07 Oct 2020 18:43:22 -0400 Sam Hartman 
wrote:
> C) I'd support non-normative documentation that we don't expect to
> approve new essential packages in the future in policy.

Worthless documentation, I think.

> A) I do support reducing the essential set over time

Fine, then you should choose one essential package and remove it from
the set for bullseye or bookworm.

smime.p7s
Description: S/MIME cryptographic signature

Bug#954794: New packages must not declare themselves Essential

[Sean Whitton]

Hello,

On Wed 07 Oct 2020 at 06:43pm -04, Sam Hartman wrote:

> Josh, my current reading is that there is not support for even the
> first step.  I believe Guillem and I have disagreed, and I haven't
> noticed support from anyone other than you.

Speaking as Policy Editor, I agree.  I don't see any sort of consensus
that we should deprive ourselves of the ability to declare packages
Essential.

> C) I'd support non-normative documentation that we don't expect to
> approve new essential packages in the future in policy.

This sounds like a good idea to me too.

-- 
Sean Whitton

Bug#954794: New packages must not declare themselves Essential

[Sam Hartman]

> "Josh" == Josh Triplett  writes:

Josh> Long-term, I'd like to see that happen. But I'm a huge fan of
Josh> incremental steps; defining the problem as "eliminate
Josh> Essential" makes it both difficult enough and controversial
Josh> enough to make it unlikely to happen at all. Right now, the
Josh> first step is "let's not let the problem get any worse, and
Josh> let's ensure that any new package that might have otherwise
Josh> used Essential must instead get packages to add a dependency".

Josh, my current reading is that there is not support for even the first
step.
I believe Guillem and I have disagreed, and I haven't noticed support
from anyone other than you.

Is there support I'm failing to remember?

I would not attempt to summarize Guillem's concerns.
My concerns are roughly that I think

1) debian-devel consensus is an
adequate block for things getting worse unless there is a good reason

2) I am not convinced that we would (or should) decline to use this
particular hammer if it really is the best tool we have available for a
bind we find ourselves in; nor do I think policy would actually bar us
if we had necessity

3) The benefit I perceive in spending more time trying to figure out
whether I could be convinced that there are no circumstances under which
I'd support a new essential package is less than  what I think we'd get
out of it , so I'd rather not spend the time.

In the interest of being constructive:

A) I do support reducing the essential set over time

B) I would support better education of the community about why we should
be hesitant to support essential: yes on debian-devel

C) I'd support non-normative documentation that we don't expect to
approve new essential packages in the future in policy.


--Sam

Bug#954794: New packages must not declare themselves Essential

[Javier Serrano Polo]

On Wed, 30 Sep 2020 18:34:06 -0700 Jonathan Nieder 
wrote:
> Even so, some *rough* consensus on the plan is very useful for
> helping people evaluate that first step.

Here is a rough plan:

   1. Policy: Packages should declare all their dependencies, even
  essential ones.
   2. Make easier this task: document dependencies, add Lintian checks,
  etc.
   3. Policy: Packages must declare all their dependencies.
   4. Wait until previous Debian releases become unsupported.
   5. Drop support for Essential.

smime.p7s
Description: S/MIME cryptographic signature

Bug#954794: New packages must not declare themselves Essential

[Jonathan Nieder]

Josh Triplett wrote:
> Jonathan Nieder wrote:
>> Josh Triplett wrote:

>>> This change does not propose eliminating the concept of Essential, nor
>>> does it propose that any specific package become non-Essential.
>>
>> I think I'd be more supportive of this change if it did.  Freezing the
>> current essential set in time feels oddly unpragmatic.  If we had a
>> plan, even one that would take a while, to shrink the essential set,
>> then it would be more likely to feel worth the cognitive load.
>
> Long-term, I'd like to see that happen. But I'm a huge fan of
> incremental steps; defining the problem as "eliminate Essential" makes
> it both difficult enough and controversial enough to make it unlikely to
> happen at all. Right now, the first step is "let's not let the problem
> get any worse, and let's ensure that any new package that might have
> otherwise used Essential must instead get packages to add a dependency".

In consensus building and implementation, I agree with you: I like to
see work in incremental steps, and it's not necessary to get full
consensus for the full plan in advance.  After all, we can learn as we
go and would need to change the plan.

Even so, some *rough* consensus on the plan is very useful for helping
people evaluate that first step.  It also gives people confidence that
it can still move forward even if some people involved no longer have
time to carry it forward themselves.  In this example, I think we
agree about the ideal first (freeze the current Essential set) and
last (eliminate Essential) steps but we don't have a clear picture of
what happens in between.  That makes it hard to be comfortable with
the first step because there's no reason to be confident that we'll
get any further than that.

If we don't have a rough plan, then the current state already seems
okay: whenever someone proposes adding new Essential functionality, we
can discuss it on debian-devel and the most likely outcome is that it
gets rejected.

Some next steps that would make me more comfortable with an explicit
freeze on Essential:

- document how to mark a package as "not safe in normal circumstances
  to uninstall" (such as apt and systemd-sysv).  Is "Priority:
  important" enough or should one use more?  As a package maintainer,
  how do I mark a package as something that a user would never want to
  deinstall in normal circumstances?

- some rough idea about how to find undeclared dependencies.  Do we
  do an NMU to add currently Essential packages as Pre-Depends on all
  packages and then unmark them as Essential and let package
  maintainers sort it out?  Is there some autodetection we can do
  using autopkgtest?

- some rough idea about what to do with packages from outside the
  Debian archive.  Do we provide a way to declare implicit Pre-Depends
  for everything from a particular archive?

To be clear, I'm not saying we need to commit to a plan (and I'm
certainly not saying it should be documented in policy), but a rough
sense of whether this seems possible and whether there are people
interested in moving it forward would help build consensus for the
first step.

Thanks,
Jonathan

Bug#954794: New packages must not declare themselves Essential

[Josh Triplett]

On Tue, Sep 29, 2020 at 05:23:38PM -0700, jrnie...@gmail.com wrote:
> Hi,
> 
> Josh Triplett wrote:
> 
> > Over the years, "Essential" has made it difficult to reduce installation
> > size, to reduce chroot/container size, or to coordinate various
> > transitions. Removing something from the Essential set requires tracking
> > down every package using it, adding a dependency, carefully managing a
> > transition across Debian releases, and risking breakage of third-party
> > packages outside Debian.
> 
> Interesting.  On the other side if we were to eliminate Essential
> would be bloat in the Packages file from e.g. ~every package needing
> Pre-Depends on a shell.

That seems like it'd be trivially handled by compression. I just did a
quick test, and adding "bash, coreutils, " to every single package
record would only add 10k to the entire gzip-compressed file (which is
currently 10816k, so, <0.1%). There are currently ~60k *packages* in the
packages file, so this adds, on average, about a sixth of a byte per
package.

> [...]
> > This change does not propose eliminating the concept of Essential, nor
> > does it propose that any specific package become non-Essential.
> 
> I think I'd be more supportive of this change if it did.  Freezing the
> current essential set in time feels oddly unpragmatic.  If we had a
> plan, even one that would take a while, to shrink the essential set,
> then it would be more likely to feel worth the cognitive load.

Long-term, I'd like to see that happen. But I'm a huge fan of
incremental steps; defining the problem as "eliminate Essential" makes
it both difficult enough and controversial enough to make it unlikely to
happen at all. Right now, the first step is "let's not let the problem
get any worse, and let's ensure that any new package that might have
otherwise used Essential must instead get packages to add a dependency".

Bug#954794: New packages must not declare themselves Essential

[jrnieder]

Hi,

Josh Triplett wrote:

> Over the years, "Essential" has made it difficult to reduce installation
> size, to reduce chroot/container size, or to coordinate various
> transitions. Removing something from the Essential set requires tracking
> down every package using it, adding a dependency, carefully managing a
> transition across Debian releases, and risking breakage of third-party
> packages outside Debian.

Interesting.  On the other side if we were to eliminate Essential
would be bloat in the Packages file from e.g. ~every package needing
Pre-Depends on a shell.

[...]
> This change does not propose eliminating the concept of Essential, nor
> does it propose that any specific package become non-Essential.

I think I'd be more supportive of this change if it did.  Freezing the
current essential set in time feels oddly unpragmatic.  If we had a
plan, even one that would take a while, to shrink the essential set,
then it would be more likely to feel worth the cognitive load.

I think there is still a problem to solve here --- e.g. maybe there is
some definition of essential that we may want to move to that would
include things like base-files but wouldn't include things like dpkg
(to take an extreme example) but I don't think we've found it yet.

So even though I'm a fan of the intent here, I agree with the
consensus that we should close this until we have a more specific
proposal.

Thanks,
Jonathan

Bug#954794: New packages must not declare themselves Essential

[Javier Serrano Polo]

El dt 29 de 09 de 2020 a les 15:08 -0700, Josh Triplett va escriure:
> I want to avoid letting the problem get any worse.

So Essential packages are a problem. Do you want to remove Essential in
the long-term? If this goal is not clear, there is little point in
changing policy. New Essential packages will exist if there is a good-
enough reason.

smime.p7s
Description: S/MIME cryptographic signature

Bug#954794: New packages must not declare themselves Essential

[Josh Triplett]

On Mon, Sep 21, 2020 at 05:15:45PM +0200, Javier Serrano Polo wrote:
> On Mon, 23 Mar 2020 08:00:04 -0700 Josh Triplett  > wrote:
> > This change does not propose eliminating the concept of Essential,
> 
> What is the point of Essential? To omit declaring dependencies on the
> false assumption that some packages are always required by all systems;
> the concept is essentially ill. Thus, if you are not pursuing the
> elimination of Essential, your effort is virtually useless. Do you want
> to remove Essential?

I want to avoid letting the problem get any worse. Eliminating the
concept entirely would have substantial backwards-compatibility issues,
and would invite substantially more controversy. Right now, I'd just
like to propose not adding any *new* Essential packages.

Bug#954794: New packages must not declare themselves Essential

[Javier Serrano Polo]

El dl 21 de 09 de 2020 a les 17:15 +0200, Javier Serrano Polo va
escriure:
> Do you want to remove Essential?

Since it looks like you do not try to eliminate Essential, I will close
this report.

smime.p7s
Description: S/MIME cryptographic signature

Bug#954794: New packages must not declare themselves Essential

[Javier Serrano Polo]

On Mon, 23 Mar 2020 08:00:04 -0700 Josh Triplett  wrote:
> This change does not propose eliminating the concept of Essential,

What is the point of Essential? To omit declaring dependencies on the
false assumption that some packages are always required by all systems;
the concept is essentially ill. Thus, if you are not pursuing the
elimination of Essential, your effort is virtually useless. Do you want
to remove Essential?

smime.p7s
Description: S/MIME cryptographic signature

Bug#954794: New packages must not declare themselves Essential

[Sam Hartman]

> "Bill" == Bill Allombert  writes:

Bill> But is it an actual problem ?  Do we see packages marked
Bill> Essential: yes by mistake ?

I think Josh's analysis brought up some important points for me that I
did not consider before and that need to be considered making decisions
in the future.
I think capturing that analysis in a pointer from policy or in policy is
important to me.

Bill> Each time we make policy longer we dilute the content.


Obviously, in the limit this is true.
I think that capturing rationale for things that have good rationale and
that could affect the project if not properly considered is worth doing
even though it makes policy longer.

Bug#954794: New packages must not declare themselves Essential

[Bill Allombert]

On Wed, Apr 01, 2020 at 05:14:13AM -0400, Sam Hartman wrote:
> I concur with the comments raised so far.
> 
> I think it would be great to do a better job of outlining the problems
> with essential packages in debian-policy.
...
> I would support a statement in policy that as of the time of writing we
> do not anticipate ever creating a new essential package.  That would
> help people considering proposing making an essential package know they
> are probably looking at things the wrong way.

But is it an actual problem ?
Do we see packages marked Essential: yes by mistake ?

Each time we make policy longer we dilute the content.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#954794: New packages must not declare themselves Essential

[Sam Hartman]

I concur with the comments raised so far.

I think it would be great to do a better job of outlining the problems
with essential packages in debian-policy.
I don't understand why we would tie our hands though.
A consensus of debian-devel seems adequate to consider those issues and
evaluate them.
If after making that consideration we decide to create a new essential
package, we're going to do that policy not withstanding.

I would support a statement in policy that as of the time of writing we
do not anticipate ever creating a new essential package.  That would
help people considering proposing making an essential package know they
are probably looking at things the wrong way.

--Sam

Bug#954794: New packages must not declare themselves Essential

[Sean Whitton]

Hello,

On Mon 23 Mar 2020 at 04:29PM +01, Bill Allombert wrote:

> I do not think this proposal make sense _as a Debian policy change_.
> What I mean is that if the release team decide that some new packages
> need to be marked Essential: yes for some technical reason, then either
> policy will be ignored and this change will be reverted. Or maybe the
> bug will be serious but not RC.
>
> Remember as Manoj used to say, policy is no a stick to beat people with.

I'm inclined to agree with Bill here.

In particular, it is not clear to me we have a project consensus that
the requirement to seek consensus on debian-devel is not sufficient.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#954794: New packages must not declare themselves Essential

[Bill Allombert]

On Mon, Mar 23, 2020 at 08:00:04AM -0700, Josh Triplett wrote:
> Package: debian-policy
> Version: 4.5.0.0
> Severity: normal
> Tags: patch
> 
> Previously discussed on the mailing list, which led to a request for
> concrete Policy language.

I do not think this proposal make sense _as a Debian policy change_.
What I mean is that if the release team decide that some new packages
need to be marked Essential: yes for some technical reason, then either
policy will be ignored and this change will be reverted. Or maybe the
bug will be serious but not RC.

Remember as Manoj used to say, policy is no a stick to beat people with.

Cheers,
Bill.

Bug#954794: New packages must not declare themselves Essential

[Josh Triplett]

Package: debian-policy
Version: 4.5.0.0
Severity: normal
Tags: patch

Previously discussed on the mailing list, which led to a request for
concrete Policy language.

Over the years, "Essential" has made it difficult to reduce installation
size, to reduce chroot/container size, or to coordinate various
transitions. Removing something from the Essential set requires tracking
down every package using it, adding a dependency, carefully managing a
transition across Debian releases, and risking breakage of third-party
packages outside Debian.

Add language to Policy that still allows existing Essential packages to
refactor or transition to different packages (coordinated via
debian-devel) but does not allow new Essential packages.

Also add language that allows packages to declare dependencies on
Essential packages as part of transitions (such dependencies would
previously violate Policy, technically), and soften language that
connects "base system" with "essential".

This change does not propose eliminating the concept of Essential, nor
does it propose that any specific package become non-Essential.

Patch attached; also available at
https://salsa.debian.org/josh-guest/policy/ on the branch
no-new-essential.

- Josh Triplett
>From a4b263709ed183e8b353c669e0e56e225ef8c818 Mon Sep 17 00:00:00 2001
Message-Id: 

From: Josh Triplett 
Date: Mon, 23 Mar 2020 07:11:27 -0700
Subject: [PATCH] New packages must not declare themselves Essential

Over the years, "Essential" has made it difficult to reduce installation
size, to reduce chroot/container size, or to coordinate various
transitions. Removing something from the Essential set requires tracking
down every package using it, adding a dependency, carefully managing a
transition across Debian releases, and risking breakage of third-party
packages outside Debian.

Add language to Policy that still allows existing Essential packages to
refactor or transition to different packages (coordinated via
debian-devel) but does not allow new Essential packages.

Also add language that allows packages to declare dependencies on
Essential packages as part of transitions (such dependencies would
previously violate Policy, technically), and soften language that
connects "base system" with "essential".

This change does not propose eliminating the concept of Essential, nor
does it propose that any specific package become non-Essential.
---
 policy/ch-binary.rst | 17 -
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
index 74a1690..ac0623f 100644
--- a/policy/ch-binary.rst
+++ b/policy/ch-binary.rst
@@ -247,7 +247,9 @@ package.
 
 Packages are not required to declare any dependencies they have on other
 packages which are marked ``Essential`` (see below), and should not do
-so unless they depend on a particular version of that package.  [#]_
+so unless they depend on a particular version of that package, or unless
+part of a transition to make the ``Essential`` package no longer
+``Essential``.  [#]_
 
 Sometimes, unpacking one package requires that another package be first
 unpacked *and* configured. In this case, the depending package must
@@ -299,7 +301,7 @@ are allowed to form part of the base system, in order to 
keep the
 required disk usage very small.
 
 The base system consists of all those packages with priority
-``required`` or ``important``. Many of them will be tagged ``essential``
+``required`` or ``important``. Some of them are tagged ``essential``
 (see below).
 
 .. _s3.8:
@@ -335,9 +337,14 @@ never done. Any capability added to an ``essential`` 
package therefore
 creates an obligation to support that capability as part of the
 Essential set in perpetuity.
 
-You must not tag any packages ``essential`` before this has been
-discussed on the ``debian-devel`` mailing list and a consensus about
-doing that has been reached.
+New packages must not declare themselves ``essential``, even if this
+requires many other packages to declare explicit dependencies on them.
+Existing ``essential`` packages may continue to use the ``essential``
+flag in new versions. If an existing ``essential`` package needs to
+change its packaging structure in a way that would introduce a new
+package with the ``essential`` flag, this must be discussed on the
+``debian-devel`` mailing list and a consensus and transition strategy
+for doing so must be reached."
 
 .. _s-maintscripts:
 
-- 
2.26.0.rc2