Re: Please, minimize your build chroots

[Santiago Vila]

El 30/1/23 a las 14:05, Guillem Jover escribió:

Given the number of packages that currently declare a dependency on
tzdata (34~), the ones that seem to have the most potential to pull it
for others are language bindings such as python3-dateutil, python3-tz
ruby-tzinfo, etc, which handle timezone stuff anyway and would keep
pulling it. So I find this assertion hard to believe. And the following
points seem to be based on this fundamental assumption.


More to the point and fun fact: The dependency of ruby-tzinfo on tzdata was 
*missing*
in unstable:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026178

It was added after I reported that 21 different ruby packages failed to build
in a completely clean build environment. (Special thanks to Antonio Terceiro).

Thanks.

Re: Please, minimize your build chroots

[Guillem Jover]

On Sun, 2023-01-29 at 12:28:37 +0200, Adrian Bunk wrote:
> On Sun, Jan 29, 2023 at 05:00:56AM +0100, Guillem Jover wrote:
> > On Sat, 2023-01-28 at 21:35:01 +0200, Adrian Bunk wrote:
> > > Removing tzdata from the essential set made sense, and I do see your 
> > > point that tzdata does not fit the "totally broken" definition of
> > > "required".
> > 
> > > What we need are technical discussions like whether packages like tzdata 
> > > should also be removed from the build essential set, or whether tzdata 
> > > being a part of the build essential set should be expressed by making 
> > > the build-essential package depend on tzdata.
> > 
> > I guess my question is, what makes tzdata build essential, besides
> > that it's currently kind-of implicitly there? To me it does not look
> > like it has the properties to be considered as such, besides that if
> > we lower its priority (as it deserves) it makes a bunch of packages
> > FTBFS.
> 
> It has historically been part of the build essential set.

(So the "that it's currently kind-of implicitly there".)

> It is used in the build of many packages.

Given the number of packages that currently declare a dependency on
tzdata (34~), the ones that seem to have the most potential to pull it
for others are language bindings such as python3-dateutil, python3-tz
ruby-tzinfo, etc, which handle timezone stuff anyway and would keep
pulling it. So I find this assertion hard to believe. And the following
points seem to be based on this fundamental assumption.

I think though, we might be able to easily answer at least how many
builds ended up with tzdata installed (even if it ended up not being
used), if Santiago still has the build logs around, those would be
easily grep-able.

(Whether it was used at all could be checked after each build through
atime I guess. But this seems all like a lot of wasted effort TBH.)

And for reference, Santiago seems (I might be wrong) to have filed
around 46 reports about this, which seems like a very tiny speck in
the grand scheme of things.

> There would be so many invisible undeclared build dependencies of 
> packages using it who have it pulled in by other packages that random
> changes in the dependency tree might cause many packages to FTBFS at
> any time if it does not stay build essential.

I fail to see how this is any different from any other transitive
dependency being pulled automatically due to some implementation
detail somewhere.

> It is required to provide glibc functionality that is frequently
> used during the build.

This seems like a reiteration of the above "it's used a lot". There
are other things that are used a lot during builds and yet they are
not build-essential (python3 for an obvious example).

> > So, for one, this is getting in the way of making our minimal
> > (non build) systems smaller.
> 
> No, it is not.
> 
> There are 3 different topics:
> 
> 1. making minimal (non build) systems smaller
> 
> Being able to exclude tzdata from a minimal system was achieved when it 
> was removed from the essential set in stretch.
> debootstrap not installing it by default would make that easier.
> Whether build-essential depends on tzdata does not make any difference.

Sure, you can, but it's not the _default_, so it *is* getting in the way.
And some seem to be even trying to tie it into the build-essential set by
proxy, wanting to bolt Priority:required into it… People keep bringing up
in this discussion the sanctity of these default installations, so it
again seems unfair to then disregard it when it's not convenient.

> 2. normal systems
> 
> In a normal (non-minimal) installation not having tzdata installed 
> would be a bug harming many users, no matter what priority it will
> have.

Sure, this can easily be achieved by making it Priority:important,
so I don't see this being a concern.

> 3. build essential
> 
> That's separate from 1. and 2.

Sure again (as long as it is not kept as Priority:required "for
reasons").

> > > I have so far not seen any technical arguments why removing tzdata from 
> > > the build essential set would be better for Debian than keeping it there.
> > > Removing tzdata reduces the size of a chroot that has the build 
> > > dependencies for the hello package installed by ~ 0.5%, this size
> > > decrease does not strike me as a sufficient reason for reducing the
> > > build essential set.
> > 
> > I don't see how this can be a pure technical decision, TBH. To me this
> > looks more like either a matter of trade-offs,
> 
> It is a tradeoff between less work and saving ~ 0.5% space in build
> chroots.

That would be the trade-off you see, yes. :)

> > or IMO more importantly
> > of clean definition of a specification (which seems rather more
> > important than the other concerns). The work to spot these problems has
> > already been done, and the changes required are trivial and minimal
> > (disregarding any severity consideration here).
> 
> The work has been done for packages that do FTBFS 

Re: Please, minimize your build chroots

[Santiago Vila]

El 30/1/23 a las 13:41, Santiago Vila escribió:

Note: I've downgraded the bugs in dispute to important,
so they are not RC anymore, per request of Sebastian Ramacher.


I mean: Sebastian started to downgrade them, and I've downgraded
the remaining open bugs which were not downgraded by him,
to complete the operation.

Thanks.

Re: Please, minimize your build chroots

[Santiago Vila]

El 27/1/23 a las 22:37, Adrian Bunk escribió:

Speaking as someone who is doing a lot of QA work, [...]


Note: I've downgraded the bugs in dispute to important,
so they are not RC anymore, per request of Sebastian Ramacher.

I just wanted to point out that the "it's more work for me"
argument goes both ways.

You are working on RC bugs, and that's great (I remember for example
a FTBFS bug in gettext for which you found a fix. It really helped).

I'm working on FTBFS bugs in general, including "FTBFS randomly" bugs,
which are (in general) considered important only.

So, for every FTBFS bug which is not fixed before the release because it's not 
RC,
there is double the work for me if I want to keep stable free of FTBFS bugs
in general, because the bug has to be fixed in unstable and then backported
to stable. Note that I'm talking in general, not about tzdata-related bugs.

I know that you sometimes work on backporting fixes from FTBFS bugs to stable,
so I believe you know well what I'm talking about. Note also that I'm not trying
to use this as an argument to have "more RC bugs", I just wanted you to look
at the other side.

Thanks.

Re: Please, minimize your build chroots

[Timo Röhling]

* Wouter Verhelst  [2023-01-30 12:07]:

What is "RC" is defined by the release team, not by policy.

The release team has clarified that these bugs are not RC.

My mistake, I conflated policy violation and RC.

Other than that, my original point stands: if you think Policy is
wrong, please help fix it, don't dismiss it as "outdated" and filled
with "many bugs".


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Please, minimize your build chroots

[Wouter Verhelst]

On Sat, Jan 28, 2023 at 01:30:42PM +0100, Timo Röhling wrote:
> Hi Andreas,
> 
> * Andreas Henriksson  [2023-01-28 12:50]:
> > Policy is not a religion. Policy has many bugs. Policy is very outdated.
> > [...]
> > Here's an example you could follow:
> > https://lists.debian.org/debian-policy/2022/12/msg00023.html
> Your argument cuts both ways. Right now, Policy says that
> the bugs are RC, and if you believe that should be different, why
> don't you propose such a change and seek consensus yourself?

What is "RC" is defined by the release team, not by policy.

The release team has clarified that these bugs are not RC.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

Re: Please, minimize your build chroots

[Santiago Vila]

El 29/1/23 a las 9:56, Sebastian Ramacher escribió:

On 2023-01-28 15:55:05 -0800, Russ Allbery wrote:

Historically, we have not treated FTBFS bugs as falling into the category
of mass bug filing or requiring this pre-discussion.  Various folks have
been mass-filing FTBFS bugs near the release freeze for many years now and
they generally don't get a debian-devel discussion first.


I don't think that those are comparable. Rebuilds with modified base
chroots have been discussed here before filing bugs. [1] is one of the
oldest examples I was able to find.

Cheers

[1] https://lists.debian.org/debian-devel/2008/01/msg00869.html


There is a big difference between such experiment and what I do.

Policy 4.2 starts by saying this:

Source packages should specify which binary packages they require to be 
installed
or not to be installed in order to build correctly.

(I think it is because the "or not" part that Adrian thinks I'm trying to
open a can of worms. But that's not the case). Policy follows:

If build-time dependencies are specified, it must be possible to build the 
package and produce working binaries on a system with only essential and 
build-essential packages installed [...]

This is when we have a "must", in a clean chroot environment. So, there is a 
"must" to build
in a completely clean environment, but there is not a "must" to build in a 
dirty environment.

Therefore, I don't think it's fair at all to put both kind of environments in 
the same
bag by calling them both "modified build environments".

Thanks.

Re: Please, minimize your build chroots

[Ansgar]

On Sun, 2023-01-29 at 13:43 +0100, Santiago Vila wrote:
> What you call current practice is only current debootstrap behaviour.

Current practice is to use Build-Depends to specify build dependencies
in a way that the buildd network will successfully build packages. The
data might also be of (limited) use for other purposes.

We do *not* specify additional Build-Conflicts (that might make builds
fail / produce different results). Neither does the buildd network
require that packages already installed are listed additionally in
Build-Depends to install those and build packages successfully.

The set of preinstalled packages in build environments is decided by
the debootstrap implementation, so yes, the debootstrap behavior is
current practice.

If you want Build-{Depends,Conflicts} to provide stronger promises,
that is a change from current practice. And could be extended to
forcing /usr/local to be empty and a sane, standard environment and
contents of $HOME and anything else that could affect build results.

Ansgar

Re: Please, minimize your build chroots

[Santiago Vila]

El 28/1/23 a las 14:41, Ansgar escribió:

Johannes Schauer Marin Rodrigues writes:

I think the much more interesting question is in what environment we want to
build our packages in. Currently, on buildds, we build them in a chroot that
has Priority:required and build-essential because of (what I think is) a bug in
debootstrap: #837060


I would rather say: The build-essential packages are those installed by
debootstrap's buildd profile. At least that seems to be current practice
for a long time.


What you call current practice is only current debootstrap behaviour.

There are already packages in bullseye having build-depends
on tzdata, and afaik it was not me who reported them to have such
build-dependency. If current practice was really not to consider
tzdata as build-essential, as you say, somebody would have reported
those build-dependencies as a bug, because we don't use build-depends
when the package is build-essential.

I would say, therefore, that current practice all this time has really
been to report those bugs and fix them, i.e. current practice is
that tzdata is not build-essential, despite debootstrap behaviour.

Thanks.

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sun, Jan 29, 2023 at 05:00:56AM +0100, Guillem Jover wrote:
> On Sat, 2023-01-28 at 21:35:01 +0200, Adrian Bunk wrote:
> > I don't think such arguments are bringing us forward,
> > we should rather resolve the problem that these differ.
> > 
> > All/Most(?) packages where they do differ are packages that were until 
> > recently part of the essential set, and since debootstrap still installs 
> > them they are in practice still part of the build essential set.
> 
> Sure.
> 
> > Removing tzdata from the essential set made sense, and I do see your 
> > point that tzdata does not fit the "totally broken" definition of
> > "required".
> 
> > What we need are technical discussions like whether packages like tzdata 
> > should also be removed from the build essential set, or whether tzdata 
> > being a part of the build essential set should be expressed by making 
> > the build-essential package depend on tzdata.
> 
> I guess my question is, what makes tzdata build essential, besides
> that it's currently kind-of implicitly there? To me it does not look
> like it has the properties to be considered as such, besides that if
> we lower its priority (as it deserves) it makes a bunch of packages
> FTBFS.

It has historically been part of the build essential set.

It is used in the build of many packages.

There would be so many invisible undeclared build dependencies of 
packages using it who have it pulled in by other packages that random
changes in the dependency tree might cause many packages to FTBFS at
any time if it does not stay build essential.

It is required to provide glibc functionality that is frequently
used during the build.

> So, for one, this is getting in the way of making our minimal
> (non build) systems smaller.

No, it is not.

There are 3 different topics:

1. making minimal (non build) systems smaller

Being able to exclude tzdata from a minimal system was achieved when it 
was removed from the essential set in stretch.
debootstrap not installing it by default would make that easier.
Whether build-essential depends on tzdata does not make any difference.

2. normal systems

In a normal (non-minimal) installation not having tzdata installed 
would be a bug harming many users, no matter what priority it will
have.

3. build essential

That's separate from 1. and 2.

> > I have so far not seen any technical arguments why removing tzdata from 
> > the build essential set would be better for Debian than keeping it there.
> > Removing tzdata reduces the size of a chroot that has the build 
> > dependencies for the hello package installed by ~ 0.5%, this size
> > decrease does not strike me as a sufficient reason for reducing the
> > build essential set.
> 
> I don't see how this can be a pure technical decision, TBH. To me this
> looks more like either a matter of trade-offs,

It is a tradeoff between less work and saving ~ 0.5% space in build
chroots.

> or IMO more importantly
> of clean definition of a specification (which seems rather more
> important than the other concerns). The work to spot these problems has
> already been done, and the changes required are trivial and minimal
> (disregarding any severity consideration here).

The work has been done for packages that do FTBFS today.

I would guess ~ 90% of the packages that did have tzdata installed 
during Santiagos builds did not have or need a direct build dependency
because something else pulled it in.
It is unknown how many of these would have a latent FTBFS bug due to an
undeclared direct build dependency.

Do we have any packages that build successfully but are broken without
tzdata installed during the build?

>...
> I appreciate the minimalism and simplicity of the definition. I've
> also heard from time to time complains that we even require a C/C++
> compiler as build-essential, when for many packages that's not even
> needed (although a C compiler is currently needed by dpkg-dev to
> determine the host arch).

I would also complain that dpkg-dev pulls the full perl into the build 
essential set.

The build essential set is so huge that I wouldn't even be surprised if 
at some point in the future this discussion becomes moot because some 
package in the build essential set gains a dependency on tzdata.

Perhaps some localtime related functionality could justify a dependency 
of perl or perl-modules-5.36 on tzdata, which would keep it in the build 
essential set due to dpkg-dev being build essential?

> Policy also has this to say:
> 
>   ,---
>   If build-time dependencies are specified, it must be possible to build
>   the package and produce working binaries on a system with only
>   essential and build-essential packages installed and also those
>   required to satisfy the build-time relationships (including any
>   implied relationships). […]
>   `---
>...

I don't think this is helpful for the discussion whether packages that 
were previously part of the essential set should stay part of the build
essential set.

Re: Please, minimize your build chroots

[Sebastian Ramacher]

On 2023-01-28 15:55:05 -0800, Russ Allbery wrote:
> Sebastian Ramacher  writes:
> 
> > As with all mass bug filings, discuss the issue first on debian-devel.
> > See developer's reference 7.1.1. This discussion could have both
> > answered the questions whther RC severity is appropriate for this type
> > of bug and whether it's a bug at all.
> 
> Historically, we have not treated FTBFS bugs as falling into the category
> of mass bug filing or requiring this pre-discussion.  Various folks have
> been mass-filing FTBFS bugs near the release freeze for many years now and
> they generally don't get a debian-devel discussion first.

I don't think that those are comparable. Rebuilds with modified base
chroots have been discussed here before filing bugs. [1] is one of the
oldest examples I was able to find.

Cheers 

[1] https://lists.debian.org/debian-devel/2008/01/msg00869.html
-- 
Sebastian Ramacher

Re: Please, minimize your build chroots

[Guillem Jover]

On Sat, 2023-01-28 at 21:35:01 +0200, Adrian Bunk wrote:
> I don't think such arguments are bringing us forward,
> we should rather resolve the problem that these differ.
> 
> All/Most(?) packages where they do differ are packages that were until 
> recently part of the essential set, and since debootstrap still installs 
> them they are in practice still part of the build essential set.

Sure.

> Removing tzdata from the essential set made sense, and I do see your 
> point that tzdata does not fit the "totally broken" definition of
> "required".

> What we need are technical discussions like whether packages like tzdata 
> should also be removed from the build essential set, or whether tzdata 
> being a part of the build essential set should be expressed by making 
> the build-essential package depend on tzdata.

I guess my question is, what makes tzdata build essential, besides
that it's currently kind-of implicitly there? To me it does not look
like it has the properties to be considered as such, besides that if
we lower its priority (as it deserves) it makes a bunch of packages
FTBFS. So, for one, this is getting in the way of making our minimal
(non build) systems smaller.

> I have so far not seen any technical arguments why removing tzdata from 
> the build essential set would be better for Debian than keeping it there.
> Removing tzdata reduces the size of a chroot that has the build 
> dependencies for the hello package installed by ~ 0.5%, this size
> decrease does not strike me as a sufficient reason for reducing the
> build essential set.

I don't see how this can be a pure technical decision, TBH. To me this
looks more like either a matter of trade-offs, or IMO more importantly
of clean definition of a specification (which seems rather more
important than the other concerns). The work to spot these problems has
already been done, and the changes required are trivial and minimal
(disregarding any severity consideration here).

> Everyone can feel free to disagree with me on the previous paragraph,
> but please argue technically and not based on wording in policy.

I can see the appeal, but the problem is that if we completely ignore
policy, then we are starting from nothing, which means deciding over
this is rather open-ended. And of course, policy is not set in stone
and can be fixed, amended, extended as needed, but I think it's our
base from where to start, so requesting to completely ignore it for
this discussion seems not ideal?

The current definition of what the build essential set is supposed to
mean, from policy, seems rather easily mappable to actual package
names (which was the intention, as to not hardcode those or their
organization in policy itself):

  ,---
  It is not necessary to explicitly specify build-time relationships on
  a minimal set of packages that are always needed to compile, link and
  put in a Debian package a standard “Hello World!” program written in C
  or C++. The required packages are called *build-essential*, and an
  informational list can be found in "/usr/share/doc/build-
  essential/list" (which is contained in the "build-essential" package).
  `---

Which maps to a C and C++ compiler (including linker and C library),
make for debian/rules, and deb toolchain.

I appreciate the minimalism and simplicity of the definition. I've
also heard from time to time complains that we even require a C/C++
compiler as build-essential, when for many packages that's not even
needed (although a C compiler is currently needed by dpkg-dev to
determine the host arch).

Policy also has this to say:

  ,---
  If build-time dependencies are specified, it must be possible to build
  the package and produce working binaries on a system with only
  essential and build-essential packages installed and also those
  required to satisfy the build-time relationships (including any
  implied relationships). […]
  `---

Given that "tzdata" does not fit into the "required" definition anyway,
so the proposal to extend build-essential to include the "required"
priority does not make sense; having to slap in there, that “oh BTW, in
addition we also need timezone data, which is not going to be needed at
all by that "Hello World!" program”, seems like a rather weird and
gratuitous requirement, and hackish definition wise, just to avoid
fixing those bug reports?

There are many things we could also include in build-essential to
avoid declaring them, which we still not do, so I'm not sure why we
need to special case timezone data here.

Regards,
Guillem

Re: Please, minimize your build chroots

[Russ Allbery]

Sebastian Ramacher  writes:

> As with all mass bug filings, discuss the issue first on debian-devel.
> See developer's reference 7.1.1. This discussion could have both
> answered the questions whther RC severity is appropriate for this type
> of bug and whether it's a bug at all.

Historically, we have not treated FTBFS bugs as falling into the category
of mass bug filing or requiring this pre-discussion.  Various folks have
been mass-filing FTBFS bugs near the release freeze for many years now and
they generally don't get a debian-devel discussion first.

Now that we've identified that there is a significant project disagreement
over the definition of build-essential, bugs specifically related to that
disagreement may be worth some discussion (which is happening now).  But I
think that's new information for Santiago, not something that was obvious
retroactively.

I think it's pretty clear from this discussion that the current Policy
wording is not sufficient, and we shouldn't have ambiguous language over
something as central as what build dependencies packages need to declare.

-- 
Russ Allbery (r...@debian.org)  

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sat, Jan 28, 2023 at 10:23:19PM +0100, Santiago Vila wrote:
> El 28/1/23 a las 22:18, Adrian Bunk escribió:
> > On Sat, Jan 28, 2023 at 09:45:14PM +0100, Santiago Vila wrote:
> > > ...
> > > The other one: There are a bunch of packages whose unit tests rely on 
> > > tzdata. The tzdata
> > > package changes often during the lifetime of stable, and as a result, 
> > > some package might
> > > stop building from source. If we wanted to know in advance which packages 
> > > might break after
> > > a tzdata update, we could use the available information in the 
> > > build-depends fields.
> > > ...
> > 
> > No, that won't work.
> > 
> > In your builds, how many percent of the packages that did have tzdata
> > installed during the build did not have a direct build dependency?
> > 
> > Looking at the dependency trees, I'd assume the vast majority of
> > packages where tzdata was installed during the build do not have
> > a direct build dependency.
> 
> I think I see your point, but my idea was not to collect packages
> with tzdata in build-depends only, but those whose build-depends
> make tzdata to be installed (i.e. including transitive dependencies).
> 
> I don't know if there is already a tool for that, nor how much difficult
> it would be to have such a tool.

It shouldn't be hard to get this information from buildinfo files.

But tzdata is not unique here.

linux-libc-dev is also part of the build essential set and it is being 
updated in every point release, and this has caused FTBFS in the past.

The proper solution for the problem you describe would be a test rebuild
of the full archive at the beginning of the week between the upload freeze
for a point release and the actual release.

> Thanks.

cu
Adrian

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sat, Jan 28, 2023 at 09:45:14PM +0100, Santiago Vila wrote:
>...
> The other one: There are a bunch of packages whose unit tests rely on tzdata. 
> The tzdata
> package changes often during the lifetime of stable, and as a result, some 
> package might
> stop building from source. If we wanted to know in advance which packages 
> might break after
> a tzdata update, we could use the available information in the build-depends 
> fields.
>...

No, that won't work.

In your builds, how many percent of the packages that did have tzdata 
installed during the build did not have a direct build dependency?

Looking at the dependency trees, I'd assume the vast majority of 
packages where tzdata was installed during the build do not have
a direct build dependency.

This could easily become the nightmare situation where one changed 
dependency somewhere makes 100 packages FTBFS that do need tzdata
during the build, but previously got it installed through other
build dependencies.

> As you requested, I think the above two are technical reasons, not
> merely "because policy says so".

Thanks, I do appreciate that.

> Thanks.

cu
Adrian

Re: Please, minimize your build chroots

[Santiago Vila]

El 28/1/23 a las 22:18, Adrian Bunk escribió:

On Sat, Jan 28, 2023 at 09:45:14PM +0100, Santiago Vila wrote:

...
The other one: There are a bunch of packages whose unit tests rely on tzdata. 
The tzdata
package changes often during the lifetime of stable, and as a result, some 
package might
stop building from source. If we wanted to know in advance which packages might 
break after
a tzdata update, we could use the available information in the build-depends 
fields.
...


No, that won't work.

In your builds, how many percent of the packages that did have tzdata
installed during the build did not have a direct build dependency?

Looking at the dependency trees, I'd assume the vast majority of
packages where tzdata was installed during the build do not have
a direct build dependency.


I think I see your point, but my idea was not to collect packages
with tzdata in build-depends only, but those whose build-depends
make tzdata to be installed (i.e. including transitive dependencies).

I don't know if there is already a tool for that, nor how much difficult
it would be to have such a tool.

Thanks.

Re: Please, minimize your build chroots

[Sebastian Ramacher]

On 2023-01-28 20:44:50 +0100, Sebastian Ramacher wrote:
> On 2023-01-28 15:03:04 +0200, Adrian Bunk wrote:
> > On Sat, Jan 28, 2023 at 12:24:47PM +0100, Santiago Vila wrote:
> > >...
> > > * Those bugs are RC by definition and have been for a long time.
> > >...
> > 
> > Please provide a pointer where a release team member has said so 
> > explicitly in recent years.
> > 
> > In my experience they are usually saying that FTBFS that do not happen 
> > on the buildds of release architectures are usually not RC.
> 
> Indeed. We require that packages are buildable on the buildds. If they
> don't and they built before, they are RC buggy. For all other FTBFS
> bugs, please use severity important at most.

I am reminded that there are of course more nuances to this. But in my
opinion it's a reasonable default. For a counter example consider
packages that access the network during build (and FTBFS in a
environment without network access). They are considered to be RC buggy
even if the builds succeed on the buildds. So, if in doubt, please
contact the release team.

My personal opinion on the tzdata topic is that we are wasting developer
time with this academic issue. Currently tzdata is effectively
build-essential due to the setup of our buildds. As I see no efforts on
the debootstrap/buildd admin side to change the current situation, the
time wasted here could be better spent on fixing some of the 300ish RC
bugs that are currently affecting bookworm.

Cheers
-- 
Sebastian Ramacher

Re: Please, minimize your build chroots

[Sebastian Ramacher]

On 2023-01-28 21:56:23 +0100, Santiago Vila wrote:
> El 28/1/23 a las 20:44, Sebastian Ramacher escribió:
> > On 2023-01-28 15:03:04 +0200, Adrian Bunk wrote:
> > > On Sat, Jan 28, 2023 at 12:24:47PM +0100, Santiago Vila wrote:
> > > > ...
> > > > * Those bugs are RC by definition and have been for a long time.
> > > > ...
> > > 
> > > Please provide a pointer where a release team member has said so
> > > explicitly in recent years.
> > > 
> > > In my experience they are usually saying that FTBFS that do not happen
> > > on the buildds of release architectures are usually not RC.
> > 
> > Indeed. We require that packages are buildable on the buildds. If they
> > don't and they built before, they are RC buggy. For all other FTBFS
> > bugs, please use severity important at most.
> 
> So: What am I supposed to do when some maintainer rejects that this is a bug
> at all and closes the bug? (See #1027364 for an example).

As with all mass bug filings, discuss the issue first on debian-devel.
See developer's reference 7.1.1. This discussion could have both
answered the questions whther RC severity is appropriate for this type
of bug and whether it's a bug at all.

> I believe Adam Borowski just does not understand the current build essential
> definition. Could somebody please explain it to him? I tried and failed.

This is already covered in another subthread. But again, this could have
been avoided by following dev ref.

> Also: What I am supposed to do when some maintainer marks the bugs as 
> "unreproducible"?
> I think that's completely missing the point on what's the meaning of 
> unreproducible.

That's a completly different topic than RC severity.

Cheers
-- 
Sebastian Ramacher

Re: Please, minimize your build chroots

[Sam Hartman]

> "Santiago" == Santiago Vila  writes:

Santiago> El 28/1/23 a las 20:44, Sebastian Ramacher escribió:
>> On 2023-01-28 15:03:04 +0200, Adrian Bunk wrote:
>>> On Sat, Jan 28, 2023 at 12:24:47PM +0100, Santiago Vila wrote:
 ...  * Those bugs are RC by definition and have been for a long
 time.  ...
>>> 
>>> Please provide a pointer where a release team member has said so
>>> explicitly in recent years.
>>> 
>>> In my experience they are usually saying that FTBFS that do not
>>> happen on the buildds of release architectures are usually not
>>> RC.
>> 
>> Indeed. We require that packages are buildable on the buildds. If
>> they don't and they built before, they are RC buggy. For all
>> other FTBFS bugs, please use severity important at most.

Santiago> So: What am I supposed to do when some maintainer rejects
Santiago> that this is a bug at all and closes the bug? (See
Santiago> #1027364 for an example).

Santiago> I believe Adam Borowski just does not understand the
Santiago> current build essential definition. Could somebody please
Santiago> explain it to him? I tried and failed.

Adam has argued that policy's definition of required implied that all
Debian systems need to include required packages including build
systems.
his argument is that the best reading of policy is that you take the
union of requiremnts:

A build system is a Debian system.  So to be supported it needs to
include required packages

And it's a build system.  So it needs to have build-essential installed.

So Adam is effectively arguing there are multiple requirements that
apply that come from different parts of policy.
That seems like a reasonable reading of policy to me.
I think Ansgar's proposed change to policy avoids ambiguity by making it
clear that build-essential needs to include required.
But Adam has convinced me he is reading policy correctly.


signature.asc
Description: PGP signature

Re: Please, minimize your build chroots

[Santiago Vila]

El 28/1/23 a las 20:44, Sebastian Ramacher escribió:

On 2023-01-28 15:03:04 +0200, Adrian Bunk wrote:

On Sat, Jan 28, 2023 at 12:24:47PM +0100, Santiago Vila wrote:

...
* Those bugs are RC by definition and have been for a long time.
...


Please provide a pointer where a release team member has said so
explicitly in recent years.

In my experience they are usually saying that FTBFS that do not happen
on the buildds of release architectures are usually not RC.


Indeed. We require that packages are buildable on the buildds. If they
don't and they built before, they are RC buggy. For all other FTBFS
bugs, please use severity important at most.


So: What am I supposed to do when some maintainer rejects that this is a bug
at all and closes the bug? (See #1027364 for an example).

I believe Adam Borowski just does not understand the current build essential
definition. Could somebody please explain it to him? I tried and failed.

Also: What I am supposed to do when some maintainer marks the bugs as 
"unreproducible"?
I think that's completely missing the point on what's the meaning of 
unreproducible.

Thanks.

Re: Please, minimize your build chroots

[Santiago Vila]

El 28/1/23 a las 20:35, Adrian Bunk escribió:

I have so far not seen any technical arguments why removing tzdata from
the build essential set would be better for Debian than keeping it there.
Removing tzdata reduces the size of a chroot that has the build
dependencies for the hello package installed by ~ 0.5%, this size
decrease does not strike me as a sufficient reason for reducing the
build essential set.


I believe tzdata not being build-essential is useful for two reasons:

One of them: I've actually found *two* cases where the build failure
(when not having tzdata in the chroot) was due to a missing *binary* dependency
(of one of the build-depends).

The missing binary bug may not be very relevant, but it was discovered thanks
to using a minimal build environment (and reporting build failures), as a side 
effect.

The other one: There are a bunch of packages whose unit tests rely on tzdata. 
The tzdata
package changes often during the lifetime of stable, and as a result, some 
package might
stop building from source. If we wanted to know in advance which packages might 
break after
a tzdata update, we could use the available information in the build-depends 
fields.

Of course, not that I personally have plenty of time for that, but in a general 
sense, having
the information of which packages use tzdata for building is better than not 
having such information
anywhere.

As you requested, I think the above two are technical reasons, not
merely "because policy says so".

Thanks.

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sat, Jan 28, 2023 at 07:34:48PM +0100, Guillem Jover wrote:
> On Sat, 2023-01-28 at 16:32:17 +0100, Adam Borowski wrote:
> > On Sat, Jan 28, 2023 at 01:59:40PM +0100, Santiago Vila wrote:
> > > Unsupported by whom? What is supported or unsupported is explained in 
> > > policy.
> > > Policy says it must work. Therefore it should be supported (by fixing the 
> > > bugs).
> > 
> > Policy §2.5:
> > # "required"
> > #Packages which are necessary for the proper functioning of the
> > #system (usually, this means that dpkg functionality depends on
> > #these packages). Removing a "required" package may cause your
> > #system to become totally broken and you may not even be able to use
> > #"dpkg" to put things back, so only do so if you know what you are
> > #doing.
> 
> As stated several times now this passage seems wrong, or inaccurate at
> best. See #950440. And I don't see how tzdata would ever fall into
> this definition even if that paragraph was correct. As mentioned
> before, the tzdata package does not seem like a "required" package at
> all, and this should be fixed by lowering its priority. Whether
> debootstrap can be fixed to not use the Priority workaround, seem
> orthogonal to the issue at hand.
> 
> > > That's a straw man. I'm not proposing anything of the sort. Policy says
> > > packages must build when essential and build-essential packages
> > > are installed (plus build-dependencies).
> > 
> > Build-essential _packages_.  Not the "build-essential" package which very
> > clearly says its dependencies are purely informational.
> 
> It does not seem fair to argue both that the build-essential package is
> just informational (when it's in fact the canonical declaration of what
> is Build-Essential, and what every tool uses to install or check for the
> Build-Essential package set), and then also argue that whatever
> debootstrap installs (which is based both on build-essential plus a
> workaround due to lack of proper dependency resolution) is the canonical
> thing.

I don't think such arguments are bringing us forward,
we should rather resolve the problem that these differ.

All/Most(?) packages where they do differ are packages that were until 
recently part of the essential set, and since debootstrap still installs 
them they are in practice still part of the build essential set.

Removing tzdata from the essential set made sense, and I do see your 
point that tzdata does not fit the "totally broken" definition of
"required".

What we need are technical discussions like whether packages like tzdata 
should also be removed from the build essential set, or whether tzdata 
being a part of the build essential set should be expressed by making 
the build-essential package depend on tzdata.

I have so far not seen any technical arguments why removing tzdata from 
the build essential set would be better for Debian than keeping it there.
Removing tzdata reduces the size of a chroot that has the build 
dependencies for the hello package installed by ~ 0.5%, this size
decrease does not strike me as a sufficient reason for reducing the
build essential set.

Everyone can feel free to disagree with me on the previous paragraph,
but please argue technically and not based on wording in policy.

> Regards,
> Guillem

cu
Adrian

Re: Please, minimize your build chroots

[Sebastian Ramacher]

On 2023-01-28 15:03:04 +0200, Adrian Bunk wrote:
> On Sat, Jan 28, 2023 at 12:24:47PM +0100, Santiago Vila wrote:
> >...
> > * Those bugs are RC by definition and have been for a long time.
> >...
> 
> Please provide a pointer where a release team member has said so 
> explicitly in recent years.
> 
> In my experience they are usually saying that FTBFS that do not happen 
> on the buildds of release architectures are usually not RC.

Indeed. We require that packages are buildable on the buildds. If they
don't and they built before, they are RC buggy. For all other FTBFS
bugs, please use severity important at most.

Cheers
-- 
Sebastian Ramacher

Re: Please, minimize your build chroots

[Guillem Jover]

On Sat, 2023-01-28 at 16:32:17 +0100, Adam Borowski wrote:
> On Sat, Jan 28, 2023 at 01:59:40PM +0100, Santiago Vila wrote:
> > Unsupported by whom? What is supported or unsupported is explained in 
> > policy.
> > Policy says it must work. Therefore it should be supported (by fixing the 
> > bugs).
> 
> Policy §2.5:
> # "required"
> #Packages which are necessary for the proper functioning of the
> #system (usually, this means that dpkg functionality depends on
> #these packages). Removing a "required" package may cause your
> #system to become totally broken and you may not even be able to use
> #"dpkg" to put things back, so only do so if you know what you are
> #doing.

As stated several times now this passage seems wrong, or inaccurate at
best. See #950440. And I don't see how tzdata would ever fall into
this definition even if that paragraph was correct. As mentioned
before, the tzdata package does not seem like a "required" package at
all, and this should be fixed by lowering its priority. Whether
debootstrap can be fixed to not use the Priority workaround, seem
orthogonal to the issue at hand.

> > That's a straw man. I'm not proposing anything of the sort. Policy says
> > packages must build when essential and build-essential packages
> > are installed (plus build-dependencies).
> 
> Build-essential _packages_.  Not the "build-essential" package which very
> clearly says its dependencies are purely informational.

It does not seem fair to argue both that the build-essential package is
just informational (when it's in fact the canonical declaration of what
is Build-Essential, and what every tool uses to install or check for the
Build-Essential package set), and then also argue that whatever
debootstrap installs (which is based both on build-essential plus a
workaround due to lack of proper dependency resolution) is the canonical
thing.

Regards,
Guillem

Re: Please, minimize your build chroots

[Holger Levsen]

On Sat, Jan 28, 2023 at 03:11:39PM +0100, Johannes Schauer Marin Rodrigues 
wrote:
> And I asked in my mail to please "decouple the policy and bug severity 
> question
> from the question of what a buildd chroot should contain" for a reason.

yes, I know. my point was that too many people won't be able to do this
(which this thread proves again.)
 
> Discussing policy questions and bug severity distracts from the actual 
> question
> that I find interesting.

yes.

(and as (too many) people are not able to do this my suggestion is to
lower the severity of these bugs. not every policy violation is RC.)


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Some of my friends and I overcommit to things, so we made "Saying No to Things"
punch cards. If you say no to 10 things, your friends have to buy you an ice
cream. In a pilot study, we found participants both said no to more things and
got more free ice cream. (@leah_pierson)


signature.asc
Description: PGP signature

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sat, Jan 28, 2023 at 03:28:58PM +0100, Johannes Schauer Marin Rodrigues 
wrote:
>...
> My proposal is to fix debootstrap #837060 (patch is in the bug report) so that
> it only installs Essential:yes, build-essential and apt and discuss if it 
> makes
> sense to have packages like tzdata or e2fsprogs in a buildd chroot or not and
> if yes, add those packages as dependencies of the build-essential package.
>...

Note that there are at least 2 potential reasons why a package should 
stay in the build essential set:

1. many users, like tzdata

2. Important: yes
Making e2fsprogs not build essential would make it legal to do
  Build-Conflicts: e2fsprogs
It might avoid problems in the future to make such nearly-essential 
packages apt refuses to remove build-essential, otherwise there
could be problems like dose3 sending packages to a buildd where
apt refuses to fulfill the Build-Conflicts.

> Thanks!
> 
> cheers, josch

cu
Adrian

Re: Please, minimize your build chroots

[Ansgar]

Johannes Schauer Marin Rodrigues wrote:
> Do you agree that the software in our archive should agree on which
> packages are needed to build a source package? Currently, they do
> not. dose3 requires only Essential:yes and build-essential being
> installed. Who is buggy?

Well, the /informational/ list in build-essential could be updated if
people find that required packages should be listed there explicitly as
well.

I find it not very interesting to try and reduce the set of packages
used considered build-essential (which does include required packages
in practice as buildds have those installed).

(Also we do consider not installing "required" packages unsupported as
per the description of what "required" is; so if your build environment
doesn't include it, you are on your own.)

Ansgar

Re: Please, minimize your build chroots

[Adam Borowski]

On Sat, Jan 28, 2023 at 01:59:40PM +0100, Santiago Vila wrote:
> El 28/1/23 a las 12:50, Andreas Henriksson escribió:
> 
> > Claiming there's no point to free software when the problem is simply
> > that you are using an *unsupported* setup?!?!
> 
> Unsupported by whom? What is supported or unsupported is explained in policy.
> Policy says it must work. Therefore it should be supported (by fixing the 
> bugs).

Policy §2.5:
# "required"
#Packages which are necessary for the proper functioning of the
#system (usually, this means that dpkg functionality depends on
#these packages). Removing a "required" package may cause your
#system to become totally broken and you may not even be able to use
#"dpkg" to put things back, so only do so if you know what you are
#doing.

> That's a straw man. I'm not proposing anything of the sort. Policy says
> packages must build when essential and build-essential packages
> are installed (plus build-dependencies).

Build-essential _packages_.  Not the "build-essential" package which very
clearly says its dependencies are purely informational.

> You and others are essentially saying I should not follow policy

No, we are saying that we believe your understanding of the policy is
flawed.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Let's make a Debian conference in Yalta, Ukraine.
⢿⡄⠘⠷⠚⠋⠀
⠈⠳⣄

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Ansgar (2023-01-28 14:41:31)
> Johannes Schauer Marin Rodrigues writes:
> > I think the much more interesting question is in what environment we want to
> > build our packages in. Currently, on buildds, we build them in a chroot that
> > has Priority:required and build-essential because of (what I think is) a 
> > bug in
> > debootstrap: #837060
> 
> I would rather say: The build-essential packages are those installed by
> debootstrap's buildd profile. At least that seems to be current practice
> for a long time.

Do you agree that the software in our archive should agree on which packages
are needed to build a source package? Currently, they do not. dose3 requires
only Essential:yes and build-essential being installed. Who is buggy? If you
think that dose3 is buggy, who is writing the dose3 patch? Is it not easier to
just fix debootstrap so that debootstraps buildd profile does the same thing as
most other pieces of software resolving build dependencies in the archive? I
know of no other build dependency resolving software in Debian that considers
Priority:required packages to be required for building source packages other
than debootstrap. Do you?

> > So there are two ways forward:
> >
> >  1. accept that Priority:required is needed for building source packages
> >
> >  - adapt Debian policy accordingly
> 
> AFAIU it would not need changes? Policy doesn't seem to explicitly state
> what packages are actually build-essential...

I'm not sure. But I also find the policy question less interesting. I find it
more interesting to first agree how the final solution should look like and
then change policy accordingly (if needed).

> >  - revert the changes to packages made due to Santiago's bugs - change
> >  all tools that do build dependency resolution to now also consider
> >  Priority:required packages
> 
> Why would they need changes? Do they explicitly include essential
> packages too?

Of course they do not do so explicitly, because all Essential:yes packages are
dependencies (and build dependencies) of all packages already *implicitly*.

> >  2. make sure that packages are built without Priority:required packages
> 
> > To me it seems that we nearly are already at (2).
> 
> I think we are already at (1) given everything works already?

Aha. Try running "apt-get build-dep" on a system without tzdata nor e2fsprogs
and witness how apt will not install tzdata nor e2fsprogs. The only reason this
usually works right now is because debootstrap installs Priority:required by
default in all situations including the buildd variant.

My proposal is to fix debootstrap #837060 (patch is in the bug report) so that
it only installs Essential:yes, build-essential and apt and discuss if it makes
sense to have packages like tzdata or e2fsprogs in a buildd chroot or not and
if yes, add those packages as dependencies of the build-essential package.

I do not propose to do this before bookworm release.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Holger Levsen (2023-01-28 14:53:37)
> On Sat, Jan 28, 2023 at 02:28:30PM +0100, Johannes Schauer Marin Rodrigues 
> wrote:
> > could we decouple the policy and bug severity question from the question of
> > what a buildd chroot should contain, please?
> [...]
> > Why do people call just accepting that Priority:required packages have to be
> > part of the buildd chroot the easier solution? [...]
> 
> because people get upset if they receive bug reports with severity serious
> when they don't perceive these bugs as serious.
> 
> happened more than 9000 times already.

And I asked in my mail to please "decouple the policy and bug severity question
from the question of what a buildd chroot should contain" for a reason.

I have no problem with downgrading the severity of these bugs to non-RC and
release bookworm without explicit build dependencies on e2fsprogs.

Discussing policy questions and bug severity distracts from the actual question
that I find interesting.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sat, Jan 28, 2023 at 02:28:30PM +0100, Johannes Schauer Marin Rodrigues 
wrote:
>...
> Why do people call just accepting that Priority:required packages have to be
> part of the buildd chroot the easier solution? We just need to fix debootstrap
> bug #837060 and we are done, no?

This is mostly a new problem, a side effect of a different recent change:
The efforts to reduce the essential set for enabling smaller installs
of Debian.

"Priority: required" packages like e2fsprogs and tzdata used to be part 
of the essential set, tzdata is no longer (transitively) essential since
stretch and e2fsprogs no longer essential since buster.

It was not an intended goal to remove such packages from the *build* 
essential set, and they are installed in all reasonable build 
environments which makes it a non-issue in practice.

Adding them to build-essential would just enforce what was enforced 
differently in the past, and what is still true in practice today.

It should at least be discussed first whether packages like tzdata that 
have been a part of the build essential set should stay there.

> Thanks!
> 
> cheers, josch

cu
Adrian

Re: Please, minimize your build chroots

[Holger Levsen]

On Sat, Jan 28, 2023 at 02:28:30PM +0100, Johannes Schauer Marin Rodrigues 
wrote:
> could we decouple the policy and bug severity question from the question of
> what a buildd chroot should contain, please?
[...]
> Why do people call just accepting that Priority:required packages have to be
> part of the buildd chroot the easier solution? [...]

because people get upset if they receive bug reports with severity serious
when they don't perceive these bugs as serious.

happened more than 9000 times already.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

This too shall pass.


signature.asc
Description: PGP signature

Re: Please, minimize your build chroots

[Shengjing Zhu]

On Sat, Jan 28, 2023 at 9:29 PM Johannes Schauer Marin Rodrigues
 wrote:
> To me it seems that we nearly are already at (2). The debootstrap bug #837060
> has a working patch and mmdebstrap exists that can do what is needed today.
> Santiago did an archive rebuild to make sure our source package compile in a
> chroot without Priority:required.
>
> Why do people call just accepting that Priority:required packages have to be
> part of the buildd chroot the easier solution? We just need to fix debootstrap
> bug #837060 and we are done, no?
>

Yes, please fix debootstrap or anything else first, and ensure buildd
uses the fixed version.

If you can't convince the buildd to follow the policy, why should the
individual package maintainers?

-- 
Shengjing Zhu

Re: Please, minimize your build chroots

[Ansgar]

Johannes Schauer Marin Rodrigues writes:
> I think the much more interesting question is in what environment we want to
> build our packages in. Currently, on buildds, we build them in a chroot that
> has Priority:required and build-essential because of (what I think is) a bug 
> in
> debootstrap: #837060

I would rather say: The build-essential packages are those installed by
debootstrap's buildd profile. At least that seems to be current practice
for a long time.

> So there are two ways forward:
>
>  1. accept that Priority:required is needed for building source packages
>
>  - adapt Debian policy accordingly

AFAIU it would not need changes? Policy doesn't seem to explicitly state
what packages are actually build-essential...

>  - revert the changes to packages made due to Santiago's bugs
>  - change all tools that do build dependency resolution to now also
>consider Priority:required packages

Why would they need changes? Do they explicitly include essential
packages too?

>  2. make sure that packages are built without Priority:required packages

> To me it seems that we nearly are already at (2).

I think we are already at (1) given everything works already?

Ansgar

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Timo Röhling (2023-01-28 13:30:42)
> Hi Andreas,
> 
> * Andreas Henriksson  [2023-01-28 12:50]:
> >Policy is not a religion. Policy has many bugs. Policy is very outdated.
> >[...]
> >Here's an example you could follow:
> >https://lists.debian.org/debian-policy/2022/12/msg00023.html
> Your argument cuts both ways. Right now, Policy says that
> the bugs are RC, and if you believe that should be different, why don't you
> propose such a change and seek consensus yourself?

could we decouple the policy and bug severity question from the question of
what a buildd chroot should contain, please?

Santiago did the work, filed bugs and the fix is to add one more line to
d/control. If you want to reproduce it, use `mmdebstrap --variant=apt
--include=build-essential unstable chroot.tar` to create your chroot tarball.

I think the much more interesting question is in what environment we want to
build our packages in. Currently, on buildds, we build them in a chroot that
has Priority:required and build-essential because of (what I think is) a bug in
debootstrap: #837060

So there are two ways forward:

 1. accept that Priority:required is needed for building source packages

 - adapt Debian policy accordingly
 - revert the changes to packages made due to Santiago's bugs
 - change all tools that do build dependency resolution to now also
   consider Priority:required packages

 2. make sure that packages are built without Priority:required packages

 - fix debootstrap #837060 or use mmdebstrap to create buildd chroots
 - Santiago already did a mass-rebuild and submitted bugs to make sure that
   packages do not FTBFS

The last time I changed all the tools involved in build dependency resolution I
had to submit patches to dpkg, sbuild, apt, dose3, debhelper, cdbs, pbuilder,
lintian, wanna-build, devscripts and others. Of those who prefer option (1)
over option (2) who is going to investigate and potentially change all the
tools who currently just add the "build-essential" package? Or is the solution
to add a dependency to build-essential on all Priority:required packages?

To me it seems that we nearly are already at (2). The debootstrap bug #837060
has a working patch and mmdebstrap exists that can do what is needed today.
Santiago did an archive rebuild to make sure our source package compile in a
chroot without Priority:required.

Why do people call just accepting that Priority:required packages have to be
part of the buildd chroot the easier solution? We just need to fix debootstrap
bug #837060 and we are done, no?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Please, minimize your build chroots

[Vincent Bernat]

On 2023-01-28 13:59, Adrian Bunk wrote:

I am not saying that trying to force maintainers to spend time on such
issues by making them release critical is better, but you are also
creating extra work and frustration for the people who are doing QA work
in Debian.


It also pushes some maintainers to give up on packages. I gave up on 
maintaining any Go package after the whole "everything should compile 
with only one CPU because policy says so" fiasco. The most rare resource 
we have is volunteer time. Creating artificial problems is not helping.

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sat, Jan 28, 2023 at 12:24:47PM +0100, Santiago Vila wrote:
>...
> * Those bugs are RC by definition and have been for a long time.
>...

Please provide a pointer where a release team member has said so 
explicitly in recent years.

In my experience they are usually saying that FTBFS that do not happen 
on the buildds of release architectures are usually not RC.

> Thanks.

cu
Adrian

Re: Please, minimize your build chroots

[Adrian Bunk]

On Sat, Jan 28, 2023 at 12:20:16AM +0100, Santiago Vila wrote:
> El 27/1/23 a las 22:37, Adrian Bunk escribió:
> > On Fri, Dec 16, 2022 at 02:15:13AM +0100, Santiago Vila wrote:
...
> > I am right now looking at #1027382, and the first question is how I can
> > make apt remove e2fsprogs so that I can reproduce the problem - it feels
> > like a real waste of my QA work to "fix" something that is incredibly
> > hard to break.
> 
> You don't have to fix #1027382. The maintainer has.
>...

Reality in Debian is that at any time a 3 digit number of maintainers is
(sort-term or long-term or permanently) away/busy/MIA.

A large part of QA work in Debian is fixing bugs in packages maintained 
by other people, for me that's > 95% of my uploads.

I am not saying that trying to force maintainers to spend time on such 
issues by making them release critical is better, but you are also 
creating extra work and frustration for the people who are doing QA work 
in Debian.

> > It has been practice for many years that FTBFS that do not happen on the
> > buildds are usually not release critical bugs, and I would appreciate if
> > this is followed by everyone.
> 
> Well, that's also wrong, because having the build-dependencies correct has 
> been
> in the list of RC bugs for many years as well. See below.
> 
> I would appreciate if we all followed Policy 4.2, which says packages MUST 
> build
> when the build-dependencies are installed.

Policy tends to be a decade behind reality because it *follows* 
existing practices.

If existing practice is that build environments have all
"Priority: required" packages installed, then policy should
be updated to follow reality.

> In general, disputing the severity because it does not happen in the buildds
> misses completely the point of what should be the goal, namely, a distribution
> which may be rebuilt by everybody following documented procedures, not
> a distribution which may only be rebuilt in our buildds.

People who follow the (incomplete) documentation in our wiki for 
creating their own buildd setup will get a reasonable setup where
builds have all "Priority: required" packages installed.

> The end user MUST be able to rebuild the packages. Otherwise our
> free software licenses are meaningless in practice.

In general this is true, but you tend to use this argument when trying 
force pretty unimportant things on the whole project.

#932795 was your failed attempt to make the Technical Committee decide
that all build failures on single-core machines are release critical 
bugs in the year 2019.

> > It is not helpful if people try to force the few people who are doing
> > QA work to spend their scarce QA time on fixing bugs that only happen
> > when building on single-core machines or in non-UTF-8 locales or without
> > packages that are in practice installed everywhere, by making such
> > issues that are not a problem on our buildds release critical bugs.
> 
> That's the wrong approach. If the end user wants to make a modification,
> they can't use our buildd network.

In #932795 there was wide consensus that documentation could be improved 
to clarify what a supported/reasonable build environment is.

E.g. the maximum amount of diskspace a package might currently use when 
building on amd64 is the undocumented tribal knowledge how much storage
is available on our amd64 buildds.

> > It also opens a gigantic can of worms, since there is the even bigger
> > opposite problem that many packages FTBFS or are built differently
> > when built in an environment that differs from our buildd setup.
> > Adding Build-Conflicts for all such cases is not feasible in practice.
> 
> This is a straw-man. I'm not opening any can of worms.

Your argument is based on treating Policy 4.2 as a holy scripture that
must be followed and never questioned.

Policy 4.2 also says
  Source packages should specify which binary packages they require to 
  be installed or not to be installed in order to build correctly.

We are not following the "not to be installed" part,
which is the can of worms you would be opening.

In the section you are referring to, Policy 4.2 also says 
  In particular, this means that version clauses should be used 
  rigorously in build-time relationships so that one cannot produce bad or 
  inconsistently configured packages when the relationships are properly 
  satisfied.

Personally I would strongly agree with that, but current practice with 
Janitor commits removing older version clauses goes in the opposite 
direction.

> > If people want to support building without tzdata [...]> but none of these 
> > are critical for our releases since
> > none of these impact how packages are built for bookworm on our buildds.
> 
> There is a list of RC issues. It's called Release Policy, and it says this:
> 
> Packages must list any packages they require to build beyond those
> that are "build-essential" in the appropriate Build-Depends: fields.
> Ref: 4.2
> 
> Source: 

Re: Please, minimize your build chroots

[Santiago Vila]

El 28/1/23 a las 13:59, Adrian Bunk escribió:

Policy 4.2 also says
   Source packages should specify which binary packages they require to
   be installed or not to be installed in order to build correctly.

We are not following the "not to be installed" part,
which is the can of worms you would be opening.


No, I'm not proposing any kind of mass proliferation of build-conflicts,
and have never proposed anything of the sort.

The strawman arguments should stop. Now.

Thanks.

Re: Please, minimize your build chroots

[Santiago Vila]

El 28/1/23 a las 12:50, Andreas Henriksson escribió:

Policy is not a religion. Policy has many bugs. Policy is very outdated.


buildd is not a religion. buildd has bugs, etc.


Claiming there's no point to free software when the problem is simply
that you are using an *unsupported* setup?!?!


Unsupported by whom? What is supported or unsupported is explained in policy.
Policy says it must work. Therefore it should be supported (by fixing the bugs).


All debootstrap variants
include Priority: required packages. As you can see they do so for a
reason!


Yes, because debootstrap has a bug. So no, there is not a reason other
than debootstrap insistence that this should be fixed by downgrading
severities.


The --exclude option of debootstrap works equally well even on
Essential: yes packages.


That's a straw man. I'm not proposing anything of the sort. Policy says
packages must build when essential and build-essential packages
are installed (plus build-dependencies).


If you think people should be able to build on top of their regular
install with various packages installed and various configurations it


Another straw man. I'm not proposing anything of the sort.


If you think every packages should list just about all of the archive in
Build-Conflicts just to not pick up unwanted extra autodetected
dependencies that make the package FTBFS then I think it would be


Straw man. I'm not proposing anything like that.

Please stop.

You and others are essentially saying I should not follow policy
and release policy (when it's absolutely trivial to do so) but instead
some sloppy rule which is against policy and release policy.

That kind of coercion to NOT follow policy must stop. Seriously.

Thanks.

Re: Please, minimize your build chroots

[Timo Röhling]

Hi Andreas,

* Andreas Henriksson  [2023-01-28 12:50]:

Policy is not a religion. Policy has many bugs. Policy is very outdated.
[...]
Here's an example you could follow:
https://lists.debian.org/debian-policy/2022/12/msg00023.html

Your argument cuts both ways. Right now, Policy says that
the bugs are RC, and if you believe that should be different, why
don't you propose such a change and seek consensus yourself?


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Please, minimize your build chroots

[Andreas Henriksson]

Hello,

I've previously discussed this topic with Santiago Vila on a bug report
but sharing my opinion here for the wider audience.

On Sat, Jan 28, 2023 at 12:20:16AM +0100, Santiago Vila wrote:
> El 27/1/23 a las 22:37, Adrian Bunk escribió:
[...]
> This is not the right time to change policy.

This is the right time to question your interpretation of policy.

Policy is not a religion. Policy has many bugs. Policy is very outdated.
Doing something just because an outdated document says so is worse than
not doing anything in my opinion. You need to understand if what you're
actually doing is correct and then see if policy agrees with you to
justify the severity.
Or as the saying goes: Policy is not a stick to beat people with!

Here's an example you could follow:
https://lists.debian.org/debian-policy/2022/12/msg00023.html
(I don't agree with the suggestion. It was however suggested on the
correct list and since there was no traction to the suggestion it was
not followed up with mass bug filing. Minimal disruption to the project.
Updating the definition/description of the Priority levels would however
be very useful. The description seem very outdated, to the point of
being irrelevant, by now.)

[...]
> In general, disputing the severity because it does not happen in the buildds
> misses completely the point of what should be the goal, namely, a distribution
> which may be rebuilt by everybody following documented procedures, not
> a distribution which may only be rebuilt in our buildds.
> 
> The end user MUST be able to rebuild the packages. Otherwise our
> free software licenses are meaningless in practice.

Claiming there's no point to free software when the problem is simply
that you are using an *unsupported* setup?!?! All debootstrap variants
include Priority: required packages. As you can see they do so for a
reason!
The --exclude option of debootstrap works equally well even on
Essential: yes packages. Being able to experiment like this is great!
It is however still just an experiment which does not warrant filing
release-critical bugs against various packages.

> > It is not helpful if people try to force the few people who are doing
> > QA work to spend their scarce QA time on fixing bugs that only happen
> > when building on single-core machines or in non-UTF-8 locales or without
> > packages that are in practice installed everywhere, by making such
> > issues that are not a problem on our buildds release critical bugs.
> 
> That's the wrong approach. If the end user wants to make a modification,
> they can't use our buildd network.
[...]

There are alot of packages which does not properly build in an unclean
environment. In practise you need a controlled build environment to
properly build debian archive packages form source (which starts with
deboostrap --variant=buildd ...).
If you think people should be able to build on top of their regular
install with various packages installed and various configurations it
would be much better if you tried to fix the many packages who fails in
this scenario.
If you think every packages should list just about all of the archive in
Build-Conflicts just to not pick up unwanted extra autodetected
dependencies that make the package FTBFS then I think it would be
a good idea to check if there's actually project consensus on that.
If you think that every package must use configure flags to override
autodetection of features, seek project consensus. Discuss it on the
policy list, where they'll probably tell you to first fix the archive
(without filing RC-bugs) and then come back.

If you want to do mass bug filing of release-critical severity you
better make sure there's project consensus on what you're doing and
than you're not just wasting everybodys limited time.
Multiple people have now raised their conserns with your approach.
As I've said before I appreciate the bug reports being filed,
even if it's just for "theoretical correctness", but a proper
severity would be wishlist. If your only motivation for this is
to be allowed to file release-critical bugs, then I think it's
better if you just stop.

Regards,
Andreas Henriksson

Re: Please, minimize your build chroots

[Santiago Vila]

El 28/1/23 a las 10:11, Vincent Bernat escribió:

On 2023-01-28 00:20, Santiago Vila wrote:

Release Policy exists as a canonical list of what should be RC and > what not, 
and it's intended to avoid stupid discussions like this one.


Extending build-essential is easier than asking many people to do pointless 
work to satisfy a set of non-existing users. It is not like we had several 
reports of people complaining they can't build a package because they are in an 
environment without tzdata. It is not OK to create problems to force many 
volunteers to do extra work.


Of course, declaring something as not a bug is easier than fixing the bug.

But there are several points to consider:

* The "extra work" is adding a single line to debian/control.
* This is not the right time for a policy change.
* I already filed all the bugs I found after building the whole of bookworm 
from source. I do not expect
there are a lot of unreported bugs right now of that type.
* Most of the bugs I reported about this are already fixed.
* Those bugs are a drop in the ocean of all FTBFS bugs which happened during 
this release cycle.
* Those bugs are RC by definition and have been for a long time.

Therefore, this discussion is absolutely pointless.

So please stop harassing me about this issue.

Thanks.

Re: Please, minimize your build chroots

[Vincent Bernat]

On 2023-01-28 00:20, Santiago Vila wrote:

Release Policy exists as a canonical list of what should be RC and > what not, 
and it's intended to avoid stupid discussions like this one.


Extending build-essential is easier than asking many people to do 
pointless work to satisfy a set of non-existing users. It is not like we 
had several reports of people complaining they can't build a package 
because they are in an environment without tzdata. It is not OK to 
create problems to force many volunteers to do extra work.

Re: Please, minimize your build chroots

[Santiago Vila]

El 27/1/23 a las 22:37, Adrian Bunk escribió:

On Fri, Dec 16, 2022 at 02:15:13AM +0100, Santiago Vila wrote:

Greetings.

I'm doing archive-wide rebuilds again.

I've just filed 21 bugs with subject "Missing build-depends on tzdata"
in bookworm (as tzdata is not build-essential).

This is of course not fun for the maintainers, but it's also not fun
for people doing QA, because those bugs could be caught earlier in the
chain, but they are not. This is extra work for everybody.


Speaking as someone who is doing a lot of QA work, for *build*
environments I would rather expand build-essential instead of doing
extra QA work that consists of manually adding build dependencies for
packages that are in practice anyway installed in all build
environments.


This is not the right time to change policy.


There are important real-world usecases where reducing the essential set
brings benefits, but for *build* essential there are not really benefits
that are worth the extra work.


That's your opinion, and I believe you are wrong. I explained the benefits
in the aborted debian-policy proposal to clarify build-essential.


I am right now looking at #1027382, and the first question is how I can
make apt remove e2fsprogs so that I can reproduce the problem - it feels
like a real waste of my QA work to "fix" something that is incredibly
hard to break.


You don't have to fix #1027382. The maintainer has.

Also, you don't have to use apt. You can use dpkg --purge inside the chroot.
Easy enough. Or you can fix the infamous debootstrap bug which is the
root of all this.


It has been practice for many years that FTBFS that do not happen on the
buildds are usually not release critical bugs, and I would appreciate if
this is followed by everyone.


Well, that's also wrong, because having the build-dependencies correct has been
in the list of RC bugs for many years as well. See below.

I would appreciate if we all followed Policy 4.2, which says packages MUST build
when the build-dependencies are installed.

In general, disputing the severity because it does not happen in the buildds
misses completely the point of what should be the goal, namely, a distribution
which may be rebuilt by everybody following documented procedures, not
a distribution which may only be rebuilt in our buildds.

The end user MUST be able to rebuild the packages. Otherwise our
free software licenses are meaningless in practice.


It is not helpful if people try to force the few people who are doing
QA work to spend their scarce QA time on fixing bugs that only happen
when building on single-core machines or in non-UTF-8 locales or without
packages that are in practice installed everywhere, by making such
issues that are not a problem on our buildds release critical bugs.


That's the wrong approach. If the end user wants to make a modification,
they can't use our buildd network.


It also opens a gigantic can of worms, since there is the even bigger
opposite problem that many packages FTBFS or are built differently
when built in an environment that differs from our buildd setup.
Adding Build-Conflicts for all such cases is not feasible in practice.


This is a straw-man. I'm not opening any can of worms.


If people want to support building without tzdata [...]> but none of these are 
critical for our releases since
none of these impact how packages are built for bookworm on our buildds.


There is a list of RC issues. It's called Release Policy, and it says this:

Packages must list any packages they require to build beyond those
that are "build-essential" in the appropriate Build-Depends: fields.
Ref: 4.2

Source: https://release.debian.org/testing/rc_policy.txt

Release Policy exists as a canonical list of what should be RC and
what not, and it's intended to avoid stupid discussions like this one.

So, according to Release Policy, your claim that a missing build-depends
is not release critical is false by definition.

Please do not spread misinformation about what is RC and what is not
when we already have a canonical list for RC issues.

Thanks.

Re: Please, minimize your build chroots

[Adrian Bunk]

On Fri, Dec 16, 2022 at 02:15:13AM +0100, Santiago Vila wrote:
> Greetings.
> 
> I'm doing archive-wide rebuilds again.
> 
> I've just filed 21 bugs with subject "Missing build-depends on tzdata"
> in bookworm (as tzdata is not build-essential).
> 
> This is of course not fun for the maintainers, but it's also not fun
> for people doing QA, because those bugs could be caught earlier in the
> chain, but they are not. This is extra work for everybody.

Speaking as someone who is doing a lot of QA work, for *build* 
environments I would rather expand build-essential instead of doing 
extra QA work that consists of manually adding build dependencies for 
packages that are in practice anyway installed in all build 
environments.

There are important real-world usecases where reducing the essential set 
brings benefits, but for *build* essential there are not really benefits 
that are worth the extra work.

>...
> Because people accept the default by debootrap "as is", chroots used
> to build packages include packages which are neither essential nor
> build-essential, like tzdata, mount or e2fsprogs.
>...

I am right now looking at #1027382, and the first question is how I can 
make apt remove e2fsprogs so that I can reproduce the problem - it feels 
like a real waste of my QA work to "fix" something that is incredibly 
hard to break.

It has been practice for many years that FTBFS that do not happen on the 
buildds are usually not release critical bugs, and I would appreciate if
this is followed by everyone.

It is not helpful if people try to force the few people who are doing
QA work to spend their scarce QA time on fixing bugs that only happen 
when building on single-core machines or in non-UTF-8 locales or without 
packages that are in practice installed everywhere, by making such 
issues that are not a problem on our buildds release critical bugs.

It also opens a gigantic can of worms, since there is the even bigger 
opposite problem that many packages FTBFS or are built differently
when built in an environment that differs from our buildd setup.
Adding Build-Conflicts for all such cases is not feasible in practice.

If people want to support building without tzdata, or cross-building, or 
building for non-release architectures, then bugs with patches are of
course welcome - but none of these are critical for our releases since
none of these impact how packages are built for bookworm on our buildds.

> Thanks.

Thanks
Adrian

Re: Please, minimize your build chroots

[David Kalnischkies]

On Sun, Dec 18, 2022 at 06:08:57PM +0100, Johannes Schauer Marin Rodrigues 
wrote:
> Quoting David Kalnischkies (2022-12-18 17:18:28)
> > On Fri, Dec 16, 2022 at 03:38:17PM +0100, Santiago Vila wrote:
> > > Then there is "e2fsprogs", which apt seems to treat as if it were
> > > an essential package:
> > > 
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826587
> > 
> > As Julian explained, it is considered "essential" because the maintainer
> > said so. If you don't think e2fsprogs should be considered "essential"
> > for a system it is installed on please talk to the maintainer.
> > 
> > Sure, the package is not (anymore) really "Essential:yes", but 'apt'
> > isn't either and will print the same message anyhow. I don't think it
> > would be a net-benefit for a user to invent words for different types of
> > essentialness in apt because in the end you either know what you are
> > doing while removing a somewhat essential package and continue or you don't
> > know what you are doing and (hopefully) get the hell out.
> 
> would it be so difficult to cater to both kind of users? For those who do not
> know the terminology, using the word "essential" is probably fine. But for
> those who do it's confusing. Why can apt not say something like:
> 
> WARNING: The following packages will be removed. Apt considers them essential
> because they are marked as Priority:required. This should NOT be done unless
> you know exactly what you are doing!

This is objectively wronger™ through: prio:required packages are not
considered essential by apt. Most are for other reasons, but priority
has nothing to do with it. The same "you are about to remove an
essential package" (paraphrased) message is shown for:
- packages marked as Essential:yes in ANY [native] version known to apt
  (if you don't modify that behaviour with pkgCacheGen::Essential)
- packages marked as Important:yes/Protected:yes in ANY [native] version
  (surprisingly Julian has not added a option here… m)
- binary packages listed via the pkgCacheGen::ForceEssential option,
  (the list can NOT be empty, it will default to "apt")
- binary packages listed via the pkgCacheGen::ForceImportant option
  (empty list by default)
- packages that are (pre-)dependencies of the other points if that
  package is removed, too.

(Note that the mentioned options do work only if you generate a cache
 and also 'taint' that cache meaning that a reused cache later without
 those options will still behave as if they were given.
 You have been warned.)

The later ensures that you can e.g. change awk providers, but be smacked
with a huge clue bat if you remove the last provider, even if that
happens to be the prio:optional gawk which as a package itself doesn't
look like it would be essential in any way without going into a lot of
details completely lost on most apt users (for good reason, after all,
if you wanted to know all that, you would probably do dpkg by hand or
at least maintain apt… and nobody wants to do THAT, am I right…)

Also: "marked as …" – by whom? If you say it like that, a user might
think they did; like they marked some package to be held back for
example and that marking can (should?) be removed.


The problem in showing something different for Essential:yes (derived)
and Protected:yes (derived) essential packages is that the difference
between the two is marginal from apts POV: Essential:yes has to work in
unpacked state, but that is a dpkg-level thing to worry about and hardly
a real concern for the general public. Just like the reduced install
order requirements in general.

Okay, things don't need to depend on Essential:yes packages if they use
them, but that tends to be the case for Protected:yes as well as not
that much really "uses" an init system for example. Other distros slap
Protected:yes on high-level meta packages like 'gnome'. Nobody depends
on that either.

All the two really do in terms of apt (front ends – the message is apt
specific, but the fields aren't so it would be kinda nice if terminology
could be reused by other front ends if they so choose) is making it
a pain to remove them, but being too upfront about that has its problems
as well as it naturally leads to the question "why?" which apt preempts
with the ultimate hammer: Its essential for the system as the individual
reason for each package might even be distro-specific. Users usually
don't question that.

It's a lie. Heck, it might even be deception. But the truth hurts more:
"Heh, you are a great user, you really are, but you know, no offense,
but I am a computer program on a device you (think you) own and should
probably be able to do whatever you want to do with it, but there are
other people who are not you who think you might be an idiot, so they
said I should not let you do that and I trust them because they… well,
how do I put it so that even you understand it… they live in the cloud,
yes? So, anyway, are you like, absolutely positively sure about this?"


> > "It 

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting David Kalnischkies (2022-12-18 17:18:28)
> On Fri, Dec 16, 2022 at 03:38:17PM +0100, Santiago Vila wrote:
> > Then there is "e2fsprogs", which apt seems to treat as if it were
> > an essential package:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826587
> 
> As Julian explained, it is considered "essential" because the maintainer
> said so. If you don't think e2fsprogs should be considered "essential"
> for a system it is installed on please talk to the maintainer.
> 
> Sure, the package is not (anymore) really "Essential:yes", but 'apt'
> isn't either and will print the same message anyhow. I don't think it
> would be a net-benefit for a user to invent words for different types of
> essentialness in apt because in the end you either know what you are
> doing while removing a somewhat essential package and continue or you don't
> know what you are doing and (hopefully) get the hell out.

would it be so difficult to cater to both kind of users? For those who do not
know the terminology, using the word "essential" is probably fine. But for
those who do it's confusing. Why can apt not say something like:

WARNING: The following packages will be removed. Apt considers them essential
because they are marked as Priority:required. This should NOT be done unless
you know exactly what you are doing!

> > This sort-of breaks sbuild when using an ordinary chroot (not overlayfs),
> > because after building a package needing e2fsprogs, it may not be removed
> > and it's kept in the chroot.
> 
> "It may". Well, certainly apt won't autoremove it. Like a lot of other
> packages it wont even through they aren't essential or protected or
> whatever… ("just" prio:required is enough for example). They are not not
> irremovable through. It might just be a little harder to remove them
> than to install them. Heck, some people believe its far easier to start
> vim than to exit it.

Note also, that you really shouldn't be using sbuild with an ordinary chroot,
that is without overlayfs or without the chroot being a tarball unpacked to a
tmpfs. The system will not only be different from before after removal of
packages at the end, if you add --allow-remove-essential to the removal
commandline in sbuild, the chroot might even be completely unusable. What is
the use-case of using sbuild with non-emphimeral chroots?

So personally, I'll not invest my own time in making sbuild better at package
removal at the end of the build process.

> > build packages. Here we would need some interface like
> > SUDO_FORCE_REMOVE=yes, or maybe just stop doing anything at all with the
> > Important:yes flag.
> 
> Ironically, one of the selling points for Protected:yes (that is how the
> field ended up named which dpkg is supporting by now) was that it allows
> to shrink the essential set (e2fsprogs even being an example) as there
> is a non-empty set of people who believe users do incredibly dumb things
> if you give them the option to.
> 
> I mean, we got practically bullied into replacing the "Do as I say!"
> prompt with a semi-hidden cmndline flag (--allow-remove-essential)
> because some wannabe YT star yolo'ed the prompt ending in misery and
> somehow that was framed as our fault by everyone as we didn't show the
> appropriate meme-gif (rendered with caca) to make them understand
> without reading [sorry, not sorry. I am not even exaggerating that much].

After this operation, 195 MS disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'

https://youtu.be/0506yDSgU7M?t=637

"I have to type 'Yes, do as I say!' in order to install it."

Sigh...

> Due to that, you are now presented with:
> | E: Removing essential system-critical packages is not permitted. This might 
> break the system.
> 
> See? "essential" again and even "system-critical" at that.
> It is all a lie of course. Nobody really needs an init system, much less
> some silly metapackage for it, as long as there is /bin/sh and a keyboard.
> I should make a video about it to – essentially – become famous & rich…

Dammit, and now you wrote that one can use in a public forum that it's possible
to add --allow-remove-essential! Think of the children!

> Btw, apt also has behaviour specifically for sbuild: 'apt-cache show
> mail-transport-agent' has a zero exitcode even through that makes no
> sense at all apart from not making (some?) sbuild versions explode.
> You are welcome. I hate it.

E... lets change that. What's the problem in sbuild?

Thanks!

cheers, josch

Re: Please, minimize your build chroots

[David Kalnischkies]

On Fri, Dec 16, 2022 at 03:38:17PM +0100, Santiago Vila wrote:
> Then there is "e2fsprogs", which apt seems to treat as if it were
> an essential package:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826587

As Julian explained, it is considered "essential" because the maintainer
said so. If you don't think e2fsprogs should be considered "essential"
for a system it is installed on please talk to the maintainer.

Sure, the package is not (anymore) really "Essential:yes", but 'apt'
isn't either and will print the same message anyhow. I don't think it
would be a net-benefit for a user to invent words for different types of
essentialness in apt because in the end you either know what you are
doing while removing a somewhat essential package and continue or
you don't know what you are doing and (hopefully) get the hell out.


> This sort-of breaks sbuild when using an ordinary chroot (not overlayfs),
> because after building a package needing e2fsprogs, it may not be removed
> and it's kept in the chroot.

"It may". Well, certainly apt won't autoremove it. Like a lot of other
packages it wont even through they aren't essential or protected or
whatever… ("just" prio:required is enough for example). They are not not
irremovable through. It might just be a little harder to remove them
than to install them. Heck, some people believe its far easier to start
vim than to exit it.


> I think apt authors did not think that apt is used by sbuild to

I think (the few) apt authors deal with way too many users with way too
many (sometimes mutually exclusive) ideas of how it should behave:


> build packages. Here we would need some interface like SUDO_FORCE_REMOVE=yes,
> or maybe just stop doing anything at all with the Important:yes flag.

Ironically, one of the selling points for Protected:yes (that is how the
field ended up named which dpkg is supporting by now) was that it allows
to shrink the essential set (e2fsprogs even being an example) as there
is a non-empty set of people who believe users do incredibly dumb things
if you give them the option to.

I mean, we got practically bullied into replacing the "Do as I say!"
prompt with a semi-hidden cmndline flag (--allow-remove-essential)
because some wannabe YT star yolo'ed the prompt ending in misery and
somehow that was framed as our fault by everyone as we didn't show the
appropriate meme-gif (rendered with caca) to make them understand
without reading [sorry, not sorry. I am not even exaggerating that much].

Due to that, you are now presented with:
| E: Removing essential system-critical packages is not permitted. This might 
break the system.

See? "essential" again and even "system-critical" at that.
It is all a lie of course. Nobody really needs an init system, much less
some silly metapackage for it, as long as there is /bin/sh and a keyboard.
I should make a video about it to – essentially – become famous & rich…


Btw, apt also has behaviour specifically for sbuild: 'apt-cache show
mail-transport-agent' has a zero exitcode even through that makes no
sense at all apart from not making (some?) sbuild versions explode.
You are welcome. I hate it.


So, long story short, apt features and behaviour are very seldom done
because we are bored and had nothing better to do. It is far more common
that it was heavily requested to be that way for $REASONS. Sometimes its
even the same $REASONS you have for disliking it. Users are the worst,
I said it here first. But no problem, there usually is an option to
change anything in apt. If not, we can usually add it.

Just don't assume that the behaviour you prefer will be the default.
We have a strong tendency to make everyone unhappy.
(I should know, I never get what I want.)


Best regards

David Kalnischkies

P.S.: You thought we are surprised by sbuild using apt? Sorry, but you
are up against ISO building needing 'apt-cache depends' output previously
unknown even to the CD team itself (https://bugs.debian.org/218995#54)
(yes, it is a decade old. It's still my favorite). Try harder.


signature.asc
Description: PGP signature

Re: Please, minimize your build chroots

[Santiago Vila]

El 16/12/22 a las 18:55, Andreas Metzler escribió:

I am wondering if there is point to this or whether policy should be
changed? Is there some value in investing work in having packages
buildable without Prioriry required packages?


I'd like to apologize to Andreas for my previous answer, as I believe
there has been a misunderstanding.

There are actually two meanings for "required package". One of them
is "packages having the 'required' value in the priority field of the
control file". The other meaning is the one you quote in policy, i.e.
packages which may make your system become broken when you remove them.

I propose that we remove certain packages from chroots, packages which
currently have the priority required in the control field, because they
are not needed for building.

Whether that requires to modify the definition of required in policy,
I don't know.

I think we definitely need to decouple "the set of required packages" with
"the set of packages needed for building", because they are different
and none of them is a proper subset of the other. For example,
we already don't install a kernel or an init system in a chroot used for
building.

Thanks.

Re: Please, minimize your build chroots

[Santiago Vila]

El 16/12/22 a las 18:55, Andreas Metzler escribió:

I am wondering if there is point to this or whether policy should be
changed? Is there some value in investing work in having packages
buildable without Prioriry required packages?


Please do not misrepresent what I said.

I never proposed such thing ("without priority required packages").

I propose that we follow Policy as closely as we reasonably can,
and Policy says this:

If build-time dependencies are specified, it must be possible to build
the package and produce working binaries on a  system with only essential
and build-essential packages installed and also those required to satisfy
the build-time relationships (including any implied relationships).

This is the policy we should follow to build packages, not the paragraph
you quoted about what disaster may happen if you deinstall a required package
in a general sense.

Certainly uninstalling tzdata does not prevent you from installing it again
using dpkg and apt.

Maybe the paragraph you quoted have to be reworded, yes, but that was not
the purpose of my initial email.

Thanks.

Re: Please, minimize your build chroots

[Guillem Jover]

On Fri, 2022-12-16 at 18:55:42 +0100, Andreas Metzler wrote:
> or apt.
> 
> I am wondering if there is point to this or whether policy should be
> changed? Is there some value in investing work in having packages
> buildable without Prioriry required packages?
> 
> Such installations can only be found as artifacts since there does not
> seem to be a supported way to upgrade them or (un)install packages
> (quoting policy: "Removing a "required" package may cause your system to
> become totally broken and you may not even be able to use "dpkg" to put
> things back, so only do so if you know what you are doing.") Essentialy
> policy is telling us it might work to install b-d, and uninstall
> Prioriry required, however dpkg might break halfway through and it would
> not be a bug.

  

Thanks,
Guillem

Re: Please, minimize your build chroots

[Andreas Metzler]

On 2022-12-16 Santiago Vila  wrote:
> Greetings.

> I'm doing archive-wide rebuilds again.

> I've just filed 21 bugs with subject "Missing build-depends on tzdata"
> in bookworm (as tzdata is not build-essential).

> This is of course not fun for the maintainers, but it's also not fun
> for people doing QA, because those bugs could be caught earlier in the
> chain, but they are not. This is extra work for everybody.

> (Similar bugs are even sliding into stable releases, I plan to report
> a few of them against bullseye after 11.6 this Saturday, as bullseye
> is the currently supported stable release).

> Because people accept the default by debootrap "as is", chroots used
> to build packages include packages which are neither essential nor
> build-essential, like tzdata, mount or e2fsprogs.
[...]

or apt.

I am wondering if there is point to this or whether policy should be
changed? Is there some value in investing work in having packages
buildable without Prioriry required packages?

Such installations can only be found as artifacts since there does not
seem to be a supported way to upgrade them or (un)install packages
(quoting policy: "Removing a "required" package may cause your system to
become totally broken and you may not even be able to use "dpkg" to put
things back, so only do so if you know what you are doing.") Essentialy
policy is telling us it might work to install b-d, and uninstall
Prioriry required, however dpkg might break halfway through and it would
not be a bug.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Please, minimize your build chroots

[Santiago Vila]

El 16/12/22 a las 4:39, Johannes Schauer Marin Rodrigues escribió:

I think truly fixing this problem is a bit more tricky because most build tools
like the sbuild schroot backend require apt being installed in the chroot. As
of today, the sbuild schroot backend is unable to function with a chroot that
doesn't contain apt. I don't think it's conceptually possible to fix the
schroot backend to work with chroots without apt because schroot (for good
reason) doesn't give you root anywhere but inside the chroot.


You are right. My email should not be interpreted as "let's fix
this once and forever" but more than "let's see how much of this
we can fix easily, and how much of it would need more work".

In my experience, both "tzdata" and "mount" may be removed without trouble.

Then there is "e2fsprogs", which apt seems to treat as if it were
an essential package:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826587

This sort-of breaks sbuild when using an ordinary chroot (not overlayfs),
because after building a package needing e2fsprogs, it may not be removed
and it's kept in the chroot.

I think apt authors did not think that apt is used by sbuild to
build packages. Here we would need some interface like SUDO_FORCE_REMOVE=yes,
or maybe just stop doing anything at all with the Important:yes flag.

Thanks.

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Santiago Vila (2022-12-16 02:15:13)
> I've just filed 21 bugs with subject "Missing build-depends on tzdata"
> in bookworm (as tzdata is not build-essential).

thank you for that!

> I can think of two solutions for this:
> 
> A) Either debootstrap, when using buildd profile, installs only
> essential and build-essential packages.
> 
> or
> 
> B) debootstrap keeps installing all required packages in the buildd profile,
> no matter if they are really build-essential or not, but those who
> are not build-essential should have their priority downgraded to "important"
> by ftpmaster.
> 
> This problem was already reported here:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837060

Thank you for finding my bug from back then. :)

> and apparently we have not decided yet if we are going to do A or B,
> or maybe some other thing. I don't really care how it's fixed, but I
> believe it's about time that we sync practice with policy, because
> currently we are doing this in a quite suboptimal way.

I think truly fixing this problem is a bit more tricky because most build tools
like the sbuild schroot backend require apt being installed in the chroot. As
of today, the sbuild schroot backend is unable to function with a chroot that
doesn't contain apt. I don't think it's conceptually possible to fix the
schroot backend to work with chroots without apt because schroot (for good
reason) doesn't give you root anywhere but inside the chroot.

To be able to install build dependencies in chroots without apt, the sbuild
unshare backend could be used. Also helmut's mdbp can easily be changed to
build packages in chroots without apt when using its mmdebstrap backend.

But of course changing debootstrap to only install essential, build-essential
and apt (but not prio:required) would already fix a large part of the problem.

> In the meantime: If you want to help QA and have any kind of chroot used
> for any kind of QA (say, ci.debian.org or reproducible-builds, or even
> your personal chroots), please try to minimize the packages there,
> do not merely accept debootstrap default behaviour.

You can use mmdebstrap to create such a chroot:

$ mmdebstrap --variant=apt --include=build-essential unstable chroot.tar

This will also install apt because most build tools need it. The mmdebstrap
package mimics debootstrap behaviour. As soon as debootstrap --variant=buildd
is fixed, I'll let mmdebstrap do the same thing.

Thanks!

cheers, josch