Bug#1020923: tech-ctte: please clarify if atomic updates are required

[Gunnar Wolf]

Hi Ansgar,

Ansgar dijo [Wed, Sep 28, 2022 at 10:18:26PM +0200]:
> Any package relevant for successful boot. Any upgrade.
> 
> As far as I can tell, the submitter requires some guarantees
> significantly stronger than what Debian requires for essential
> packages.
> 
> In particular, boot-relevant packages are demanded to work in
> unconfigured state, with not fully satisfied dependencies (possibly not
> even unpackaged?), in partly unpackaged states, after maintainer script
> errors, and all of that in combination with system crashes that might
> result in partly written data to filesystems. And possibly in other
> interesting system states too.

Humm, as the maintainer for raspi-firmware, this definitively
addresses an area where I'm responsible. So this is naturally
interesting for me even outside my TC role.

There is a point I somewhat agree with Bug#1020920's submitter:
Packages modifying the packages involved in system boot need to be
extra careful to reduce the window of vulnerability for an unbootable
system as much as possible.

However, no matter how careful we are, I do not think it's expectable
that we can guarantee the atomic interaction mode Zack W. suggests —
There is no syscall matching "rename and create a symlink". And even
if we had one, it would most probably still become two separate
filesystem operations in the end. Of course, the supported
filesystems' code could be modified so that said operation sequence
could be added to the journal beforehand, so they can be effectively
considered as atomic, but...

That's quite an unrealistic expectation. We cannot expect to implement
actions not expressable in the set of primitives Linux exposes to
us. We cannot expect a (quite invasive I'd expect) kernel patch to be
applied just because we want to run usrmerge.

> > (2) The TC is a decision-making body of last resort.  The bug you
> >     mention was filed today.  Might this be premature?
> 
> Well, if we close it or don't act on it, people will complain and/or
> demand to remove us from Debian for not acting on it (the latter might
> be limited to people just sitting on their porch).
> 
> The other tech-ctte bug about usrmerge also suggested it would just end
> up here either way.

There is a high chance we might end up getting this bug in the TC,
given the spirits we have seen around merged-/usr. However, I agree
with Sean: This bug is too early to summon the TC.

I know that if I suggest you to bring the issue to d-devel@l.d.o it
will fuel a flamewar, but I see no other proper way to handle _this
particular mail_. Maybe the request could be phrased differently, in a
way it could encompass this bug report (i.e. "ask the TC whether we
might use sloppy techniques when upgrading, considering the risks we
take as acceptable" (of course, I don't mean your job is sloppy, it's
just an example text that will not be accepted if asked )

So... I'm also inclined to ask you to please close this TC bug, as it
is not acceptable for a TC ruling. (Also, how many rulings does it
make sense for the TC to hold on the same tired topic of merged-/usr?)

Greetings,

- Gunnar.


signature.asc
Description: PGP signature

Bug#1020923: tech-ctte: please clarify if atomic updates are required

[Ansgar]

On Wed, 2022-09-28 at 13:05 -0700, Sean Whitton wrote:
> On Wed 28 Sep 2022 at 08:00PM +02, Ansgar wrote:
> > Package: tech-ctte
> > X-Debbugs-Cc: Zack Weinberg 
> > Control: block 1020920 by -1
> > 
> > Hi,
> > 
> > please clarify if atomic updates are mandatory for the Debian system.
> > Or other measures to ensure that system crashes at *any* time do not
> > render a system unbootable.
> > 
> > See also: https://bugs.debian.org/1020920
> 
> (1) Do you mean any package update?  Certain packages?  dist-upgrade?

Any package relevant for successful boot. Any upgrade.

As far as I can tell, the submitter requires some guarantees
significantly stronger than what Debian requires for essential
packages.

In particular, boot-relevant packages are demanded to work in
unconfigured state, with not fully satisfied dependencies (possibly not
even unpackaged?), in partly unpackaged states, after maintainer script
errors, and all of that in combination with system crashes that might
result in partly written data to filesystems. And possibly in other
interesting system states too.

> (2) The TC is a decision-making body of last resort.  The bug you
>     mention was filed today.  Might this be premature?

Well, if we close it or don't act on it, people will complain and/or
demand to remove us from Debian for not acting on it (the latter might
be limited to people just sitting on their porch).

The other tech-ctte bug about usrmerge also suggested it would just end
up here either way.

Ansgar

Bug#1020923: tech-ctte: please clarify if atomic updates are required

[Sean Whitton]

Hello,

On Wed 28 Sep 2022 at 08:00PM +02, Ansgar wrote:

> Package: tech-ctte
> X-Debbugs-Cc: Zack Weinberg 
> Control: block 1020920 by -1
>
> Hi,
>
> please clarify if atomic updates are mandatory for the Debian system.
> Or other measures to ensure that system crashes at *any* time do not
> render a system unbootable.
>
> See also: https://bugs.debian.org/1020920

(1) Do you mean any package update?  Certain packages?  dist-upgrade?

(2) The TC is a decision-making body of last resort.  The bug you
mention was filed today.  Might this be premature?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Russ Allbery]

Zack Weinberg  writes:

> That does not clearly say that a *pair* of packages, where one installs
> files in /path and the other one installs files in /usr/path, is not 
> allowed.  I'm guessing it was the *intent*, but the example, at least,
> makes it sound like it's only talking about within a single package.

Thanks, good catch.  We've been dealing with this elsewhere as the other
replies pointed out, but we should update the wording in Policy to make
this explicit.

I'll open a separate bug for that.

-- 
Russ Allbery (r...@debian.org)  

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On 2022-09-28 2:16 PM, Russ Allbery wrote:

"Zack Weinberg"  writes:


1. Is there already a rule (or multiple rules) somewhere that forbids
the existence of pairs of packages where one ships /X/Y and the
other ships /usr/X/Y, where X is a directory on non-merged-/usr
systems and a symlink on merged-/usr systems, and Y is any name?


Policy 10.1:

 To support merged-/usr systems, packages must not install files in
 both /path and /usr/path. For example, a package must not install both
 /bin/example and /usr/bin/example.


That does not clearly say that a *pair* of packages, where one installs 
files in /path and the other one installs files in /usr/path, is not 
allowed.  I'm guessing it was the *intent*, but the example, at least, 
makes it sound like it's only talking about within a single package. 
Suggest maybe the editorial addition of


For another example, if one package installs /bin/example, then
no package which is coinstallable with that package may install
/usr/bin/example.

(the "coinstallable with" clause means to clarify that the existing pair 
discussed upthread is fine because there's a package-level Conflicts)


zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Russ Allbery]

"Zack Weinberg"  writes:

> 1. Is there already a rule (or multiple rules) somewhere that forbids
>the existence of pairs of packages where one ships /X/Y and the
>other ships /usr/X/Y, where X is a directory on non-merged-/usr
>systems and a symlink on merged-/usr systems, and Y is any name?

Policy 10.1:

To support merged-/usr systems, packages must not install files in
both /path and /usr/path. For example, a package must not install both
/bin/example and /usr/bin/example.

If a file is moved between /path and /usr/path in revisions of a
Debian package, and a compatibility symlink at the old path is needed,
the symlink must be managed in a way that will not break when /path
and /usr/path are the same underlying directory due to symlinks or
other mechanisms.

We've had that rule in Policy since May of 2017.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Svante Signell]

Hi,

I would really appreciate if you quote your reply properly:
It was not Andreas Metzler who sent the below:

> On Tue, 2022-09-27 at 18:25 +0200, Andreas Metzler wrote:
> > On 2022-09-27 Zack Weinberg  wrote:
> > [...]

On Wed, 2022-09-28 at 11:55 +0100, Luca Boccassi wrote:

> > 
> > > This number is of usrmerged systems is so large that we cannot mark
> > > them as unsupported ("Please reinstall"). Whether this percentage
> > > is 25% or 90% does not matter.
> > 
> > You can easily revert any system having usrmerge installed with dpkg-
> > fsys-usrunmess. This should be known by all Debian users, by some
> > suitable channel.
> > 
> > And for example the latest init-system-helpers release should add
> > this to the package description (if not reverted). This applies to
> > other present and future packages having usrmerge as a dependency
> > too.
> 
> Please note that this will result in an unsupported system, as per CTTE
> decision. It is important to note this, for the record. So no, that
> will not be added anywhere, package description or otherwise, and in
> fact the last time it was "made known by all Debian users" it caused
> quite a lot of damage, and was forcefully reverted.

Please learn how to properly quote/reply to peoples postings :(

Thanks!

Bug#1020792: marked as done (tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed)

[Debian Bug Tracking System]

Your message dated Wed, 28 Sep 2022 11:03:56 -0700
with message-id <87a66j4bqr@melete.silentflame.com>
and subject line Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until 
dpkg filesystem damage bugs are fixed
has caused the Debian Bug report #1020792,
regarding tech-ctte: Halt merged-/usr transition until dpkg filesystem damage 
bugs are fixed
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)


-- 
1020792: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020792
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: tech-ctte
Severity: normal
X-Debbugs-Cc: z...@owlfolio.org

I formally request that the Technical Committee call a halt to the
merged-/usr transition until such time as all of the bugs in dpkg that
can, on a merged-/usr system, cause damage to the contents of the
filesystem (e.g. packaged files being silently deleted on upgrade)
have been fixed to the satisfaction of the dpkg maintainers.

## Background

It has been known for some time that dpkg has bugs which prevent it
from correctly handling merged-/usr systems.  #858331/#848622 is the
only such bug (that I can find) that has actually been recorded in the
BTS, and it *appears* to be a relatively harmless problem, affecting
only dpkg-query output.

However, Simon Richter  showed in
https://lists.debian.org/debian-devel/2021/08/msg00326.html that the
bad dpkg-query output is only the most obvious symptom of a much more
serious problem, and that, under conditions that will plausibly occur
in the archive after the release of bookworm (assuming all continues
as presently planned) upgrading packages on a merged-/usr system can
cause packaged files to disappear from the filesystem.  The files most
likely to be affected are the files that are currently packaged in
/bin, /sbin, and /lib, including, just to mention a few, /bin/bash,
/bin/systemd, and /lib/$ARCH/libc.so.6.  Thus, the dpkg bugs can
render systems unbootable on upgrade, and should be considered
critical severity.

(N.B. Simon’s message is the only thing I can find that gives
step-by-step instructions to reproduce a problem that could plausibly
cause catastrophic filesystem damage during package upgrades, but the
scenario it describes should _not_ be considered the only problem that
needs to be fixed urgently.  Several other equally dire scenarios
are listed in https://wiki.debian.org/Teams/Dpkg/MergedUsr under the
“merged-/usr-via-aliased-dirs” subheading, albeit only tersely.)

Despite the severity of the dpkg bugs having been known since
_at least_ August of 2021, the people working on the merged-/usr
transition have made almost no effort to fix them.  A patch exists
(https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=858331;filename=file_move_moratorium.patch;msg=46)
that at least partially addresses the problem, but only one desultory
attempt was made to get it merged.

The people working on the merged-/usr transition have repeatedly
claimed that the several years of experience Debian now has with
systems that were _originally installed_ with merged /usr, and the
somewhat longer baseline for Ubuntu doing the same thing, indicates
the dpkg bugs are not a problem in practice.  That claim is incorrect.
The dpkg bugs are *latent* right now because, until the actual release
of bookworm, packages have to continue supporting non-merged systems.
Therefore they cannot move files from /{bin,sbin,lib} to
/usr/{bin,sbin,lib}, which is one of the conditions required to
trigger the most serious known issue (disappearance of packaged files).

I anticipate that if the merged-/usr transition proceeds as planned,
*all systems* that track either testing or unstable will be rendered
unbootable at least once in the bookworm+1 development cycle.  This
level of fallout is not acceptable, even in potentiality.  Since it
seems clear that the people working on the merged-/usr transition have
no intention of getting the bugs fixed, project-level action is
required.

## Specific actions requested

I ask the Committee to require all of the following, partially
reversing the decision taken in #978636, and overriding maintainers
as necessary:

 - The usrmerge and usr-is-merged packages are to be removed from
   testing immediately, and severity:critical (justification: “makes
   unrelated software break” and/or “breaks the entire system”) bugs
   are to be filed to prevent them from re-entering testing.

 - Those bugs may not be closed, nor may their severity be lowered,
   until *the dpkg maintainers agree* that dpkg can now fully support
   merged-/usr systems.

 - No 

Processed: tech-ctte: please clarify if atomic updates are required

[Debian Bug Tracking System]

Processing control commands:

> block 1020920 by -1
Bug #1020920 [usrmerge] usrmerge: if "the system crashes at a really bad time" 
during conversion it may be left unbootable
1020920 was not blocked by any bugs.
1020920 was not blocking any bugs.
Added blocking bug(s) of 1020920: 1020923

-- 
1020920: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020920
1020923: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020923
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#1020923: tech-ctte: please clarify if atomic updates are required

[Ansgar]

Package: tech-ctte
X-Debbugs-Cc: Zack Weinberg 
Control: block 1020920 by -1

Hi,

please clarify if atomic updates are mandatory for the Debian system.
Or other measures to ensure that system crashes at *any* time do not
render a system unbootable.

See also: https://bugs.debian.org/1020920

Thanks,
Ansgar

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Wed, Sep 28, 2022, at 5:08 AM, Svante Signell wrote:
>
> You can easily revert any system having usrmerge installed with dpkg-
> fsys-usrunmess. This should be known by all Debian users, by some
> suitable channel.

Having used it myself a couple of times, I would question "easily".  If all 
goes well, yes, you run it and you reboot and you're done, but if all *doesn't* 
go well you're left having to manually repair a system with important files not 
existing in *either* their unmerged or their merged location, which may require 
booting from recovery media.

I'd say that if Debian were going to widely advertise the availability of 
dpkg-fsys-usrunmess, first it ought to be revamped to ensure that it's fully 
restartable, idempotent, and never, not even transiently, places the system in 
a state where it cannot boot at least as far as single user mode (in systemd 
terms, rescue.target, *not* just emergency.target).

Of course the exact same criticism applies to convert-usrmerge -- skimming its 
code just now, it appears to be idempotent and restartable in principle, but if 
"the system crashes at a really bad time" (to quote its own comments) I think 
it _could_ leave the system in a state where it cannot boot as far as 
rescue.target.  In fact, see #1020920.

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Wed, Sep 28, 2022, at 1:40 PM, Helmut Grohne wrote:
> Hi Zack,
>
> On Wed, Sep 28, 2022 at 12:29:19PM -0400, Zack Weinberg wrote:
>> I thought about this a bunch yesterday evening and I believe I see a
>> concrete scenario that can cause problems but is not covered by the
>> moratorium: Suppose there exist two packages, one of which ships
>> /bin/foo, and the other ships /usr/bin/foo.  On a non-merged system,
>> these packages can coexist.  On a merged system, they have a file
>> conflict that dpkg will not detect.
>
> $ ssh delfin.debian.org sqlite3 
> /srv/dedup.debian.org/database/dedup.sqlite3 '"'"SELECT * FROM content 
> AS ca JOIN content AS cb ON ca.filename = './usr' || 
> substr(cb.filename, 2) WHERE cb.filename NOT LIKE './usr/%';"'"'
> 166447797|96615|./usr/bin/systemctl|201531|221889683|4004|./bin/systemctl|1173136
> 210029518|32365|./usr/sbin/update-service|2917|223295080|31077|./sbin/update-service|4573
> $
>
> So we have systemctl vs systemd and daemontools-run vs runit, both of
> which declare Conflicts.

*nod* I don't think we need to worry about this when there's a declared 
package-level conflict.

> Depends on whether you consider that one-liner above tooling I guess.

Good enough for now, probably -- it would be nice to have some automation 
searching for such overlaps in packages that *don't* declare Conflicts, and 
filing bugs, but I won't call that a blocker.

> Do you see any other issues?

Not at present, but I'm not the person you should be asking!  The only person 
who could possibly say "yeah, the rules in place are sufficient to prevent 
problems post-bookworm until the bugs are actually fixed", with the level of 
confidence we need before proceeding with this transition, is ... the dpkg 
maintainer.

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Helmut Grohne]

Hi Zack,

On Wed, Sep 28, 2022 at 12:29:19PM -0400, Zack Weinberg wrote:
> I thought about this a bunch yesterday evening and I believe I see a
> concrete scenario that can cause problems but is not covered by the
> moratorium: Suppose there exist two packages, one of which ships
> /bin/foo, and the other ships /usr/bin/foo.  On a non-merged system,
> these packages can coexist.  On a merged system, they have a file
> conflict that dpkg will not detect.

$ ssh delfin.debian.org sqlite3 /srv/dedup.debian.org/database/dedup.sqlite3 
'"'"SELECT * FROM content AS ca JOIN content AS cb ON ca.filename = './usr' || 
substr(cb.filename, 2) WHERE cb.filename NOT LIKE './usr/%';"'"'
166447797|96615|./usr/bin/systemctl|201531|221889683|4004|./bin/systemctl|1173136
210029518|32365|./usr/sbin/update-service|2917|223295080|31077|./sbin/update-service|4573
$

So we have systemctl vs systemd and daemontools-run vs runit, both of
which declare Conflicts.

> So questions for you and everyone else reading this:
> 
> 1. Is there already a rule (or multiple rules) somewhere that forbids
>the existence of pairs of packages where one ships /X/Y and the
>other ships /usr/X/Y, where X is a directory on non-merged-/usr
>systems and a symlink on merged-/usr systems, and Y is any name?

I think Marco et al have been filing bugs about these kind of issues and
NMUing the remainders a very long time ago way before debootstrap
defaulted to merged /usr.

> 2. Does Debian already have tooling that can *find* pairs of such
>packages?  (This is a fully independent question from 1.  If
>there's a rule but no automation to enforce it then we don't *know*
>nobody's breaking the rule.  If there is no rule then, before we
>consider adding such a rule, we need to know whether any packages
>exist that break it.)

Depends on whether you consider that one-liner above tooling I guess.

Do you see any other issues?

Helmut

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Ansgar]

On Wed, 2022-09-28 at 12:29 -0400, Zack Weinberg wrote:
> 1. Is there already a rule (or multiple rules) somewhere that forbids
>    the existence of pairs of packages where one ships /X/Y and the
>    other ships /usr/X/Y, where X is a directory on non-merged-/usr
>    systems and a symlink on merged-/usr systems, and Y is any name?

*sigh*

There has been such a rule for many, many years already.

I really feel you lack investigating the issue before filing yet
another ctte bug about it.

Ansgar

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Tue, Sep 27, 2022, at 4:12 PM, Sebastian Ramacher wrote:
> On 2022-09-27 10:26:36 -0400, Zack Weinberg wrote:
>> On Tue, Sep 27, 2022, at 5:15 AM, Sebastian Ramacher wrote:
>> >> I'd like to make sure that the bug submitter has not identified
>> >> something new here.
>> >
>> > I've not seen any new issues appearing since the last round I file bugs.
>>
>> I wasn’t aware that you have been filing bugs related to the
>> transition.  What criteria are you using to find and file those bugs?
>
> I only checked for packages violating the moratorium. Thankfully a check
> for that was implemented by Luca in piuparts. If we have additional
> patterns that could cause issues for upgrades, the check would ideally
> be extended with that information.

I thought about this a bunch yesterday evening and I believe I see a
concrete scenario that can cause problems but is not covered by the
moratorium: Suppose there exist two packages, one of which ships
/bin/foo, and the other ships /usr/bin/foo.  On a non-merged system,
these packages can coexist.  On a merged system, they have a file
conflict that dpkg will not detect.

For practical reasons I doubt that two such packages actually exist --
nobody wants the concrete implementation of "foo" to be selected by
what order the user happened to list /usr/bin and /bin in their PATH,
this is what alternatives are for -- but, I don't see anything in
Policy that *forbids* it outright.  (I could have missed something.)
Also, the undetected file conflict can happen in *any* directory that
is converted to a symlink on a merged system, and  it's at least
vaguely plausible to me that there might be packages shipping
small variations on the same library as /lib/foo.so and /usr/lib/foo.so
(perhaps one has fewer dependencies, to be used during early boot).

So questions for you and everyone else reading this:

1. Is there already a rule (or multiple rules) somewhere that forbids
   the existence of pairs of packages where one ships /X/Y and the
   other ships /usr/X/Y, where X is a directory on non-merged-/usr
   systems and a symlink on merged-/usr systems, and Y is any name?

2. Does Debian already have tooling that can *find* pairs of such
   packages?  (This is a fully independent question from 1.  If
   there's a rule but no automation to enforce it then we don't *know*
   nobody's breaking the rule.  If there is no rule then, before we
   consider adding such a rule, we need to know whether any packages
   exist that break it.)

zw

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Zack Weinberg]

On Tue, Sep 27, 2022, at 12:25 PM, Andreas Metzler wrote:
> On 2022-09-27 Zack Weinberg  wrote:
> [...]
>> What I am asking for is a schedule change: specifically, that the
>> merged /usr transition not be allowed to proceed past the status quo
>> as of two weeks ago (i.e. *before* init-system-helpers added a
>> dependency on usrmerge|usr-is-merged) until after the dpkg bugs are
>> fixed to the satisfaction of the dpkg maintainers.
> [...]
>
> Hello Zack,
>
> Afaiui the only thing the change two weeks caused is an increased
> percentage of usrmerged Debian installations.

It is *conceivable* that a system that has been *converted* to
merged-/usr, in the way usrmerge does it, is different from a system
that was originally installed with merged /usr, in a way that matters
to whether the dpkg bugs can be triggered on that system.  I thought I
had come up with just such a scenario yesterday, in fact.  On further
consideration I was wrong -- but that doesn't mean there are no such
scenarios.

However:

> Afaict the problem is unchanged: There is a very large number of
> usrmerged systems (every system installed with bullseye installer or
> newer unless some very specific steps were taken to avoid this) which
> are prone to bugs due to dpkg not having been changed *first*. This
> number is of usrmerged systems is so large that we cannot mark them as
> unsupported ("Please reinstall"). Whether this percentage is 25% or 90%
> does not matter.

I basically agree with this assertion.  For instance, I think it's not
realistic to roll back every such system to an un-merged state, and
then restart the entire transition, this time following the procedure
that was used when /usr/man was moved to /usr/share/man, which is, it
appears to me, what Guillem would ideally like to have happen.

(I will point out, though, that if we had done it that way starting in
2015 or even 2017, *we would be done by now*, since that process takes
exactly three releases to complete.)

If I had had the authority to make the relevant decision at the time,
I probably would have insisted on getting the dpkg bugs fixed before
the installer's default could be changed.  But that ship has sailed.

However, I think there's still value from a process-management
perspective in declaring that non-merged systems may not be converted,
and are to remain supported, until all critical components of the
distribution -- in particular dpkg -- fully support the merged state.
For instance, it means that the proponents of the transition have a
concrete incentive to get *all* of the remaining work done, and not
just the piece that matters to them.

zw

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Luca Boccassi]

On Wed, 2022-09-28 at 09:51 +0200, Helmut Grohne wrote:
> Hi Luca,
> 
> As much as I agree with you on other matters...
> 
> On Tue, Sep 27, 2022 at 09:11:18PM +0100, Luca Boccassi wrote:
> > baseless, patently false statements - I frankly find it quite upsetting
> > to see claimed that "we have refused to fix any bug" as a self-evident
> > fact, when even a cursory look at the distribution packages/bugs
> > trackers in the past couple of months tells a very different story.
> 
> This is not as clear cut as it may seem and we have disagreed on this
> before. In earlier days, we had multiple disagreements about whether
> something actually is a bug. And when it came to fix dpkg-shlibdeps,
> there clearly was a lack of action until Raphael and Guillem handled the
> matter in despair. I agree that the broad accusation is distant from the
> truth.  Recent history has a lot of quick responses and fixes even in
> the face of inappropriate communication styles used by others.  However,
> when the thing to touch is dpkg, I think refusal is the right term.
> Admittedly, you do have reasons to refuse, but that's still refusal. We
> keep these bugs (whose severity we disagree about) and have mitigations
> and workarounds for them in place.
>
> This is a niche aspect of the whole matter. I think it deserves
> correction, but it doesn't change the broad picture that there are
> little news in this report that would change the CTTE evaluation of the
> matter.

Adding mitigations, workarounds and enhancing our test suites to detect
possible issues in my book does count as working on it. It might not be
some people's preferred form of contribution, but on the other hand
there's no universal law of physics demanding that everything needs to
be fixed to the complete satisfaction of any and all passerbys. Also,
none of you are paying my salary ;-)

Could the mentioned issues happen before? Maybe. Can they happen now,
after the work that has been done? No (again to the best of my
knowledge, reproducers that can fly undetected are always welcome).
Ignoring this does not strike me as a good way to start a conversation,
much less to make demands (not referring to you, obviously).

> There also is one more recent disagreement that keeps popping up here
> and there. We used to have a bootstrap sequence that did not require
> auxiliary setup code. Initially, the transition was deployed by usrmerge
> and all was fine from a bootstrap point of view. Now with usr-is-merged,
> this interface is broken. From my point of view, this is a significant
> regression and when I've been discussing this with proponents, the
> reaction could reasonably be described as refusal to recognize the
> existence of a problem. I fear we need to revisit this at some point.
> The transition is also not complete until that auxiliary code is gone.

This will sound like a long-winded rant but please bear with me. In
modern OS design the end goal has to be that vendors ship under /usr
and optionally /etc (see the concept of hermetic /usr and/or /etc).
Everything else happens at the moment a system is instantiated. And
that does not only include directory layout like /var, /root, /home and
so on - which cannot and must not be shipped by packages to make this
work - but it _must_ include setting up security mechanisms such as
encryption and attestation.

Linux is woefully and embarassingly behind Windows and OSX on the
security front these days, and it's a shame because for a long time we
were so far ahead. For starters, we need to move toward transparent and
automated encryption (and much more) by default or we will keep falling
behind, and we can't live forever off the meme that "Linux is more
secure by default" when that is patently not true anymore, because at
some point the zeitgeist will catch up.

The key point I want to put across is that the idea that you can get
from zero to a working, finalized system _exclusively_ by installing
packages is not something that is fit for the modern world of
computing.

You should of course be able to create an atomic, hermetic vendor tree
(/usr) just by installing packages. My hope is that with bookworm+1 we
will be in a place where no packages install anything under /bin, /sbin
and /lib*, and everything has moved, so that we will be in that place
again. How we will get there, well, there's a few options - one thing
at a time.

> In the end, we probably agree to disagree on what it means to have this
> transition finished. I expect that from your point of view, the
> transition is now finished given that init-system-helper forces the
> migration. I very much disagree that we're done. The question now is who
> will do the work to finish it and who will refuse to do that work?

At the cost of sounding like management - depends on your definition of
"finished". My top-priority goal is that from bookworm onward we can
assume and take for granted that all systems are converted and we do
not have to keep supporting both 

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Luca Boccassi]

> On Tue, 2022-09-27 at 18:25 +0200, Andreas Metzler wrote:
> > On 2022-09-27 Zack Weinberg  wrote:
> > [...]
> > > What I am asking for is a schedule change: specifically, that the
> > > merged /usr transition not be allowed to proceed past the status
> > > quo as of two weeks ago (i.e. *before* init-system-helpers added
> a
> > > dependency on usrmerge|usr-is-merged) until after the dpkg bugs
> are
> > > fixed to the satisfaction of the dpkg maintainers.
> > [...]
> > 
> > Hello Zack,
> > 
> > Afaiui the only thing the change two weeks caused is an increased
> > percentage of usrmerged Debian installations.
> > 
> > Afaict the problem is unchanged: There is a very large number of
> > usrmerged systems (every system installed with bullseye installer
> or
> > newer unless some very specific steps were taken to avoid this)
> which
> > are prone to bugs due to dpkg not having been changed *first*. This
> > number is of usrmerged systems is so large that we cannot mark them
> > as unsupported ("Please reinstall"). Whether this percentage is 25%
> > or 90% does not matter.
> 
> You can easily revert any system having usrmerge installed with dpkg-
> fsys-usrunmess. This should be known by all Debian users, by some
> suitable channel.
> 
> And for example the latest init-system-helpers release should add
> this
> to the package description (if not reverted). This applies to other
> present and future packages having usrmerge as a dependency too.

Please note that this will result in an unsupported system, as per CTTE
decision. It is important to note this, for the record. So no, that
will not be added anywhere, package description or otherwise, and in
fact the last time it was "made known by all Debian users" it caused
quite a lot of damage, and was forcefully reverted.

To elaborate, in practice this means that after bookworm, things can
start assuming that all systems are merged-usr, and it also means that
changes of any kind will very likely not be held back on the off-chance
that they might not work on unmerged systems (and no testing will be
required to detect that either). There are already taints in place to
detect unmerged systems.

In other words, one is of course free to do as they wish on their
systems, but they also get to keep the pieces.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Svante Signell]

On Tue, 2022-09-27 at 18:25 +0200, Andreas Metzler wrote:
> On 2022-09-27 Zack Weinberg  wrote:
> [...]
> > What I am asking for is a schedule change: specifically, that the
> > merged /usr transition not be allowed to proceed past the status
> > quo as of two weeks ago (i.e. *before* init-system-helpers added a
> > dependency on usrmerge|usr-is-merged) until after the dpkg bugs are
> > fixed to the satisfaction of the dpkg maintainers.
> [...]
> 
> Hello Zack,
> 
> Afaiui the only thing the change two weeks caused is an increased
> percentage of usrmerged Debian installations.
> 
> Afaict the problem is unchanged: There is a very large number of
> usrmerged systems (every system installed with bullseye installer or
> newer unless some very specific steps were taken to avoid this) which
> are prone to bugs due to dpkg not having been changed *first*. This
> number is of usrmerged systems is so large that we cannot mark them
> as unsupported ("Please reinstall"). Whether this percentage is 25%
> or 90% does not matter.

You can easily revert any system having usrmerge installed with dpkg-
fsys-usrunmess. This should be known by all Debian users, by some
suitable channel.

And for example the latest init-system-helpers release should add this
to the package description (if not reverted). This applies to other
present and future packages having usrmerge as a dependency too.

Thanks!

Re: Bug#1020792: tech-ctte: Halt merged-/usr transition until dpkg filesystem damage bugs are fixed

[Helmut Grohne]

Hi Luca,

As much as I agree with you on other matters...

On Tue, Sep 27, 2022 at 09:11:18PM +0100, Luca Boccassi wrote:
> baseless, patently false statements - I frankly find it quite upsetting
> to see claimed that "we have refused to fix any bug" as a self-evident
> fact, when even a cursory look at the distribution packages/bugs
> trackers in the past couple of months tells a very different story.

This is not as clear cut as it may seem and we have disagreed on this
before. In earlier days, we had multiple disagreements about whether
something actually is a bug. And when it came to fix dpkg-shlibdeps,
there clearly was a lack of action until Raphael and Guillem handled the
matter in despair. I agree that the broad accusation is distant from the
truth.  Recent history has a lot of quick responses and fixes even in
the face of inappropriate communication styles used by others.  However,
when the thing to touch is dpkg, I think refusal is the right term.
Admittedly, you do have reasons to refuse, but that's still refusal. We
keep these bugs (whose severity we disagree about) and have mitigations
and workarounds for them in place.

This is a niche aspect of the whole matter. I think it deserves
correction, but it doesn't change the broad picture that there are
little news in this report that would change the CTTE evaluation of the
matter.

There also is one more recent disagreement that keeps popping up here
and there. We used to have a bootstrap sequence that did not require
auxiliary setup code. Initially, the transition was deployed by usrmerge
and all was fine from a bootstrap point of view. Now with usr-is-merged,
this interface is broken. From my point of view, this is a significant
regression and when I've been discussing this with proponents, the
reaction could reasonably be described as refusal to recognize the
existence of a problem. I fear we need to revisit this at some point.
The transition is also not complete until that auxiliary code is gone.

In the end, we probably agree to disagree on what it means to have this
transition finished. I expect that from your point of view, the
transition is now finished given that init-system-helper forces the
migration. I very much disagree that we're done. The question now is who
will do the work to finish it and who will refuse to do that work?

Helmut