Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

2023-08-13 Thread Holger Levsen
On Fri, Aug 11, 2023 at 10:56:03PM +0200, Helmut Grohne wrote: > > what about cdebootstrap? > cdebootstrap (and mmdebstrap) never implemented a merging step[1] and to > this date rely on the usrmerge package doing it at postinst time. Once > base-files ships the aliasing symlinks, both will

Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

2023-08-11 Thread Helmut Grohne
Hi Holger, On Fri, Aug 11, 2023 at 09:28:51AM +, Holger Levsen wrote: > On Fri, Aug 11, 2023 at 09:38:02AM +0100, Luca Boccassi wrote: > > > This is implemented in > > > https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96 > > what about cdebootstrap? cdebootstrap (and

Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

2023-08-11 Thread Holger Levsen
On Fri, Aug 11, 2023 at 09:38:02AM +0100, Luca Boccassi wrote: > > This is implemented in > > https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96 what about cdebootstrap? -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org

Re: Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

2023-08-11 Thread Luca Boccassi
On Fri, 11 Aug 2023 at 05:52, Helmut Grohne wrote: > > Hi, > > This is picking up on the debootstrap matter and is kinda crucial. > > On Thu, Jul 13, 2023 at 01:31:04AM +0100, Luca Boccassi wrote: > > > After having sorted this out, what part of your safety concerns with 3C > > > do remain? > > >

Request for review of debootstrap change [was: Re: Second take at DEP17 - consensus call on /usr-merge matters]

2023-08-10 Thread Helmut Grohne
Hi, This is picking up on the debootstrap matter and is kinda crucial. On Thu, Jul 13, 2023 at 01:31:04AM +0100, Luca Boccassi wrote: > > After having sorted this out, what part of your safety concerns with 3C > > do remain? > > Nothing, as that stemmed from a misunderstanding of what the >

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-12 Thread Luca Boccassi
On Tue, 11 Jul 2023 at 09:44, Helmut Grohne wrote: > > Hi Luca, > > On Tue, Jul 11, 2023 at 12:27:04AM +0100, Luca Boccassi wrote: > > You have said in the original mail and on the writeup that this option > > requires all the affected packages to be upgraded at the same time, > > and in the

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-11 Thread Helmut Grohne
Hi Luca, On Tue, Jul 11, 2023 at 12:27:04AM +0100, Luca Boccassi wrote: > You have said in the original mail and on the writeup that this option > requires all the affected packages to be upgraded at the same time, > and in the correct order, or things will break. What happens if any of This

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-11 Thread Simon Richter
Hi, On 7/11/23 00:55, Sam Hartman wrote: * The more I look at this, I think the real complexity is not in bootstrapping, but is in the rest of the proposal for canonicalizing paths. I am very uncomfortable overall; it seems complicated enough that we probably will break something

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-10 Thread Luca Boccassi
On Sun, 9 Jul 2023 at 22:07, Helmut Grohne wrote: > > Hi Sam, > > Thanks for trying to wrap your head around the complexity. > > On Sat, Jul 08, 2023 at 07:57:40AM -0600, Sam Hartman wrote: > > So for me, a 3C proposal would have two components: > > > > 1) An explanation of what the archive looks

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-10 Thread Sam Hartman
> "Timo" == Timo Röhling writes: Nod, I was wrong. Wanted to ex plicitly acknowledge that, although I think it is also obvious from other mails.

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-10 Thread Sam Hartman
Hi. I have read both of your messages over the weekend multiple times. I don't think replying point-by-point is going to be all that helpful, although if there are any cases where you'd like me to do that, I will. * I am really impressed with the work you are putting in on all this. You have

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-09 Thread Helmut Grohne
Hi Sam, Thanks for trying to wrap your head around the complexity. On Sat, Jul 08, 2023 at 07:57:40AM -0600, Sam Hartman wrote: > So for me, a 3C proposal would have two components: > > 1) An explanation of what the archive looks like at time of bootstrap > (and changes to any bootstrap

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-09 Thread Luca Boccassi
Hi Helmut, Let me restate once again that I think you are doing a stellar job at tackling this problem, and you have my thanks for it. From what I can see, I think 99% of us are on the same page for 90% of the problem. The remaining part on how to make bootstrapping safe is the last bit. And in

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-08 Thread Sam Hartman
> "Helmut" == Helmut Grohne writes: Helmut> Hi Sam, Helmut> On Fri, Jul 07, 2023 at 08:50:49AM -0600, Sam Hartman wrote: >> >> TL;DR: It looks like if we do not want to encode merged /usr into >> the bootstrap protocol, we must keep some aliases and cannot move >>

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-08 Thread Timo Röhling
Hi Gioele, * Gioele Barabucci [2023-07-08 10:53]: Even the most convoluted and lock-stepped procedure can surely be carried out over a single day in an all-hands-on-deck effort. Especially if the files of non-critical packages are moved before the flag day. I agree. Maybe my wording was

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-08 Thread Gioele Barabucci
On 08/07/23 09:44, Timo Röhling wrote: Alternatively, we could move all files except for the few critical ones (/bin/sh, dynamic loader) Allow me to add some hard data to this discussion. In essential (proper) there are 153 files that need to be moved, distributed across 15 packages (+

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-08 Thread Helmut Grohne
Hi Sam, On Fri, Jul 07, 2023 at 08:50:49AM -0600, Sam Hartman wrote: > > TL;DR: > It looks like if we do not want to encode merged /usr into the bootstrap > protocol, we must keep some aliases and cannot move all files in > data.tar. Reading both of your recent mails left me very confused. It

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-08 Thread Timo Röhling
Hi Sam, * Sam Hartman [2023-07-07 08:50]: TL;DR: It looks like if we do not want to encode merged /usr into the bootstrap protocol, we must keep some aliases and cannot move all files in data.tar. I think removing all aliasing is important and so I am firmly in the camp of doing usrmerge in

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-07 Thread Sam Hartman
> "Sam" == Sam Hartman writes: Sam> TL;DR: It looks like if we do not want to encode merged /usr Sam> into the bootstrap protocol, we must keep some aliases and Sam> cannot move all files in data.tar. I think removing all Sam> aliasing is important and so I am firmly in the

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-07 Thread Sam Hartman
TL;DR: It looks like if we do not want to encode merged /usr into the bootstrap protocol, we must keep some aliases and cannot move all files in data.tar. I think removing all aliasing is important and so I am firmly in the camp of doing usrmerge in the bootstrap protocol. > "Helmut" ==

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-07 Thread Holger Levsen
On Fri, Jul 07, 2023 at 09:55:05AM +0200, Helmut Grohne wrote: > Thus far, my impression was that temporarily (<1week, preferably <1day) > breaking the ability to debootstrap was an acceptable risk and is > something we experience every now and then anyway (with adduser most > recently).

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-07 Thread Simon Richter
Hi, On 7/7/23 16:55, Helmut Grohne wrote: If we have a consensus we're unwilling to wait for a patch, it doesn't matter whether that's because: While you try to remove the reasoning for this point of view from the consensus, the "willing to wait" language implies a reason of "too slow"

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-07 Thread Helmut Grohne
Hi Sam, On Thu, Jul 06, 2023 at 01:51:04PM -0600, Sam Hartman wrote: > BUT I don't think it matters. > If we have a consensus we're unwilling to wait for a patch, it doesn't > matter whether that's because: Indeed, this is not how I looked at it. It also is a view that I don't subscribe to,

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-06 Thread Sam Hartman
> "Russ" == Russ Allbery writes: Russ> Helmut Grohne writes: Russ> What Sam is trying to Russ> say, I think, is that a different phrasing offers a way to Russ> avoid that discussion and agree to disagree on the best Russ> architecture in the abstract by pointing out an

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-07-02 Thread Helmut Grohne
Hi Simon, On Fri, Jun 30, 2023 at 08:06:15PM +0900, Simon Richter wrote: > I think "backports" are missing as a user story. I fully agree. What a serious omission. As a first step, I have updated DEP17 to indicate which mitigations happen to work when being backported. For instance, changing

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-30 Thread Simon Richter
Hi Helmut, On 6/29/23 04:37, Helmut Grohne wrote: Consensus proposal #1: This updated DEP17 is a complete representation of the known and relevant problems and known mitigations under discussion at the time of this writing. Do you miss a related problem important to you?

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-30 Thread Luca Boccassi
On Fri, 30 Jun 2023 at 06:31, Simon Richter wrote: > > Hi, > > On 6/30/23 02:32, Helmut Grohne wrote: > > >> If the dpkg maintainer were to merge aliasing support, I haven't seen > >> anyone objecting strong enough to try and override that maintainer > >> action for example. > > Correct. The

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Simon Richter
Hi, On 6/30/23 02:32, Helmut Grohne wrote: If the dpkg maintainer were to merge aliasing support, I haven't seen anyone objecting strong enough to try and override that maintainer action for example. Correct. The proponents of the "work around dpkg" approach quite often insinuate that it

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Josh Triplett
Russ Allbery wrote: > This is more of a high-level design intuition that stems from some basic > architectural principles, such as "dpkg should be the authority for what > Debian installs on the file system so that it can ensure global > consistency." > > But to give you a concrete answer, here

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Russ Allbery
Helmut Grohne writes: > To me, this leaves more question marks than earlier. What applications > of aliasing do you envision that would benefit here? Anything concrete? So, I do want to stay focused on Sam's point, which is that we should avoid this specific argument by noting that we're not

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Luca Boccassi
On Thu, 29 Jun 2023 at 13:34, Helmut Grohne wrote: > > Hi Luca, > > On Thu, Jun 29, 2023 at 12:49:16PM +0100, Luca Boccassi wrote: > > Essentially, this boils down to risks vs benefits. The risk of going > > 3c is that every single Debian installation in existence breaks in > > some interesting

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Helmut Grohne
Hi Russ, On Thu, Jun 29, 2023 at 11:51:57AM -0700, Russ Allbery wrote: > I think I fall into the category that Sam is thinking of. I don't agree > that aliasing support in dpkg is useful only for this transition. I think > there are other cases where two directories could end up being aliases

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Luca Boccassi
On Thu, 29 Jun 2023 at 20:39, Simon McVittie wrote: > > On Thu, 29 Jun 2023 at 19:32:15 +0200, Helmut Grohne wrote: > > I think > > that we will have to touch debootstrap in any case. If you specify > > --variant=buildd, you get an unmerged chroot even when you do it on > > trixie or unstable. >

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Simon McVittie
On Thu, 29 Jun 2023 at 19:32:15 +0200, Helmut Grohne wrote: > I think > that we will have to touch debootstrap in any case. If you specify > --variant=buildd, you get an unmerged chroot even when you do it on > trixie or unstable. ... > our buildds, which are still unmerged and get >

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Russ Allbery
Helmut Grohne writes: > On Wed, Jun 28, 2023 at 02:55:28PM -0600, Sam Hartman wrote: >> I think you might get a more clear consensus if you phrase that in >> terms of whether people are willing to wait for major changes in dpkg. >> If the dpkg maintainer were to merge aliasing support, I haven't

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Helmut Grohne
Hi Sam, On Wed, Jun 28, 2023 at 02:55:28PM -0600, Sam Hartman wrote: > I have read the mail, not the full updated DEP, so I cannot yet ack > this. Hmm. Do you intend to do that? If you are short on time, I think the problem section is more important than the mitigation section. > Helmut>

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Andreas Metzler
On 2023-06-28 Helmut Grohne wrote: > The category of generic changes includes > imposing an ordering on initial unpacks (e.g. base-files first). Hello, I have not dug deeply into this but in the back of my mind a voice is vaguely remebering that we already had multiple times wished we had this

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Wookey
On 2023-06-29 12:53 +0200, Johannes Schauer Marin Rodrigues wrote: > Choosing #3c gets us more than just a simple and clean design. Encoding the > information of how a chroot should look like in the packages instead of the > bootstrapping tool allows creating chroots for Debian unstable all the

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Helmut Grohne
Hi Luca, On Thu, Jun 29, 2023 at 12:49:16PM +0100, Luca Boccassi wrote: > Essentially, this boils down to risks vs benefits. The risk of going > 3c is that every single Debian installation in existence breaks in > some interesting ways, as fixing the bootstrapping corner case is > delegated to

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Timo Röhling
Hi Luca, * Luca Boccassi [2023-06-29 12:49]: The sole benefit is that one of the two bootstrapping tools in widespread use keeps its internal code a bit 'cleaner' from the point of view of some technically unnecessary and self-imposed design constraints (yes there's 2 more tools as pointed out

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Luca Boccassi
On Thu, 29 Jun 2023 at 11:53, Johannes Schauer Marin Rodrigues wrote: > > Hi, > > mmdebstrap author here. This is the other bootstrapping tool which is > currently > sitting at ~17% of the popcon value of debootstrap. > > Quoting Helmut Grohne (2023-06-28 21:37:44) > > Once that is settled, the

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-29 Thread Johannes Schauer Marin Rodrigues
Hi, mmdebstrap author here. This is the other bootstrapping tool which is currently sitting at ~17% of the popcon value of debootstrap. Quoting Helmut Grohne (2023-06-28 21:37:44) > Once that is settled, the next big question is how to handle bootstrapping. > We had a number of people arguing in

Re: amount of bootstrapping tools in Debian (Re: Second take at DEP17 - consensus call on /usr-merge matters)

2023-06-28 Thread Holger Levsen
On Wed, Jun 28, 2023 at 08:59:18PM +, Holger Levsen wrote: > however it's author also said on #-devel just now: [...] < josch> | h01ger: i'm not multistraps author though (josch AFAICS is the last maintainer of it, maintaining it from 2016 to 2018.) my point is: it's more than two tools

Re: amount of bootstrapping tools in Debian (Re: Second take at DEP17 - consensus call on /usr-merge matters)

2023-06-28 Thread Holger Levsen
On Wed, Jun 28, 2023 at 08:28:28PM +, Holger Levsen wrote: > On Wed, Jun 28, 2023 at 09:15:30PM +0100, Luca Boccassi wrote: > > > to how you see things moving forward? > > Changing the bootstrap tools seems much safer. It is just two tools, > three: debootstrap, mmdebstrap and cdebootstrap.

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-28 Thread Sam Hartman
> "Helmut" == Helmut Grohne writes: I believe I'm up on this discussion to at least comment on the consensus calls you discuss in the mail. Except where noted below, I support your reading of consensus. Helmut> Consensus proposal #1: Helmut> This updated DEP17 is a complete

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-28 Thread Ansgar
Hi, On Wed, 2023-06-28 at 21:37 +0200, Helmut Grohne wrote: > Among these options, the first has a working prototype (debootstrap), > but it is unclear how that extends to use of snapshot.d.o and how to > make it work with debootsnap and debbisect as those tend to use a > suite name rather than a

amount of bootstrapping tools in Debian (Re: Second take at DEP17 - consensus call on /usr-merge matters)

2023-06-28 Thread Holger Levsen
On Wed, Jun 28, 2023 at 09:15:30PM +0100, Luca Boccassi wrote: > > to how you see things moving forward? > Changing the bootstrap tools seems much safer. It is just two tools, three: debootstrap, mmdebstrap and cdebootstrap. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁

Re: Second take at DEP17 - consensus call on /usr-merge matters

2023-06-28 Thread Luca Boccassi
On Wed, 28 Jun 2023 at 20:38, Helmut Grohne wrote: > > Hi, > > thanks for all the valuable feedback on the huge DEP17 thread. As > promised, I looked into condensing that discussion into something > shorter. That shorter thing still has more than 3000 words and is > available as source at After

Second take at DEP17 - consensus call on /usr-merge matters

2023-06-28 Thread Helmut Grohne
Hi, thanks for all the valuable feedback on the huge DEP17 thread. As promised, I looked into condensing that discussion into something shorter. That shorter thing still has more than 3000 words and is available as source at https://salsa.debian.org/dep-team/deps/-/merge_requests/5 and I