Hi Sam,
On Wed, Jun 28, 2023 at 02:55:28PM -0600, Sam Hartman wrote:
> I have read the mail, not the full updated DEP, so I cannot yet ack
> this.
Hmm. Do you intend to do that? If you are short on time, I think the
problem section is more important than the mitigation section.
> Helmut> When we get into mitigations, consensus is hard to come
> Helmut> by. My impression is that we have roughly two camps. One is
> Helmut> in favour of major changes to dpkg and the other is not.
>
> I think you might get a more clear consensus if you phrase that in terms
> of whether people are willing to wait for major changes in dpkg.
> If the dpkg maintainer were to merge aliasing support, I haven't seen
> anyone objecting strong enough to try and override that maintainer
> action for example.
I think this is a misrepresentation. There is no readily mergable patch
for aliasing support. The most complete one is the tree from Simon
Richter and he considers that work incomplete still. At this time, it's
no a question of merging or not really.
>From my point of view, the question really is whether we want to
permanently extend dpkg's features in a way that we only really see
useful for one transition while still having to maintain it for
backwards compatibility.
In any case, your reply demonstrates why it is so difficult to get to a
good consensus item on this.
> Ah, this was really helpful, because it allowed me to understand that at
> least you and I haven't even been thinking about the problem space the
> same.
Cool. I've also learned something about how debootstrap works
differently than I originally thought from Ansgar. Both of these are
indications that this discussion is constructive.
> Let's explore whether debootstrapping tools need to have release
> knowledge at all.
Yes, that's an important question. I implied it in #3a vs #3b/#3c.
> Support for creating a merged /usr installation was first added in
> debootstrap 1.0.83 in September of 2016.
> It was enabled by default in 1.0.85 in October of 2016.
> So, everything since stretch has supported merged /usr.
In essence, you are proposing to permanently make the setup of symbolic
links part of the bootstrap protocol. Given that installing usrmerge has
even worked on older releases, we could simply treat them as
retroactively merged.
> 1) Do we actually still need to support boostrapping things older than
> stretch at all using modern bootstrap tools?
A customer of mine (the one that makes me work on /usr-merge :) still
works with jessie and developing updates for jessie still tends to
happen on unstable systems. So I argue that around 10 DDs probably still
have a need for doing this.
> If you're really installing something that old, can't you use a
> container image to use an older bootstrapping tool?
So after having answered the question of whether some people still need
this, I think installing an older debootstrap would be far more annoying
than the workaround you propose:
> 2) Even if we do, I think it's okay to say that you need to specify
> --no-merged-usr when installing something older than stretch, just as
> you need to specify that if you want a buster, stretch, or bullseye
> version that is not merged /usr.
While that workaround seems simple enough, plugging an option into
debootstrap can be difficult at times. I have also been working on
setting up autopkgtests and learned that debci wraps the debootstrap
call in quite some layers. Stuffing --no-merged-usr through those layers
is a non-trivial effort. I know this, because I recently sent multiple
MRs for debci to get --keyring through those layers.
Quite obviously, I am in a really special situation of actually working
with jessie and working on autopkgtests for jessie. No sane person would
do that and I cannot expect that Debian goes through extra hoops just
for me. That said, it's not like your strategy is without cost. It just
happens elsewhere.
> So my proposal is to modify the bootstrap protocol, and unless an
> administrator specifically requests a non merged /usr system, then merge
> /usr.
This sounds as if we'd just have to patch mmdebstrap and cdebootstrap
(and remove multistrap) while keeping debootstrap the way it is. I think
that we will have to touch debootstrap in any case. If you specify
--variant=buildd, you get an unmerged chroot even when you do it on
trixie or unstable. This already surprised some users and probably needs
changing. So debootstrap is the thing that definitely needs an update
(even in stable) while the others may not need an update if we end up
picking #3c.
> My initial analysis is that you're making this more complicated than it
> needs to be.
I disagree with my personal and Freexian collaborator hats, but I see
how you get here and that my use cases are all but representative. I
find it plausible that we'd get a majority for the opinion that having
to specify --no-merged-usr for old releases would be an acceptable
workaround.
> My assumption here is also that the change needed to the bootstrap
> protocol is the same as the change that debootstrap has been doing for
> years.
Your assumption evidently does not hold (due to how it handles
--variant=buildd). If we end up shipping symlinks in base-files (which
is one of the proposed measures to prevent their deletion during
upgrades), we need more changes to debootstrap (in stable) still. We
already know that what we have been doing in debootstrap for years will
not work for trixie as is.
This is actually one of the reasons why we cannot just move all the
files. Doing so would kill our buildds, which are still unmerged and get
debootstrapped as unmerged once a week. Uploading debootstrap is one of
the first tasks once we know where we want to go.
> If there are additional changes that would be harmful when bootstrapping
> stretch, buster, bullseye, or bookworm,
> I agree it gets more complicated.
> I'd like to understand those changes though.
I don't think there are. Johannes explained in his other mail that
complications arise when going to jessie or older.
The thing I see about #3c is that more and more I see consensus on
moving forward by moving files to canonical locations (in this thread
and earlier). In order to keep using the existing bootstrap protocol, we
only need two more ingredients:
* base-files ships the symbolic links
* We upload ~10 essential source packages at roughly the same time to
keep the time of not being bootstrappable small.
And really the main benefit I see with doing this is that it allows us
to delete complexity and code. Once finished, we delete the usrmerge
binary package. At a later time, we may also delete the merging code
from debootstrap. That long-term vision of removing complexity down the
road is not shared as much with either #3a or #3b.
I see how I am biased about this, which makes it even more difficult to
establish consensus around the bootstrap topic.
Can we maybe turn the question upside down and phrase it in terms of how
to place the aliasing symlinks?
Option #3d:
A binary package (such as base-files) shall contain the aliasing
symlinks in its data.tar.
Option #3e:
No package shall contain the aliasing symbolic links in a data.tar
and dpkg shall not track them in its filesystem database.
I'll skip the implications between #3[abc] and #3[de] to not spoil the
consensus effort.
Helmut
