Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Dear Luca, On 27/08/2023 03:16, Luca Boccassi wrote: [things] You've already been asked by a couple of people to moderate your tone in this thread. I appreciate there is a lot of frustration around /usr-merge, but your contributions are not helping with that at all. Nor do they help us have a constructive discussion. The DEP-17 work continues whilst this discussion is ongoing; this discussion is not delaying that work. I think it is fair to say that when ctte decided on /usr-merge it was expected that a) the process would be reasonably straightforward and b) dpkg could be taught to understand directory aliasing, so we would not have to be doing things "behind dpkg's back", as it were. I think the need for a releases-long moratorium on moving files and the volume and depth of technical work represented by DEP17 demonstrate that those expectations turned out not to be met. Given that, it seems to me that a) warnings that "it's not that simple & easy" were not FUD and describing them thus is unhelpful b) it's reasonable to at least ask the question "given what we know now, are we still sure this is the correct approach?" Any such consideration must be mindful of the fact that the majority of Debian installs are now /usr-merged, which means that the complexity of unwinding such installs has to be a heavy factor in thinking about alternative approaches. I'm hopeful, but not certain, that the DEP17 work will get us to a coherent state again by foxy (which still feels a long time off); I don't think we should underestimate the work to be done in properly completing /usr-merge. This discussion has, I think, been broadly useful in clarifying some folks' thinking in this area. I would very much like if we could keep it focused on the technical matters. Regards, Matthew
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Hi Matthew, On Sun, 2023-08-27 at 11:30 +0100, Matthew Vernon wrote: > Any such consideration must be mindful of the fact that the majority of > Debian installs are now /usr-merged, which means that the complexity of > unwinding such installs has to be a heavy factor in thinking about > alternative approaches. Yes, I think there are many technical challenges required before Debian would be in a position to adopt the proposed Jackson filesystem layout. If Debian would choose to adopt it at all (an open question). Besides conversion, there is also the telemetry service that seems to be required to safely move to the proposed layout (AFAIK no alternative to this has been proposed yet). I'm not sure if there is already any work done on the path by the proposers? I'm also still missing an overview what the Jackson layout's advantages over the existing filesystem layout (merged-/usr) or the 2000's layout is (split-/usr). As far as I can tell it combines the disadvantages of both with much additional work required to get to it; I don't really see any advantage so far (besides "our tools can't handle anything else", but in practice it seems to work fine, and of course avoiding stuff associated with a certain company which I understand is a goal in itself for some people)... I would appreciate a list of technical and ideological reasons why switching to the Jackson layout is important for Debian. Ansgar
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
Hi Ian,
On Sat, Aug 26, 2023 at 11:24:33AM +0100, Ian Jackson wrote:
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> > On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > > And, the approach being taken very seriously privileges Debian itself,
> > > and those well-staffed derivatives able to do the necessary transition
> > > auditing (albeit, indeed, with tooling from Debian). I am
> > > firmly ideologically opposed to such a tradeoff.
> >
> > I have difficulties disagreeing with that. Getting this done is more
> > important to me though.
>
> I have hoisted this to the start of the mail. I think it is a hugely
> important point.
>
> Debian is not simply a technical project trying to thread its way in a
> complicated world. Debian is an ideological project. At its best,
> Debian is the infrastructrure that enables vast swathes of people to
> massively enhance their own technological autonomy. Many of our most
> controversial decisions served this goal, and stood the test of time.
>
> That's why *I'm* involved in Debian. Our technical choices should
> serve those goals, always.
>
> (To an extent, this divergence in goals may explain why at times our
> comments have been talking slightly past each other.)
I think I understand this and it resonates with me, but there are limits
to that. I don't think that Debian is that technological leader that you
perceive it to be. I hoped that other distributions would adopt the
multiarch directory layout to regain compatibility with Debian and none
did even though there is a clear, technical advantage to doing this.
Debian does not exist in isolation. It is dependent on a lot of
upstreams and in order for that relationship to be healthy, there needs
to be cooperation.
> If you want to think about it on more practical (or even, selfish)
> level, we want Debian to continue to be the preferred choice, when
> someone is choosing an upstream. We didn't get where we are today by
> following bad technical decisions made by others.
In the grand theme of things, the aliasing symlinks may be a suboptimal
technical approach. Please keep in mind though, that this change in
large parts is about people rather than something deeply technical.
After we stopped supporting booting without /usr being mounted in the
initramfs, the split between / and /usr was effectively random and stuff
was moving back and forth every so often and inconsistent between
different distributions or releases. I still remember iptables being an
annoying instance in this regard. So leaving technical aspects aside,
having large parts of the open source community agree on having those
aliasing symlinks already is a significant benefit even if it has
technical downsides.
In order to prefer Debian over something else, we want it to be similar
enough to make switching feasible while making it differ from others
enough to make switching worthwhile. Not having aliasing symlinks hardly
seems like an aspect that makes Debian more suitable to people. I guess
that you disagree with this and that is fine.
> This is indeed a plausible practical reason to do it the aliased way.
> >From my point of view, it amounts to saying "everyone else has made
> this mistake, so to be compatible, we must too".
I wouldn't say it that way, but it comes quite close.
> But I think that seriously underestimates our influence. Debian
> derivatives make up well over half of all Linux installations.
> They're the default basis for most CI images. If we decide this was a
> failed experiment, then indeed there will be some pain for a while,
> but fairly soon people will stop making this assumption.
Quite evidently, we judge this differently. The two of us value the
benefit of the end states differently and the cost to getting to each of
them. Therefore we arrive at different conclusions.
> I don't like the phrase "symlink farm" because it suggests that all,
> most, or even a substantial minority, of files have these symlinks.
> True, at the start, there will be at least a symlink allotment
> but I'm hoping that in the end it'll be a symlink flowerbed.
Let me suggest that this is wishful thinking. It's not only me saying
this, but you can read this from other responses as well. I encourage
you to use codesearch.d.n to see how your flowerbed really is a farm.
Let me give some examples to get you started:
libreoffice -> /bin/python
ghostscript, imagemagick, mesa -> /bin/env
bind9, manpages, net-snmp, qtbase-opensource-src -> /bin/perl
So I see that we either get a symlink farm or we get to include a lot of
Debian-specific patches or we get to argue with a lot of upstreams about
something that may seem entirely pointless to them. In any of these
cases, I consider that a significant cost.
> But pushing ahead won't lead to such a state. As I say, I think
> people will keep introducing new references to files by their
> non-physical names, and we'll keep getting lossage, essentially
> forev
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
TL;DR: I think I understand one of Ian's points. I explain, but do not believe it is compelling as an argument to switch direction. > "Helmut" == Helmut Grohne writes: >> I think "package management" is the wrong term here. It's not >> just our tools and packages that are affected. All forms of >> operating system management are affected: anything that changes >> the software, and not just its configuration. >> >> Affected tooling includes not just our .debs, but also remote >> configuration management systems like Ansible, image construction >> (Dockerfiles), and 3rd-party software installation progrmas >> (including both 3rd-party .debs and 3rd-party script-based >> installation systems). Helmut> I don't follow the reasoning. Much of the tasks you'd carry Helmut> out with (wlog) ansible - even when updating files - will Helmut> continue to work in the aliasing layout. The reason that Helmut> dpkg is more affected is that it has an inventory of files Helmut> and reasons about their ownership in terms of Helmut> packages. That's not how any kind of configuration Helmut> management operates. If you just "make install" something, Helmut> chances are that it'll just work with an aliasing layout Helmut> even when installing with --prefix=/. I continue to argue Helmut> that the problems we are seeing are quite specific to dpkg Helmut> in large parts. I spent some time with Ian's paragraph you quote above trying to understand it, and I think I got somewhere. I considered replying yesterday but decided against, because I think we are already seeing these effects, and have been seeing them long enough that we would have a good feel for how serious the problems are. I do think that configuration management does have enough of an inventory of files and reasoning about structure that some of these problems could arise. I agree that configuration management does not typically reason in terms of packages. But mechanisms like * ADD/COPY in Containerfile * The rsync module in ansible * The file module in Ansible * various inventories related to modification detection in configuration management do reason about the filesystem. I don't know what would happen if I did hosts: lots remote_user: root name: Does this shoot me in the foot tasks: - rsync: src=install_root dest=/ where install_root has a bin directory containing a couple of scripts. I don't know if rsync will replace a symlink with a directory, or will just write through the symlink in that situation. If rsync happens to write through the symlink, there's probably some other tool somewhere else thatreplaces the symlink. And when an admin does that, they get an unbootable system, and fix their playbook/whatever. Similarly, I bet someone somewhere has integrity management scripts that want to look at what pam modules are installed under /lib/security, and depending on how it worked, they had to adjust the config when that moved to /lib/security/x86_64-linux-gnu and then again some of them had to adjust when /lib became a symlink. And they were probably frustrated during the time when new installs had /lib as a symlink and upgrades did not. Similarly, I bet you can run into trouble with apparmor profiles if you try to profile /bin/python rather than /usr/bin/python or similar. I bet people who had custom selinux policies had to adjust their labeling rules and again some of them probably found the mixed state where upgraded systems behaved differently than installed systems frustrating. I seem to recall having to make some minor adjustments at my day job related to usrmerge back in the buster/bullseye time frame. I don't remember what they were. I think we've already committed to this pain, and I think we have enough of a window into that commitment that it doesn't seem to be very much pain. I mean spread across all our users, yes people have had to spend hundreds or thousands of hours dealing with this. But that's true with any upgrade. If we move back--if we unwind--things would get much much worse WRT this type of pain for a while. I think many more things would be sensitive to /bin/perl being a symlink than to /bin being a symlink. You could get into situations where /usr/bin/perl and /bin/perl were both files and differed. You could get into situations where /bin/perl vanished on one system. Etc. And if we actually try to have a symlink flower patch rather than a symlink farm, we are left with the pain of where things live differing between distributions and releases in a distribution. Mostly all we have left is the dpkg database. That pain will only affect people producing debs, which isn't just us. And yes, it will dis proportionally affect people who don't have our infrastructure and the ability to catch problems. Perhaps we should think about ways to mitigate that. There will be other smaller pains; if someone else had a system like
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sun, 27 Aug 2023 at 18:03, Sam Hartman wrote: > > > TL;DR: I think I understand one of Ian's points. I explain, but do not > believe it is compelling as an argument to switch direction. > > > "Helmut" == Helmut Grohne writes: > >> I think "package management" is the wrong term here. It's not > >> just our tools and packages that are affected. All forms of > >> operating system management are affected: anything that changes > >> the software, and not just its configuration. > >> > >> Affected tooling includes not just our .debs, but also remote > >> configuration management systems like Ansible, image construction > >> (Dockerfiles), and 3rd-party software installation progrmas > >> (including both 3rd-party .debs and 3rd-party script-based > >> installation systems). > > Helmut> I don't follow the reasoning. Much of the tasks you'd carry > Helmut> out with (wlog) ansible - even when updating files - will > Helmut> continue to work in the aliasing layout. The reason that > Helmut> dpkg is more affected is that it has an inventory of files > Helmut> and reasons about their ownership in terms of > Helmut> packages. That's not how any kind of configuration > Helmut> management operates. If you just "make install" something, > Helmut> chances are that it'll just work with an aliasing layout > Helmut> even when installing with --prefix=/. I continue to argue > Helmut> that the problems we are seeing are quite specific to dpkg > Helmut> in large parts. > > I spent some time with Ian's paragraph you quote above trying to > understand it, and I think I got somewhere. > I considered replying yesterday but decided against, because I think we > are already seeing these effects, and have been seeing them long enough > that we would have a good feel for how serious the problems are. > > I do think that configuration management does have enough of an > inventory of files and reasoning about structure that some of these > problems could arise. I agree that configuration management does not > typically reason in terms of packages. > > But mechanisms like > > * ADD/COPY in Containerfile > > * The rsync module in ansible > * The file module in Ansible > > * various inventories related to modification detection in > configuration management do reason about the filesystem. > > I don't know what would happen if I did > > hosts: lots > remote_user: root > name: Does this shoot me in the foot > tasks: > - rsync: src=install_root dest=/ > > where install_root has a bin directory containing a couple of scripts. > I don't know if rsync will replace a symlink with a directory, or will > just write through the symlink in that situation. > If rsync happens to write through the symlink, there's probably some > other tool somewhere else thatreplaces the symlink. > And when an admin does that, they get an unbootable system, and fix > their playbook/whatever. > > Similarly, I bet someone somewhere has integrity management scripts that > want to look at what pam modules are installed under /lib/security, and > depending on how it worked, > they had to adjust the config when that moved to > /lib/security/x86_64-linux-gnu and then again some of them had to adjust > when /lib became a symlink. And they were probably frustrated during > the time when new installs had /lib as a symlink and upgrades did not. > > Similarly, I bet you can run into trouble with apparmor profiles if you > try to profile /bin/python rather than /usr/bin/python or similar. > > I bet people who had custom selinux policies had to adjust their > labeling rules and again some of them probably found the mixed state > where upgraded systems behaved differently than installed systems > frustrating. AppArmor, SELinux, Ansible, Chef, Puppet and whatnot all exist and are used on all distributions. Such distributions have install bases vastly higher in numbers than Debian, and have adopted the usrmerged layout a decade ago or so. Is there any evidence of AppArmor, SELinux, Ansible, Chef, Puppet and whatnot breaking catastrophically as a consequence? Speculating about what might or might not happen with future events is all good and well, but this is in the past. These things either all broke and suddenly became unusable on Arch, Fedora, CentOS, RHEL, Alma, Rocky, SUSE, Mandriva, Yocto, Ubuntu and so on, or they did not. If there's no evidence they did, then it's not even worth mentioning or discussing, it's all moot. > I seem to recall having to make some minor adjustments at my day job > related to usrmerge back in the buster/bullseye time frame. > I don't remember what they were. This is much more likely to be closer to reality. As with any change or upgrade, there's some minor adjustment here or there, that most won't even remember about after a while given how trivial it is. I certainly do not remember all the changes I've had to do to software I maintain when switch
Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sun, 27 Aug 2023 at 11:30, Matthew Vernon wrote: > > Dear Luca, > > On 27/08/2023 03:16, Luca Boccassi wrote: > > [things] > > You've already been asked by a couple of people to moderate your tone in > this thread. I appreciate there is a lot of frustration around > /usr-merge, but your contributions are not helping with that at all. Nor > do they help us have a constructive discussion. Constructive discussions about past events need to be based on facts and evidence. Both are conspicuously absent from the bug submitter's messages. Without any of that, there was never any possibility that this bug could be in any way constructive since the moment it was opened, long before I first replied. > The DEP-17 work continues whilst this discussion is ongoing; this > discussion is not delaying that work. > > I think it is fair to say that when ctte decided on /usr-merge it was > expected that a) the process would be reasonably straightforward and b) > dpkg could be taught to understand directory aliasing, so we would not > have to be doing things "behind dpkg's back", as it were. Sure, but then it turned out that package maintainers are not only allowed to ignore the CTTE and its formal decisions, but can also actively block them in their packages. How could responsibility of that be assigned to anybody but those who chose to do that completely eludes me - and yet, that's exactly what the bug submitter does. > I think the need for a releases-long moratorium on moving files and the > volume and depth of technical work represented by DEP17 demonstrate that > those expectations turned out not to be met. > > Given that, it seems to me that a) warnings that "it's not that simple & > easy" were not FUD and describing them thus is unhelpful b) it's > reasonable to at least ask the question "given what we know now, are we > still sure this is the correct approach?" a) except that warnings weren't as mild and meek as "it's not that simple", we were clearly told everything would break left and right, and yet that didn't happen. Sure, dpkg has annoying bugs that need workarounds, but that's hardly surprising given its situation. But these are all packaging-only issues that affect some distribution maintainers for some package builds and maintenance tasks, and that's about it. No installation is affected. No user is affected. No external/third party developer is affected. So, yes, all of that was and still is FUD. b) no, in August 2023 it is very much not reasonable, and no fully informed and evidence-based good faith discussion would ever ask something like that. The reasonable questions to ask in August 2023 are along the lines of "what's the best way to work around those dpkg bugs and lift the moratorium?", which is what we are doing in actual good faith discussions elsewhere, with very productive results. > Any such consideration must be mindful of the fact that the majority of > Debian installs are now /usr-merged, which means that the complexity of > unwinding such installs has to be a heavy factor in thinking about > alternative approaches. > > I'm hopeful, but not certain, that the DEP17 work will get us to a > coherent state again by foxy (which still feels a long time off); I > don't think we should underestimate the work to be done in properly > completing /usr-merge. > > This discussion has, I think, been broadly useful in clarifying some > folks' thinking in this area. I would very much like if we could keep it > focused on the technical matters. I honestly cannot see how there can be any useful technical discussion about a completely evidence-free proposal to move to a symlinks-farm layout in 2023, with all the already well-known wider context that has been explained multiple times. The useful and productive and high-quality technical discussions are happening elsewhere.
