Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Nikolaus Rath]

On May 17 2023, Andrea Pappacoda  wrote:
> Hi all,
>
> first of all thank you for this great thread. While I could feel some tension 
> while
> reading it, it's completely normal and I've learned a lot.
>
> I have a question though: if /lib64/ld-linux-x86-64.so.2 is already a symlink 
> on
> non-merged-/usr systems, pointing to 
> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, why would
> it be an issue to have /lib64/ld-linux-x86-64.so.2 point to 
> /usr/lib64/ld-linux-x86-64.so.2?

I don't think it would be, and I don't think anyone else is saying it would be.

> Why do we want binaries to look for the loader in
> /usr/lib64/ld-linux-x86-64.so.2 if that would still be a symlink to
> /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2?

My understanding is that there is a desire for the /lib64 symlink not to
be needed, because it would simplify bootstrapping new systems.


Best,
Nikolaus
-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:
> On Wed, 17 May 2023 at 01:05, Russ Allbery  wrote:

>> I do think the industry is moving away (well, has already moved away)
>> from Linux Standards Base pre-compiled C binaries without wrappers like
>> snap or flatpak, although there are some very notable exceptions, such
>> as Steam's special use cases explained elsewhere on the thread.  I
>> don't believe the industry is moving away from Go and Rust binaries, so
>> this is, I think, a more complicated picture than your description
>> indicates.

> I know absolutely nothing about Go, and preciously little about Rust,
> but my understanding from news article and so on was that both of those
> ecosystems pretty openly avoid "traditional" distributions, in favour of
> their own self-contained ecosystems.

I think apt-cache search will show that's a significant overstatement and
quite a lot of Go and Rust code is packaged for Debian.  Lots of Debian
maintainers put a lot of hard work into that.

> So would that use case be affected, even by an hypothetical and at this
> point extremely unlikely archive-wide change?

If we only modified specific core packages to use a different PT_INTERP,
obviously only those packages would be affected and execution of general
Go and Rust binaries would not be affected.  I suspect none of those core
packages today are Go or Rust.  (I do think it's likely that we'll have
critical bootstrapping packages written in Rust within the next decade,
though.)

If all Debian packages were built with a different PT_INTERP but the
toolchain itself was not modified, then Go and Rust binaries packaged for
Debian could not be copied to and run on non-Debian Linux systems (unless
it happened to work by accident because those Linux systems were also
/usr-merged and thus the non-canonical PT_INTERP also was present on those
systems).  This would be unfortunate and surprising and I'm sure that it
would break use cases that currently work, but it probably wouldn't be
catastrophic.  Still, it would obviously be a regression.

If the toolchain shipped with Debian were modified to build binaries with
a different PT_INTERP, then Go and Rust binaries built with that toolchain
would not work on other Linux distributions.  If every other major Linux
distribution were /usr-merged and thus the modified PT_INTERP worked
somewhat by accident on most other distributions, this likewise would
still break some things for some people, but similar to the previous point
it would be a regression but probably wouldn't be catastrophic.  It would
be more serious than the previous problem, though, since now we're
affecting user-built binaries and not just packaged binaries.  Many more
of those will be built with the intent to copy them to other systems.

If changes caused Go and Rust binaries built on Debian to not work on
numerous major non-Debian Linux distributions, I would consider that a
serious release-critical bug.  But that doesn't seem all that likely from
this specific change assuming that the modified PT_INTERP would also be
present on other major Linux distributions such as Red Hat, Fedora, SuSE,
Arch, Gentoo, etc.  I am *assuming* those are all /usr-merging, and thus
it might be, but I have not tracked this and do not know if that's true.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Wed, 17 May 2023 at 01:05, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > It does say something interesting. When we started, the assertion was
> > that packages not relying on the symlink being present was fundamental
> > for portability and cross-compatibility. Then, it shrinked to
> > portability and cross-compatibility of a subset of packages. Now it
> > further shrank to portability and cross-compatibility of a subset of
> > packages of a subset of vintages.
>
> I think it's pretty clear that I'm totally failing to communicate what I
> was getting at with this example and the process is becoming incredibly
> frustrating, so I'm going to stop trying here.
>
> > But does it really work, or do we just hope it does? I personally see a
> > great deal of signals that say it doesn't. For one, in 2023, "build one
> > binary and run it everywhere" doesn't really seem to me to be the
> > principal choice in terms of portability.
>
> Well, believe what you believe, but I literally do that daily, as does
> anyone else who regularly uses software from a Rust or Go ecosystem.  Not
> a single work day goes by without me running, on some random Ubuntu or Red
> Hat or Debian system, binaries that were compiled on some random other
> Linux distribution (often I have no idea which one).
>
> > It might have been once, and there might be a number of relevant use
> > cases still, but it seems to me the industry has largely gone in a very
> > different direction nowadays.
>
> I do think the industry is moving away (well, has already moved away) from
> Linux Standards Base pre-compiled C binaries without wrappers like snap or
> flatpak, although there are some very notable exceptions, such as Steam's
> special use cases explained elsewhere on the thread.  I don't believe the
> industry is moving away from Go and Rust binaries, so this is, I think, a
> more complicated picture than your description indicates.

I know absolutely nothing about Go, and preciously little about Rust,
but my understanding from news article and so on was that both of
those ecosystems pretty openly avoid "traditional" distributions, in
favour of their own self-contained ecosystems. So would that use case
be affected, even by an hypothetical and at this point extremely
unlikely archive-wide change? Meaning, are those randomly compiled
Go/Rust binaries you run for work taken from the Debian archive, or
locally/externally/self compiled? I do know we do ship a number of
those, although I am very oblivious to the number and nature of that
subset. And conversely, if the hypothetical change was just to the
essential set, I assume there's no Go/Rust there, right?

> > Not everything of course, but wouldn't it be worth it to specify core
> > items such as cross-compatibility requirements? Not in terms of
> > implementation details, but in terms of what the goals and the minimum
> > expectations are. There are obviously some, and they are obviously of
> > great importance to many, and yet to me it feels like they shift every
> > time I ask a different question.
>
> I am certainly never going to say no to more maintained documentation.
> That would be great.  If you're feeling motivated and you're willing to
> really listen to what people are saying and accept and understand a lot of
> ambiguity and a lot of assumptions that don't match your assumptions, go
> for it!
>
> I personally am not even doing justice to Policy in its existing scope, so
> this isn't something I will personally be tackling.  Honestly, I should
> have spent all of the time I spent on this thread working on Policy
> instead.  :)

Sure, I can add that to my todo list. But I have already two policy
items open for review, so certainly won't get to it before those are
sorted, hint hint :-)
(probably it would have been best not to mention it, lol)

> > But if you prefer to focus on first principles, that's great! I've been
> > asking this a lot: is cross-distribution harmonization a core
> > requirement? Is it important enough to trump other goals and
> > distro-local benefits? If so, isn't it worth to discuss it and then
> > pencil it down?
>
> I think some parts of it are a core requirement.  It would be very
> surprising, and very bad, if we couldn't run Go and Rust binaries built on
> another distribution, for example, or if Go or Rust binaries built on
> Debian couldn't be used on other distributions, both subject to the normal
> constraints of glibc cross-compatibility that everyone building binaries
> already knows about.  I think other parts of it are not a core
> requirement, but still something that is nice to have and that we
> shouldn't break without a really good reason.
>
> I think the specific details of the Linux Standards Base process mostly
> didn't turn into something the Linux world wanted to support going
> forward, and thus LSB harmonization, while an interesting idea, is no
> longer a requirement in general.  However, we still follow some pieces of
> it that 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Roger Lynn]

On 15/05/2023 19:00, Simon McVittie wrote:
> On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
>> People build things on Debian that are not Debian packages. People
>> compile binaries on Debian, and expect them to work on any system that
>> has sufficiently new libraries.
> 
> *raises hand*
> 
> Hello, I represent an example of those people. With my $work hat on,
> I'm involved in maintaining a family of Debian derivatives (the Steam
> Runtime) that is used to run Steam games on arbitrary distributions,
> including but not limited to Debian. Some of these binaries are built
> on a Debian derivative, but run on an arbitrary other distribution,
> using a RPATH[1] to find their non-glibc dependencies.

As another much smaller example, which I hope is still relevant, I have
taken binaries from Debian stable or oldstable armel and run them for
diagnostic purposes on embedded boards which only have Busybox installed and
for which I don't have a convenient build environment.

Regards,
Roger

PS Apologies if I've followed up incorrectly - I read debian-devel through
an NNTP gateway. - And I screwed up the reply so Simon and the bug got a
different copy.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Jeremy Stanley  writes:

> Throwing another common one on the heap, similar to the previous Steam
> example, Python wheels with compiled extensions are often distributed on
> PyPI for a fictional "manylinux" platform which indicates they're
> intended to be usable on most GNU/Linux distributions (though in their
> case, they compile on ~ancient RHEL versions to achieve that). One way
> that manulinux wheels are less like Rust or Go is that they typically
> don't use static compilation, but rather vendor in additional
> dynamically-linked libs which are unlikely to be present on the target
> installations.

Yeah, I've not been digging too deeply into cases like this since they
wouldn't be affected by the original proposal that started this off (it
wouldn't change binaries compiled with normal Debian tools or change what
ABI Debian can execute), but if we move into the more general question of
"does anyone care about building binaries on one Linux system and running
them on another," I think there are quite a lot of examples.

Another one just from the Python world is conda, which provides a parallel
set of pre-compiled libraries in addition to the Python modules and
expects to be able to install and run them on arbitrary systems outside of
a chroot or container, just by manipulating the dynamic loader search
path.  Conda is very widely used in scientific computing, where a lot of
use cases require Python libraries with substantial underlying C library
components that often do not have compatible or sufficiently optimized
versions packaged for the underlying distribution.

There are also, obviously, non-free software use cases galore, including I
suspect some in the Debian non-free archive, where binaries built on other
Linux distributions are being run on Debian systems.  I remember, in a
previous job, changing a few files in /etc to convince the Oracle
installer that a Debian system was actually a Red Hat system so that it
would run, and the resulting binaries then worked fine (and it would have
been a huge problem for us if they hadn't).  I think Oracle subsequently
added more support for Debian, or at least Ubuntu, but I'm sure there are
other cases like that still around today.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Jeremy Stanley]

On 2023-05-16 17:05:25 -0700 (-0700), Russ Allbery wrote:
[...]
> Well, believe what you believe, but I literally do that daily, as
> does anyone else who regularly uses software from a Rust or Go
> ecosystem.  Not a single work day goes by without me running, on
> some random Ubuntu or Red Hat or Debian system, binaries that were
> compiled on some random other Linux distribution (often I have no
> idea which one).
[...]

Throwing another common one on the heap, similar to the previous
Steam example, Python wheels with compiled extensions are often
distributed on PyPI for a fictional "manylinux" platform which
indicates they're intended to be usable on most GNU/Linux
distributions (though in their case, they compile on ~ancient RHEL
versions to achieve that). One way that manulinux wheels are less
like Rust or Go is that they typically don't use static compilation,
but rather vendor in additional dynamically-linked libs which are
unlikely to be present on the target installations.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Andrea Pappacoda]

Hi all,

first of all thank you for this great thread. While I could feel some 
tension while reading it, it's completely normal and I've learned a lot.


I have a question though: if /lib64/ld-linux-x86-64.so.2 is already a 
symlink on non-merged-/usr systems, pointing to 
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, why would it be an issue to 
have /lib64/ld-linux-x86-64.so.2 point to 
/usr/lib64/ld-linux-x86-64.so.2? Why do we want binaries to look for 
the loader in /usr/lib64/ld-linux-x86-64.so.2 if that would still be a 
symlink to /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2?


Sorry in advance if this sounds like a dumb question, and thanks again 
for your involvement in all of this!

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:

> It does say something interesting. When we started, the assertion was
> that packages not relying on the symlink being present was fundamental
> for portability and cross-compatibility. Then, it shrinked to
> portability and cross-compatibility of a subset of packages. Now it
> further shrank to portability and cross-compatibility of a subset of
> packages of a subset of vintages.

I think it's pretty clear that I'm totally failing to communicate what I
was getting at with this example and the process is becoming incredibly
frustrating, so I'm going to stop trying here.

> But does it really work, or do we just hope it does? I personally see a
> great deal of signals that say it doesn't. For one, in 2023, "build one
> binary and run it everywhere" doesn't really seem to me to be the
> principal choice in terms of portability.

Well, believe what you believe, but I literally do that daily, as does
anyone else who regularly uses software from a Rust or Go ecosystem.  Not
a single work day goes by without me running, on some random Ubuntu or Red
Hat or Debian system, binaries that were compiled on some random other
Linux distribution (often I have no idea which one).

> It might have been once, and there might be a number of relevant use
> cases still, but it seems to me the industry has largely gone in a very
> different direction nowadays.

I do think the industry is moving away (well, has already moved away) from
Linux Standards Base pre-compiled C binaries without wrappers like snap or
flatpak, although there are some very notable exceptions, such as Steam's
special use cases explained elsewhere on the thread.  I don't believe the
industry is moving away from Go and Rust binaries, so this is, I think, a
more complicated picture than your description indicates.

> Not everything of course, but wouldn't it be worth it to specify core
> items such as cross-compatibility requirements? Not in terms of
> implementation details, but in terms of what the goals and the minimum
> expectations are. There are obviously some, and they are obviously of
> great importance to many, and yet to me it feels like they shift every
> time I ask a different question.

I am certainly never going to say no to more maintained documentation.
That would be great.  If you're feeling motivated and you're willing to
really listen to what people are saying and accept and understand a lot of
ambiguity and a lot of assumptions that don't match your assumptions, go
for it!

I personally am not even doing justice to Policy in its existing scope, so
this isn't something I will personally be tackling.  Honestly, I should
have spent all of the time I spent on this thread working on Policy
instead.  :)

> But if you prefer to focus on first principles, that's great! I've been
> asking this a lot: is cross-distribution harmonization a core
> requirement? Is it important enough to trump other goals and
> distro-local benefits? If so, isn't it worth to discuss it and then
> pencil it down?

I think some parts of it are a core requirement.  It would be very
surprising, and very bad, if we couldn't run Go and Rust binaries built on
another distribution, for example, or if Go or Rust binaries built on
Debian couldn't be used on other distributions, both subject to the normal
constraints of glibc cross-compatibility that everyone building binaries
already knows about.  I think other parts of it are not a core
requirement, but still something that is nice to have and that we
shouldn't break without a really good reason.

I think the specific details of the Linux Standards Base process mostly
didn't turn into something the Linux world wanted to support going
forward, and thus LSB harmonization, while an interesting idea, is no
longer a requirement in general.  However, we still follow some pieces of
it that were properly implemented (like the FHS), and while we shouldn't
do that blindly forever (if for no other reason than the FHS is no longer
maintained), it's also valuable to not change that too fast and to only
break compatibility with now-widely-expected file system layout properties
when we have a really good reason.  Ideally, we would pick some smaller
subset of the LSB that still matters and agree with other major
distributions on some points of compatibility to, at the very least, help
ease the common problem of needing to administer multiple systems running
different Linux distributions.

There is no one answer to whether this trumps other goals and distro-local
benefits.  It depends on what those benefits are and what those goals are
and how important they are.  For Guix, obviously their immutable tree and
hard dependency pinning are more important to them than cross-distro
compatibility, and given their goals, that seems like an entirely
reasonable decision.  I would disagree vehemently with that decision for
Debian because Debian is not Guix.

In other words, it depends.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Tue, 16 May 2023 at 09:27, Simon McVittie  wrote:
>
> On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> > This sounds like a very interesting use case, and the first real one
> > mentioned, which is great to see - but I do not fully follow yet, from
> > what you are saying it seems to me that what you need is for your
> > binaries to use the usual pt_interp, that bit is clear. But why does
> > it matter if /usr/bin/ls on the host uses a different one?
>
> We don't need to run the ls from the host, but we do need to run
> glibc-related executables like ldconfig and localedef from either the
> host or the container runtime, whichever is newer. Because glibc is
> a single source package, executables and libraries within the glibc
> bubble sometimes make use of private symbols in libraries that are also
> within the glibc bubble (and IMO they have a right to do so), even though
> executables from outside glibc would be discouraged or disallowed from
> doing so. This means that when we have chosen a particular version of
> glibc (which, again, must be whichever one is newer), we try to use its
> matching version for *everything* originating in the glibc source package.
>
> In principle we could get exactly the same situation if we've imported a
> library from the host system (as a dependency of the graphics stack) that
> calls an executable as a subprocess and expects it to be >= the version
> it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

Thanks for the clarification, so if I understood correctly, your use
case is that sometimes (eg: when they are newer) you pull binaries
(eg: ldconfig) from the host, and run them from the container? So, in
case let's say ldconfig on the host points to /usr/lib/ld, but because
your container is not usr-merged, it wouldn't find the interpreter and
fail?

> The wider point than my specific use-case, though, is that when there's a
> standard, you can't predict what other software authors have looked at the
> statement "you can rely on this" and ... relied on it. See also Russ's
> (as ever, excellent) mails to the same thread.
>
> I appreciate that you are trying to explore the edges of the
> problem/constraint space and say "what if we did this, could that work?",
> and it's good that you are doing that; but part of that process is
> working with the other people on this list when they say "no, we can't
> do that because...", and respecting their input.

I respect and appreciate the input, but I want to understand it too,
hence the "because..." part is what I was looking for - so thanks for
providing it, it is really useful.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Tue, 16 May 2023 at 04:22, Russ Allbery  wrote:
> Luca Boccassi  writes:
> > On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:
>
> >> Note that we're not talking about complicated packages with lots of
> >> runtime like, say, Emacs.  As I understand it your proposal wouldn't
> >> change PT_INTERP for that binary anyway.  We're presumably talking
> >> about the kind of binaries that you need to bootstrap a minimal system,
> >> so packages like coreutils or bash.  And I would indeed expect those
> >> binaries to be generally portable, as long as the same critical shared
> >> libraries are available on other systems (in this case, PCRE2 and
> >> ncurses).
>
> > Is that really the case? Let's test that hypothesis:
>
> I think you may have not read my whole message before replying, and also
> please assume that I know really basic facts about glibc compatibility and
> am not referring to that.
>
> I said "of a similar vintage" (farther down in my message) because of
> course we all know that binaries built against newer versions of glibc
> don't run on systems with older versions of glibc (and likewise for shared
> libraries in general and thus libselinux), and you tested a Debian
> unstable package on an Ubuntu system from 2020.  This says nothing
> interesting and has nothing to do with my point.

It does say something interesting. When we started, the assertion was
that packages not relying on the symlink being present was fundamental
for portability and cross-compatibility. Then, it shrinked to
portability and cross-compatibility of a subset of packages. Now it
further shrank to portability and cross-compatibility of a subset of
packages of a subset of vintages.
Why is the requirement that libselinux and glibc are not too recent
fine to have on coreutils, but the requirement that there's a symlink
is out of the question? If anything, the latter is trivial to add, so
the formers should feel more "disruptive" and game-breaking, no?

> > Whoops. Does this make coreutils rc-buggy now? ;-)
>
> You are the only person who is talking about RC bugs.  The bar here is not
> "prove to me that this is RC buggy," the bar is "you have to prove to a
> bunch of Debian core maintainers that they should break the ABI in their
> packages" (not to mention the additional small but permanent build
> complexity).  Demanding they prove to you that it's a bad idea is not how
> this works.

It's a tongue-in-cheek comment, I had hoped the emojii would give that away.

> The point of standards like an ABI is that a bunch of entirely unrelated
> people who never talk to each other and never look at each other's
> software are allowed to rely on them and assume no one else will break
> them.  This is how free software scales; without invariants that everyone
> can rely on without having to explain how they're relying on them, it is
> much more difficult to get an ecosystem to work together.  We don't just
> break things because they don't seem important; the space of people who
> may be relying on this standard is unknowable, which is the entire point.
> Opening those boxes is really expensive (in time, planning, communication,
> debugging, and yes, compatibility) and we should only do it when it
> really, really matters.

But does it really work, or do we just hope it does? I personally see
a great deal of signals that say it doesn't. For one, in 2023, "build
one binary and run it everywhere" doesn't really seem to me to be the
principal choice in terms of portability. It might have been once, and
there might be a number of relevant use cases still, but it seems to
me the industry has largely gone in a very different direction
nowadays.

> > It did look like a veto to me. More importantly, isn't relying on
> > passersby to spot alleged harmful changes dangerous, especially for
> > undocumented, uncodified and untested use cases, like unspecified and
> > vague cross-compatibility requirements?
>
> I'm honestly boggled.  This is a thread on debian-devel, which is
> literally how we do architecture vetting in this project.
>
> I absolutely do not think that we can write down everything of importance
> in designing a distribution so that we don't have to have conversations
> with other people in the project who have deep expertise when considering
> a significant architectural change like changing the PT_INTERP path of
> core binaries.

Not everything of course, but wouldn't it be worth it to specify core
items such as cross-compatibility requirements? Not in terms of
implementation details, but in terms of what the goals and the minimum
expectations are. There are obviously some, and they are obviously of
great importance to many, and yet to me it feels like they shift every
time I ask a different question.

> >> I mostly jumped in because it felt like you and Steve were just yelling
> >> at each other and I thought I might be able to explain some of where he
> >> was coming from in a way that may make more sense.
>
> > I don't 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Didier 'OdyX' Raboud  writes:

> This has existed in a (now distant) past as the "Linux Distribution
> Checker", in the context of the Linux Standard Base, that Debian and
> Ubuntu stopped caring about in late 2015.

Ah, yes, thank you, that makes sense.

> I'm not aware of more recent efforts in that direction; but it's an
> understatement to say the landscape has changed quite a bit since:
> containers, sandbox environments (and others) have forever changed the
> way we think about distributing binary executable. LSB had that
> ambition, and failed.

While that is certainly true, I feel like the pendulum may be swinging
back in a slightly different way with Go and Rust popularizing the idea
that you should be able to copy around a binary and run it on any Linux
system with a compatible architecture.  This is a much smaller problem
than LSB was trying to solve since LSB was trying to standardize things
like the shared library ABI and SONAMEs, which Go and Rust intentionally
avoid with static linking.  But they do rely very deeply on every system
being able to execute binaries built to the Linux ABI and glibc.  (I
realize that's a different question than the one discussed in this
thread.)

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Didier 'OdyX' Raboud]

Le mardi, 16 mai 2023, 17.06:38 h CEST Russ Allbery a écrit :
> I don't know if anyone has written an ABI compliance test for binaries.
> That sounds like something that would be in scope for the Linux Test
> Project, though, and it's possible their existing tests do some of this.

This has existed in a (now distant) past as the "Linux Distribution Checker", 
in the context of the Linux Standard Base, that Debian and Ubuntu stopped 
caring about in late 2015.

I'm not aware of more recent efforts in that direction; but it's an 
understatement to say the landscape has changed quite a bit since: containers, 
sandbox environments (and others) have forever changed the way we think about 
distributing binary executable. LSB had that ambition, and failed.

-- 
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

James Addison  writes:

> We've almost certainly all encountered limitations in upstream
> specifications and wondered when it's worth attempting a perceived
> improvement despite potential friction.

> If Debian did want/need to change the PT_INTERP path, is there a way to
> achieve that in both a standards-compliant and also a distro-compatible
> way?

My recollection is that we considered this when starting with multiarch
but none of the other distributions outside of Ubuntu were very enthused,
so we dropped it.  I saw those discussions on the glibc development lists,
which is not really the place for detailed x86_64 ABI discussion but which
would certainly be an interested party and has roughly the right people
present.  If I saw a problem that would need to be addressed with an ABI
change and didn't have anyone else to ask, that's where I personally would
start, but the Debian gcc and glibc maintainers would probably know more.

The bar for justifying a change will be fairly high, based on past
discussions.  I would expect it to require some sort of significant bug
that can't be worked around in another way.

> Although tests may not exist publicly in this case, the idea of using
> tests where possible to catch regressions seems relevant to me.  Tests
> can help people to identify when a compatibility boundary has been
> reached or overstepped, and, socially, they can provide a clear place to
> gather discussion if & when they become outdated (particularly if the
> tests are themselves are provided free and open source).  Copying
> binaries and running them seems like a form of testing, but perhaps we
> could find better ways.

I don't know if anyone has written an ABI compliance test for binaries.
That sounds like something that would be in scope for the Linux Test
Project, though, and it's possible their existing tests do some of this.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> This sounds like a very interesting use case, and the first real one
> mentioned, which is great to see - but I do not fully follow yet, from
> what you are saying it seems to me that what you need is for your
> binaries to use the usual pt_interp, that bit is clear. But why does
> it matter if /usr/bin/ls on the host uses a different one?

We don't need to run the ls from the host, but we do need to run
glibc-related executables like ldconfig and localedef from either the
host or the container runtime, whichever is newer. Because glibc is
a single source package, executables and libraries within the glibc
bubble sometimes make use of private symbols in libraries that are also
within the glibc bubble (and IMO they have a right to do so), even though
executables from outside glibc would be discouraged or disallowed from
doing so. This means that when we have chosen a particular version of
glibc (which, again, must be whichever one is newer), we try to use its
matching version for *everything* originating in the glibc source package.

In principle we could get exactly the same situation if we've imported a
library from the host system (as a dependency of the graphics stack) that
calls an executable as a subprocess and expects it to be >= the version
it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

The wider point than my specific use-case, though, is that when there's a
standard, you can't predict what other software authors have looked at the
statement "you can rely on this" and ... relied on it. See also Russ's
(as ever, excellent) mails to the same thread.

I appreciate that you are trying to explore the edges of the
problem/constraint space and say "what if we did this, could that work?",
and it's good that you are doing that; but part of that process is
working with the other people on this list when they say "no, we can't
do that because...", and respecting their input.

Thanks,
smcv

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[James Addison]

On Tue, 16 May 2023 at 04:22, Russ Allbery  wrote:
>
> > It did look like a veto to me. More importantly, isn't relying on
> > passersby to spot alleged harmful changes dangerous, especially for
> > undocumented, uncodified and untested use cases, like unspecified and
> > vague cross-compatibility requirements?
>
> I'm honestly boggled.  This is a thread on debian-devel, which is
> literally how we do architecture vetting in this project.
>
> I absolutely do not think that we can write down everything of importance
> in designing a distribution so that we don't have to have conversations
> with other people in the project who have deep expertise when considering
> a significant architectural change like changing the PT_INTERP path of
> core binaries.

We've almost certainly all encountered limitations in upstream
specifications and wondered when it's worth attempting a perceived
improvement despite potential friction.

If Debian did want/need to change the PT_INTERP path, is there a way
to achieve that in both a standards-compliant and also a
distro-compatible way?

And if there isn't: how would we resolve that?


This conversation is already lengthy, and based on recent experience
I'm definitely the kind of person who doesn't always read all the
details - so I don't know whether it's a good idea for anyone to
respond to my message.  But I haven't seen anyone discussing or
providing a safe migration path.

Although tests may not exist publicly in this case, the idea of using
tests where possible to catch regressions seems relevant to me.  Tests
can help people to identify when a compatibility boundary has been
reached or overstepped, and, socially, they can provide a clear place
to gather discussion if & when they become outdated (particularly if
the tests are themselves are provided free and open source).  Copying
binaries and running them seems like a form of testing, but perhaps we
could find better ways.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Bastian Blank]

Hi Steve

On Mon, May 15, 2023 at 02:01:15AM +0100, Steve McIntyre wrote:
> Russ has described copying *binaries* out of packages and running them
> elsewhere. I've done that too, from time to time. This is one of the
> things made possible by the ABI contract being followed.

And nothing in that requires that the interpreter matches.  Because
glibc is clever enough to allow running the interpreter standalone.  So
in any way you can just call:

/lib64/ld-linux-x86-64.so.2 /bin/ls

Bastian

-- 
Oh, that sound of male ego.  You travel halfway across the galaxy and
it's still the same song.
-- Eve McHuron, "Mudd's Women", stardate 1330.1

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

I'm dropping the TC bug from this thread, since I don't think it has
anything to do with that discussion at this point.  I probably should also
change the Subject line, but I'm keeping it to make it easier for the
people who want to tune out this thread, since I very much doubt they want
to tune back in at this point.

Luca Boccassi  writes:
> On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:

>> Note that we're not talking about complicated packages with lots of
>> runtime like, say, Emacs.  As I understand it your proposal wouldn't
>> change PT_INTERP for that binary anyway.  We're presumably talking
>> about the kind of binaries that you need to bootstrap a minimal system,
>> so packages like coreutils or bash.  And I would indeed expect those
>> binaries to be generally portable, as long as the same critical shared
>> libraries are available on other systems (in this case, PCRE2 and
>> ncurses).

> Is that really the case? Let's test that hypothesis:

I think you may have not read my whole message before replying, and also
please assume that I know really basic facts about glibc compatibility and
am not referring to that.

I said "of a similar vintage" (farther down in my message) because of
course we all know that binaries built against newer versions of glibc
don't run on systems with older versions of glibc (and likewise for shared
libraries in general and thus libselinux), and you tested a Debian
unstable package on an Ubuntu system from 2020.  This says nothing
interesting and has nothing to do with my point.

> Whoops. Does this make coreutils rc-buggy now? ;-)

You are the only person who is talking about RC bugs.  The bar here is not
"prove to me that this is RC buggy," the bar is "you have to prove to a
bunch of Debian core maintainers that they should break the ABI in their
packages" (not to mention the additional small but permanent build
complexity).  Demanding they prove to you that it's a bad idea is not how
this works.

The point of standards like an ABI is that a bunch of entirely unrelated
people who never talk to each other and never look at each other's
software are allowed to rely on them and assume no one else will break
them.  This is how free software scales; without invariants that everyone
can rely on without having to explain how they're relying on them, it is
much more difficult to get an ecosystem to work together.  We don't just
break things because they don't seem important; the space of people who
may be relying on this standard is unknowable, which is the entire point.
Opening those boxes is really expensive (in time, planning, communication,
debugging, and yes, compatibility) and we should only do it when it
really, really matters.

> It did look like a veto to me. More importantly, isn't relying on
> passersby to spot alleged harmful changes dangerous, especially for
> undocumented, uncodified and untested use cases, like unspecified and
> vague cross-compatibility requirements?

I'm honestly boggled.  This is a thread on debian-devel, which is
literally how we do architecture vetting in this project.

I absolutely do not think that we can write down everything of importance
in designing a distribution so that we don't have to have conversations
with other people in the project who have deep expertise when considering
a significant architectural change like changing the PT_INTERP path of
core binaries.

>> I mostly jumped in because it felt like you and Steve were just yelling
>> at each other and I thought I might be able to explain some of where he
>> was coming from in a way that may make more sense.

> I don't believe I've done any yelling here.

I'm using yelling pretty broadly and probably rather inaccurately here.
Maybe a better way of putting it is that Steve was yelling and you didn't
appear to be listening or understanding why he was yelling and were
responding in ways that were guaranteed to make him yell more.

You *are* coming across as kind of contemptuous of other people's
arguments, but then it's entirely possible that I am as well, so I'm
trying to ignore it.

> What if "setting the parking brake" is not enough, as the wheels are
> already off and rolling downhill, as shown above, because while
> everybody was looking at the parking brakes lever somebody ran off with
> the bolts that kept the wheels attached? Why is it worth worrying about
> compatibility with something that is already not compatible, and it's
> not expected to be compatible for almost all other aspects?

You can certainly decide to try to make the argument that ABI
compatibility between Linux distributions is a lost cause and we should
give up and not care about it any more.  That is a valid architectural
argument.  (To be clear, I don't think this is the argument you've been
making up to this point, which is about a very limited ABI break in
specific packages to achieve a specific purpose.)  I don't agree with that
argument, which doubtless doesn't come as a surprise, and your

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 18:54, Simon McVittie  wrote:
>
> On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> > People build things on Debian that are not Debian packages. People
> > compile binaries on Debian, and expect them to work on any system that
> > has sufficiently new libraries.
>
> *raises hand*
>
> Hello, I represent an example of those people. With my $work hat on,
> I'm involved in maintaining a family of Debian derivatives (the Steam
> Runtime) that is used to run Steam games on arbitrary distributions,
> including but not limited to Debian. Some of these binaries are built
> on a Debian derivative, but run on an arbitrary other distribution,
> using a RPATH[1] to find their non-glibc dependencies.
>
> At the moment those binaries are built in ancient environments (based
> on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
> ubiquitous, we'll want to start relying on newer versions of Debian
> (which will still be very old versions *at the time*, but I'm sure that
> by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
> are not released yet). In this use-case, yes we do need to be using the
> interoperable ELF interpreter path.
>
> We were able to use (Ubuntu and) Debian for this *because* Debian is
> a relatively "ordinary" distribution that tends to follow cross-distro
> standards. The major counterexample is multiarch library paths, but
> multiarch library paths have some compelling technical advantages, and
> because there's no ambiguity about which architecture uses which directory,
> they're actually better for interop in some ways than /usr/lib (which
> is ambiguous, because the Red Hat family of distros expects to find 32-bit
> libraries there, but the Arch family expects 64-bit libraries, and it's
> not possible to construct a tree where both get what they want).
>
> I've spent a lot of the last 5 years working on putting Steam games in
> containers so that they'll work more reliably on all distros, including
> Debian, with less reliance on weird library search paths (although we
> still have to use binaries built on an ancient Debian derivative with a
> non-trivial RPATH to bootstrap the container environment). Because of
> constraints around recent GPUs needing current graphics drivers even
> if running on an otherwise ancient library stack, Steam's container
> runtime has to construct a hybrid environment where glibc is either the
> one from the host or the one from the container runtime environment,
> whichever is newer; and while doing that, we have experienced some
> broken situations that were caused by distributions diverging from the
> interoperable ELF interpreter. One concrete example is that Arch Linux
> uses the interoperable ELF interpreter for *almost* all executables,
> but uses a different ELF interpreter for executables from the glibc
> source package, for whatever reason; we were able to work around this,
> but for a while it caused Arch to fail to launch these containers where
> Debian/Fedora/etc. could.

This sounds like a very interesting use case, and the first real one
mentioned, which is great to see - but I do not fully follow yet, from
what you are saying it seems to me that what you need is for your
binaries to use the usual pt_interp, that bit is clear. But why does
it matter if /usr/bin/ls on the host uses a different one? It's your
executables that you ship as part of that runtime that are the entry
points that need the usual loader path for your chroot-on-steroids,
no? The loader would still be reachable as it always was in this
theoretical exercise. I am probably missing something in how this
works in details.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
> > On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:
>
> >> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
> >> major player in Linux distributions, and I'm not sure how much they
> >> care about compatibility with anyone else.)
>
> > This is a counter-example to confute the assertion that *everybody* does
> > the same thing, which has been made multiple times. I'm not sure whether
> > it's an experiment or not, I mean if you ask their users/developers I
> > think they'd tell you it's very much production ready, but it is largely
> > irrelevant: it exists, and that's the only reason why it was mentioned,
> > as it shows that it is _possible_ to do that and be a working
> > distribution. Doesn't imply it's automatically desirable, but that was
> > not the intention.
>
> Ah, okay, I'm happy to agree with that point: you can violate the ABI and
> continue to be a working distribution.  (There are a lot of parts of the
> ABI that if you violated them you would not have a working distribution,
> but this is not one of them so far as I can tell.)
>
> > Not quite: my argument is that binaries from these packages are not
> > intended and not required to be ran on non-Debian systems, so there's no
> > incompatibility introduced in the first place - everything still works
> > where it is supposed to, exactly as it was before.
>
> I think we're saying the same thing but quibbling over phrasing.  I'd put
> that as saying that it's fine for the binaries of certain core Debian
> packages to be incompatible with the ABI because they're not intended to
> be used outside of Debian.  (In other words, I'm talking about
> incompatibility as a concrete, testable property of a binary, and I think
> you're talking about incompatibility as a more abstract concept of a
> distribution.)
>
> > No aggression intended whatsoever, sorry if it appeared that way. I am
> > trying to understand what the rules are.
>
> Well, the rule that I'd ideally set is don't break the ABI, even if it's
> not obvious why breaking the ABI is a bad idea or you can't see any bad
> consequences that could come from it, unless the reason for breaking the
> ABI is absolutely central to the mission and purpose of Debian.
>
> That said, it's not like we've never shipped a binary in Debian with a
> different PT_INTERP.  (I vaguely remember that some programming language
> uses PT_INTERP tricks for some sort of private binary scheme?  Objective
> CAML or something?  I ran across it once years ago and can't remember the
> details.  Also, IIRC klibc does some sort of PT_INTERP trick in some
> situations that I don't remember the details of, although I don't think it
> does that with general binaries.)  So I do see your point that you would
> prefer the rule to be more pragmatic than that.
>
> My counterargument is that this proposal seems to mostly be about avoiding
> having to create a symlink at a critical point in the bootstrap process,
> and while it's tricky to get the timing right (and therefore kind of
> annoying), the resulting usable system has to have that symlink anyway (I
> think there's no disagreement about that).  Not following the ABI for core
> binaries seems like a scary change with unknown consquences to a bunch of
> core packages to solve what looks like a relatively minor (if admittedly
> annoying!) problem.
>
> Note that the target of PT_INTERP on Debian is *already* a symlink, to
> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
> that we actually want to use.  We're already ensuring compatibility with a
> symlink and I think we should just keep doing that and not venture into
> these waters.

It's not even a proposal, it's a discussion to see "what would
actually break if X happened". That's why I'm asking for concrete use
cases, rather than theoretical points. Also, I am interested in what
the rules are. For example, glibc compatibility is just as important
if not more, why does that follow a different standard? If anything,
adding a missing symlink is trivial and a single command. Adding a
missing symbol to a shared object... not quite. There is an endless
list of downstream-only debianisms that break cross-compatibility in
just the same way (if not worse), and yet nobody cares about those.
Why?

> >> The world does not divide neatly into supported and unsupported use
> >> cases.  There are a lot of things I do to computers that I expect to
> >> work in some situations but not in others.  That includes, say, having
> >> a Debian chroot on some other OS and running binaries from that chroot
> >> without going into the chroot.  Often that will absolutely not work.
> >> Sometimes it will work, and it's convenient that it will work for some
> >> obscure recovery situations or other weird one-off use cases.  I've
> >> also copied files from working systems to broken systems running a
> >> different distribution before, 

Standards compliance (Was: Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited))

[Jeroen Dekkers]

On Mon, 15 May 2023 02:42:27 +0200,
Luca Boccassi wrote:
>
> On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
> > An obvious specific example of such a system would be one that didn't
> > merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
> > path, but that's just one obvious example.  There may be others; the whole
> > point of an ABI is that you do not change things like this, not even if
> > you can't personally imagine why your change wouldn't be harmful.  There's
> > a whole process for changing an ABI that involves everyone else agreeing
> > as well, and unless one goes through that process, the ABI is what it is.
> > Debian not building ABI-compliant binaries would be highly surprising.
>
> That's self-evidently not true, as there are other distributions where
> that already happens, it's been already mentioned. Besides, we are not
> talking about sacred religious texts - the point is making things
> work. If they do, is it _really_ non-compliant/incompatible?

The point of a standard is definitely not that people ignore it and do things
that are in direct contradiction with the standard. The point of a standard is
that people agree on a specification and will also follow that specification.

Something working and something being compliant are different concepts. Software
can be compliant and not working and software can also be non-compliant and
working. Something that is in direct contradiction with a standard is definitely
non-compliant. Whether it works or not is not relevant for that.

Kind regards,

Jeroen Dekkers

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Mon, 15 May 2023 at 06:48:04 +0200, Johannes Schauer Marin Rodrigues wrote:
> Obviously, with Luca's proposal, binaries from packages built with a different
> dynamic linker path in them would not work on distributions without 
> merged-/usr
> symlinks. But if the property of stuff from Debian being able to run on
> non-Debian non-merged-/usr systems is an important one, then why was it okay 
> to
> have merged-/usr as the default? Because with merged-/usr we already changed
> the interface contract for a lot of things because now binaries and libraries
> can also be found at other locations than on non-merged-/usr systems. A script
> with a /usr/bin/bash shebang built on and for Debian will not work on a system
> without the symlinks.

pre-bookworm gcc writes /lib64/ld-linux-x86-64.so.2 into the ELF header
of amd64 binaries and I think post-bookworm gcc should continue to do so
even though that has never been the physical path to the ELF interpreter
on Debian (it's really /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, or
historically a version-numbered path). People who are only testing on
Debian systems, even in pre-merged-/usr releases like Debian 9, could
already have been relying on /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
existing; and now that we're saying merged-/usr is mandatory,
people who are only testing on Debian >= 12 could also start
relying on /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 or
/usr/lib64/ld-linux-x86-64.so.2; but we arrange for both/all paths
to exist, and we advise developers that they should only rely on the
interoperable /lib64/ld-linux-x86-64.so.2, treating all other paths that
lead to the same binary as an implementation detail.

I don't think it's any different to say that both /usr/bin/sh and /bin/sh
exist and will work, but ask that everyone should continue to use /bin/sh
and treat /usr/bin/sh as an implementation detail.

The way I think about merged /usr is a variant of Postel's principle:
be conservative in what you require, but be liberal about what will work.
A merged-/usr system can run software that assumes merged-/usr, and can
also run software that doesn't (because of the compatibility symlinks);
a non-merged-/usr system can run software that doesn't assume merged-/usr,
but software that makes that assumption will sometimes fail to work on it.

Consistent with that, despite being in favour of merged-/usr myself,
I didn't switch my development laptop or my autopkgtest VM images
to the merged-/usr setup until it became effectively mandatory
during the bookworm cycle - because if a package was doing something
not-maximally-portable, I wanted my development laptop or my test VM to
be one of the places it would fail to work, so that I would notice and
report that bug.

Conversely, I *did* switch non-development computers (servers and family
laptops) to merged-/usr, because on those systems the important thing is
for software to work, even if in theory it "doesn't deserve" to work.

On post-bookworm, merged-/usr Debian systems, if we keep using #!/bin/sh
and #!/usr/bin/perl scripts, and avoiding #!/usr/bin/sh and #!/bin/perl
scripts, then I think that's a positive thing for cross-distro interop -
and using the interoperable path to the ELF interpreter (dynamic linker)
is completely consistent with that.

As far as I can see, post-bookworm Lintian will continue to warn about
the non-interoperable spellings like #!/usr/bin/sh and #!/bin/perl,
unless changed to special-case them.

Note that the paths that are canonical from the point of view
of cross-distro interoperability (like #!/bin/sh) are not always
the same as the paths that are canonical from the point of view of
realpath(). *At the moment*, they are usually canonical from dpkg's
point of view, but that won't be the case forever, and I think that's
fine. /lib64/ld-linux-x86-64.so.2 isn't considered to be canonical by
either realpath() or dpkg either (dpkg knows it's a symlink, even on
non-merged-/usr systems where /lib64 is a real directory), but it's
canonical for the x86_64-linux-gnu ABI and that's what we say is most
important.

> With merged-/usr we already *did* "change fundamental things around" for
> reasons that are really not clear to me (but which i do not want to discuss
> here) and as a result did not "care about interoperability" (with those who do
> not also adopt it).

Didn't we? With merged /usr, the "theoretically correct" #!/bin/sh
scripts that have always worked will continue to work, and additionally,
"incorrect" #!/usr/bin/sh scripts will *also* work. That sounds like
caring about interoperability to me!

The one piece of interop we lose with merged-/usr becoming mandatory is
that if a developer has only tested on Debian (and other merged-/usr distros
like Fedora), if they have used a less-interoperable spelling of the path
like #!/usr/bin/sh or #!/bin/perl, their testing will not highlight that
source of potential non-interop with others.

However, I don't think it's credible to claim 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.

*raises hand*

Hello, I represent an example of those people. With my $work hat on,
I'm involved in maintaining a family of Debian derivatives (the Steam
Runtime) that is used to run Steam games on arbitrary distributions,
including but not limited to Debian. Some of these binaries are built
on a Debian derivative, but run on an arbitrary other distribution,
using a RPATH[1] to find their non-glibc dependencies.

At the moment those binaries are built in ancient environments (based
on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
ubiquitous, we'll want to start relying on newer versions of Debian
(which will still be very old versions *at the time*, but I'm sure that
by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
are not released yet). In this use-case, yes we do need to be using the
interoperable ELF interpreter path.

We were able to use (Ubuntu and) Debian for this *because* Debian is
a relatively "ordinary" distribution that tends to follow cross-distro
standards. The major counterexample is multiarch library paths, but
multiarch library paths have some compelling technical advantages, and
because there's no ambiguity about which architecture uses which directory,
they're actually better for interop in some ways than /usr/lib (which
is ambiguous, because the Red Hat family of distros expects to find 32-bit
libraries there, but the Arch family expects 64-bit libraries, and it's
not possible to construct a tree where both get what they want).

I've spent a lot of the last 5 years working on putting Steam games in
containers so that they'll work more reliably on all distros, including
Debian, with less reliance on weird library search paths (although we
still have to use binaries built on an ancient Debian derivative with a
non-trivial RPATH to bootstrap the container environment). Because of
constraints around recent GPUs needing current graphics drivers even
if running on an otherwise ancient library stack, Steam's container
runtime has to construct a hybrid environment where glibc is either the
one from the host or the one from the container runtime environment,
whichever is newer; and while doing that, we have experienced some
broken situations that were caused by distributions diverging from the
interoperable ELF interpreter. One concrete example is that Arch Linux
uses the interoperable ELF interpreter for *almost* all executables,
but uses a different ELF interpreter for executables from the glibc
source package, for whatever reason; we were able to work around this,
but for a while it caused Arch to fail to launch these containers where
Debian/Fedora/etc. could.

This is not something that any of the distributions involved is
going to formally support, so the overall system consists of various
unsupported-but-works-in-practice things happening; but there is
absolutely no chance of Valve/Steam shipping separate binaries for
each distro (there are too many distros for that, even if you don't
count source-based distros like Gentoo that have an almost unlimited
number of concrete instantiations as binaries). However loudly we
might insist that our small subset-of-a-subset is the one they should
be targeting, what we're going to get is binaries that work on "mostly
interoperable" distros. As a result, if we want games on Linux (and
anything else requiring binary interop) to continue to be a thing,
pragmatically we should aim to be one of those "mostly interoperable"
distros, and encourage our friends and/or rivals in other distros to
do the same. Otherwise, the most likely outcome is for developers to go
back to an attitude of "I'm not going to support Linux, too many random
differences" and releasing for Windows only, and we all lose out.

As a result of all this, I would strongly prefer our compiler to
default to hard-coding the same interoperable ELF interpreter that
glibc upstream and the various major distros have agreed on (for example
/lib64/ld-linux-x86-64.so.2 on amd64), and I would also prefer it if we
can use that interoperable path in all the binaries we ship, including
src:glibc and the rest of the transitively Essential set.

One way that I like to think about this sort of thing is that we have a
finite number of "weird Debianism" tokens available, and we should aim
to spend them on the things that give us the best cost/benefit ratio.
We've never considered changing the ELF interpreter to be one of those,
to the extent of having a /lib64 solely for its benefit (on amd64)
even though by policy we don't generally use lib64.

(Incidentally, Steam's container runtime is always a merged-/usr
environment, to provide an environment that maximizes the probability
of any given script or binary working correctly; but it 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:
> On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:

>> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
>> major player in Linux distributions, and I'm not sure how much they
>> care about compatibility with anyone else.)

> This is a counter-example to confute the assertion that *everybody* does
> the same thing, which has been made multiple times. I'm not sure whether
> it's an experiment or not, I mean if you ask their users/developers I
> think they'd tell you it's very much production ready, but it is largely
> irrelevant: it exists, and that's the only reason why it was mentioned,
> as it shows that it is _possible_ to do that and be a working
> distribution. Doesn't imply it's automatically desirable, but that was
> not the intention.

Ah, okay, I'm happy to agree with that point: you can violate the ABI and
continue to be a working distribution.  (There are a lot of parts of the
ABI that if you violated them you would not have a working distribution,
but this is not one of them so far as I can tell.)

> Not quite: my argument is that binaries from these packages are not
> intended and not required to be ran on non-Debian systems, so there's no
> incompatibility introduced in the first place - everything still works
> where it is supposed to, exactly as it was before.

I think we're saying the same thing but quibbling over phrasing.  I'd put
that as saying that it's fine for the binaries of certain core Debian
packages to be incompatible with the ABI because they're not intended to
be used outside of Debian.  (In other words, I'm talking about
incompatibility as a concrete, testable property of a binary, and I think
you're talking about incompatibility as a more abstract concept of a
distribution.)

> No aggression intended whatsoever, sorry if it appeared that way. I am
> trying to understand what the rules are.

Well, the rule that I'd ideally set is don't break the ABI, even if it's
not obvious why breaking the ABI is a bad idea or you can't see any bad
consequences that could come from it, unless the reason for breaking the
ABI is absolutely central to the mission and purpose of Debian.

That said, it's not like we've never shipped a binary in Debian with a
different PT_INTERP.  (I vaguely remember that some programming language
uses PT_INTERP tricks for some sort of private binary scheme?  Objective
CAML or something?  I ran across it once years ago and can't remember the
details.  Also, IIRC klibc does some sort of PT_INTERP trick in some
situations that I don't remember the details of, although I don't think it
does that with general binaries.)  So I do see your point that you would
prefer the rule to be more pragmatic than that.

My counterargument is that this proposal seems to mostly be about avoiding
having to create a symlink at a critical point in the bootstrap process,
and while it's tricky to get the timing right (and therefore kind of
annoying), the resulting usable system has to have that symlink anyway (I
think there's no disagreement about that).  Not following the ABI for core
binaries seems like a scary change with unknown consquences to a bunch of
core packages to solve what looks like a relatively minor (if admittedly
annoying!) problem.

Note that the target of PT_INTERP on Debian is *already* a symlink, to
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
that we actually want to use.  We're already ensuring compatibility with a
symlink and I think we should just keep doing that and not venture into
these waters.

>> The world does not divide neatly into supported and unsupported use
>> cases.  There are a lot of things I do to computers that I expect to
>> work in some situations but not in others.  That includes, say, having
>> a Debian chroot on some other OS and running binaries from that chroot
>> without going into the chroot.  Often that will absolutely not work.
>> Sometimes it will work, and it's convenient that it will work for some
>> obscure recovery situations or other weird one-off use cases.  I've
>> also copied files from working systems to broken systems running a
>> different distribution before, and there's a list of caveats as long as
>> my arm, but sometimes it's a quick fix for something.

> Vast, vast majority of binaries from existing packages will already
> not work out of the box in that use case though.

I'm not sure why you think this is true and it makes me wonder if maybe my
intuition about cross-distribution compatibility is wrong.  I would expect
to be able to copy, say, most (all?) binaries from coreutils from a Debian
system to some other distribution and run it (and that's exactly the sort
of binary that is useful in this kind of cross-distribution rescue case).
Is this not true today?  What breaks?

Note that we're not talking about complicated packages with lots of
runtime like, say, Emacs.  As I understand it your proposal wouldn't
change PT_INTERP for that binary anyway. 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 14:36, Steve McIntyre  wrote:
>
> Hey Johannes,
>
> On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues 
> wrote:
> >So did we not years ago decide, that the result of the "cross- and
> >inter-project discussion" is, that everybody is going merged-/usr and that's
> >why we need it too and that's why it is okay to build a system where binaries
> >and scripts built for it just may not run on those other systems that do not 
> >do
> >it?  With merged-/usr we already *did* "change fundamental things around" for
> >reasons that are really not clear to me (but which i do not want to discuss
> >here) and as a result did not "care about interoperability" (with those who 
> >do
> >not also adopt it). In my own Debian work I so far only got extra work 
> >because
> >of merged-/usr and I do not see the benefits (yet) and I was hoping that
> >"changing fundamental things around Linux and (basically) not caring about
> >interoperability" was *not* Debian's attitude but alas here we are.
> >
> >So have we not already burned the bridges to the non-merged-/usr world? Why 
> >was
> >it okay back then to say "we can make this change because all other important
> >players are doing merged-/usr so we can/have to as well". And now in the
> >PT_INTERP discussion somehow we care again about those systems? I thought we
> >already had the "cross- and inter-project discussion" about merged-/usr and
> >because the result was "yes, go for it" we did it too. But if that is the 
> >case,
> >why do we now care for a subset of the interoperability problems caused by
> >merged-/usr for systems that don't have it?
>
> This change is absolutely *not* needed to make merged-/usr work; if
> anybody is claiming that it is, then they are not being 100% honest
> with us. All the other distros doing merged-/usr have done it without
> making this change, and it's also been working OK for us so far
> without this change.

That is absolutely true, it is not mandatory. It is one possible
solution (of many) to a particular use case being sounded out, that's
all. I don't think it was mentioned by anybody as needed, if it was,
happy to clarify.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

Hey Johannes,

On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues 
wrote:
>Quoting Steve McIntyre (2023-05-15 02:54:02)
>> 
>> Pointing at gentoo or nixos as examples of projects that have decided
>> to break compatibility doesn't cut it, I'm afraid. They're well known
>> for changing fundamental things around Linux and (basically) not caring about
>> interoperability. That attitude is *not* Debian's.
>
>me and Luca have different ideas about how bootstrapping a Debian chroot should
>look like and I don't want to make an argument *for* changing PT_INTERP here as
>I think that keeping compatibility with others by following ABI is a good thing
>and because I think (and hope -- but Helmut is doing that analysis right now)
>that the debootstrap problem can be solved in a way I envision without changing
>PT_INTERP. But what I do not understand about the argument against Luca's
>proposal is:
>
>Obviously, with Luca's proposal, binaries from packages built with a different
>dynamic linker path in them would not work on distributions without merged-/usr
>symlinks. But if the property of stuff from Debian being able to run on
>non-Debian non-merged-/usr systems is an important one, then why was it okay to
>have merged-/usr as the default? Because with merged-/usr we already changed
>the interface contract for a lot of things because now binaries and libraries
>can also be found at other locations than on non-merged-/usr systems. A script
>with a /usr/bin/bash shebang built on and for Debian will not work on a system
>without the symlinks.

Despite the massive upheavals of merged-/usr in *other* ways, it's
actually a *minor* change as far as compatibility is concerned
here. The runtime linker (aka interpreter) is responsible for
resolving symbols and finding needed libraries. So long as *that* bit
works OK, then ~everything else should follow OK. This is how it's
possible to have things work across distros: binaries don't actually
care where the libraries live, or whether they came from rpm or deb
packaging, etc.

The issue at hand here is that the interpreter path is the most basic
thing that matters for this compatibility. Break this and *nothing*
can work.

>So did we not years ago decide, that the result of the "cross- and
>inter-project discussion" is, that everybody is going merged-/usr and that's
>why we need it too and that's why it is okay to build a system where binaries
>and scripts built for it just may not run on those other systems that do not do
>it?  With merged-/usr we already *did* "change fundamental things around" for
>reasons that are really not clear to me (but which i do not want to discuss
>here) and as a result did not "care about interoperability" (with those who do
>not also adopt it). In my own Debian work I so far only got extra work because
>of merged-/usr and I do not see the benefits (yet) and I was hoping that
>"changing fundamental things around Linux and (basically) not caring about
>interoperability" was *not* Debian's attitude but alas here we are.
>
>So have we not already burned the bridges to the non-merged-/usr world? Why was
>it okay back then to say "we can make this change because all other important
>players are doing merged-/usr so we can/have to as well". And now in the
>PT_INTERP discussion somehow we care again about those systems? I thought we
>already had the "cross- and inter-project discussion" about merged-/usr and
>because the result was "yes, go for it" we did it too. But if that is the case,
>why do we now care for a subset of the interoperability problems caused by
>merged-/usr for systems that don't have it?

This change is absolutely *not* needed to make merged-/usr work; if
anybody is claiming that it is, then they are not being 100% honest
with us. All the other distros doing merged-/usr have done it without
making this change, and it's also been working OK for us so far
without this change.

Breaking an agreed interface contract like this is axiomatically
*wrong* and will hurt us and the rest of the Linux ecosystem.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"You can't barbecue lettuce!" -- Ellie Crane

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Steve McIntyre (2023-05-15 02:54:02)
> On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> >On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
> >
> >> The x86-64 ABI is set. Feel free to make the case to the next
> >> architecture designer that their new ABI should have the dynamic linker
> >> in `/usr/lib`. That would *not* have the same downsides, as long as
> >> everyone agrees on a path.
> >
> >In practice it is not, though. There are other distributions that
> >change PT_INTERP for their own purposes, they've already been listed
> >in this thread. And I am still not hearing any concrete, factual use
> >case that would be impaired by such a change. I'm beginning to
> >seriously think there aren't any? Is that really the case?
> 
> The ABI has been agreed and set down in documentation that *just
> about* everybody has been following since its inception. This includes
> the most basic set of definitions of what an x86-64 program must look
> like, including the interpreter path. If this path is changed, then
> *at the most basic level* we'd be making programs that are not valid
> by the ABI we've agreed to. This is an *external interface contract*,
> not something we should ever consider changing without significant
> cross- and inter-project discussion.
> 
> Pointing at gentoo or nixos as examples of projects that have decided
> to break compatibility doesn't cut it, I'm afraid. They're well known
> for changing fundamental things around Linux and (basically) not caring about
> interoperability. That attitude is *not* Debian's.

me and Luca have different ideas about how bootstrapping a Debian chroot should
look like and I don't want to make an argument *for* changing PT_INTERP here as
I think that keeping compatibility with others by following ABI is a good thing
and because I think (and hope -- but Helmut is doing that analysis right now)
that the debootstrap problem can be solved in a way I envision without changing
PT_INTERP. But what I do not understand about the argument against Luca's
proposal is:

Obviously, with Luca's proposal, binaries from packages built with a different
dynamic linker path in them would not work on distributions without merged-/usr
symlinks. But if the property of stuff from Debian being able to run on
non-Debian non-merged-/usr systems is an important one, then why was it okay to
have merged-/usr as the default? Because with merged-/usr we already changed
the interface contract for a lot of things because now binaries and libraries
can also be found at other locations than on non-merged-/usr systems. A script
with a /usr/bin/bash shebang built on and for Debian will not work on a system
without the symlinks.

So did we not years ago decide, that the result of the "cross- and
inter-project discussion" is, that everybody is going merged-/usr and that's
why we need it too and that's why it is okay to build a system where binaries
and scripts built for it just may not run on those other systems that do not do
it?  With merged-/usr we already *did* "change fundamental things around" for
reasons that are really not clear to me (but which i do not want to discuss
here) and as a result did not "care about interoperability" (with those who do
not also adopt it). In my own Debian work I so far only got extra work because
of merged-/usr and I do not see the benefits (yet) and I was hoping that
"changing fundamental things around Linux and (basically) not caring about
interoperability" was *not* Debian's attitude but alas here we are.

So have we not already burned the bridges to the non-merged-/usr world? Why was
it okay back then to say "we can make this change because all other important
players are doing merged-/usr so we can/have to as well". And now in the
PT_INTERP discussion somehow we care again about those systems? I thought we
already had the "cross- and inter-project discussion" about merged-/usr and
because the result was "yes, go for it" we did it too. But if that is the case,
why do we now care for a subset of the interoperability problems caused by
merged-/usr for systems that don't have it?

As I said, I don't care much about the PT_INTERP value but I don't understand
yet, why this argument about interoperability with non-merged-/usr systems is
working now but it didn't wasn't enough to stop another very fundamental change
in how we build a Linux distro.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > That's self-evidently not true, as there are other distributions where
> > that already happens, it's been already mentioned.
>
> You've mentioned this a couple of times but I don't think I've seen the
> message where the details were explained.  Maybe this was only in your
> message posted to debian-gcc, which wasn't part of this thread?  (It's
> also possible that I just missed it somewhere.)
>
> That message only mentions GUIX, which I don't know very much about, but
> my recollection (maybe wrong?) is that it's a NIX variant that is doing
> special tricks to support immutable package trees and
> roll-forward/roll-back upgrades.  I can see why that might be motivation
> to build incompatible binaries in order to preserve some other invariant
> they're trying for as the point of their distribution (in particular, I
> suspect they're pinning binaries to a specific version of the dynamic
> loader as part of the whole immutable tree strategy).  That's a perfectly
> fine decision in a distribution that's trying to do something very
> different and is a bit of a science experiment, but I don't think that
> describes Debian.
>
> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, major
> player in Linux distributions, and I'm not sure how much they care about
> compatibility with anyone else.)

This is a counter-example to confute the assertion that *everybody*
does the same thing, which has been made multiple times. I'm not sure
whether it's an experiment or not, I mean if you ask their
users/developers I think they'd tell you it's very much production
ready, but it is largely irrelevant: it exists, and that's the only
reason why it was mentioned, as it shows that it is _possible_ to do
that and be a working distribution. Doesn't imply it's automatically
desirable, but that was not the intention.

> > Besides, we are not talking about sacred religious texts - the point is
> > making things work. If they do, is it _really_
> > non-compliant/incompatible?
>
> I understand your point in making this argument, but please understand
> that this sort of willingness to change things if one didn't think they
> would cause problems didn't work very well, and was part of what led to
> the development of standardized ABIs in the first place.  Those of us who
> have been around longer than Linux have ABIs have a bit of a strong
> reaction here (I think this is also what you're seeing from Steve),
> because we remember the bad old days.  I still have compatibility code
> around to handle the fact that gcc on IRIX miscompiled calls to inet_ntoa
> because gcc didn't correctly implement the IRIX ABI.
>
> People are very bad at judging whether their new idea would be *really*
> incompatible.  This is why these days everyone tries to follow the ABI
> pretty closely.
>
> And in any case, changing PT_INTERP is trivially and obviously
> incompatible; the binary will simply not run on a system that doesn't have
> that path.  So it's not like we have to carefully judge nuance here.  Your
> argument, so far as I can tell, is basically "but no one will ever want to
> run those binaries on a non-/usr-merged system anyway," which is basically
> conceding the incompatibility point since the ABI doesn't require merged
> /usr.

Not quite: my argument is that binaries from these packages are not
intended and not required to be ran on non-Debian systems, so there's
no incompatibility introduced in the first place - everything still
works where it is supposed to, exactly as it was before.

> There's also some other history here: Debian is not super-happy with the
> PT_INTERP because ideally we'd prefer it use a path compatible with our
> multiarch approach.  I believe we raised that and no one had any interest
> in trying to change anything, so we lived with the limitations that
> creates.  (And I think that was the right decision.)
>
> > Thanks, that's the first actual real example mentioned so far. And it's
> > an interesting one: taking a $random Debian package and using it on a
> > completely different, non-Debian system. Is that a supported use case?
> > If so, does that mean that I can go ahead and raise a Severity: serious
> > bug on any package that doesn't work in such a way?
>
> I feel like you're distorting my argument here to try to make some sort of
> slippery slope argument, and it's coming across as possibly more
> aggressive than you had intended.

No aggression intended whatsoever, sorry if it appeared that way. I am
trying to understand what the rules are.

> The world does not divide neatly into supported and unsupported use cases.
> There are a lot of things I do to computers that I expect to work in some
> situations but not in others.  That includes, say, having a Debian chroot
> on some other OS and running binaries from that chroot without going into
> the chroot.  Often that will absolutely not work.  Sometimes it will 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:

> That's self-evidently not true, as there are other distributions where
> that already happens, it's been already mentioned.

You've mentioned this a couple of times but I don't think I've seen the
message where the details were explained.  Maybe this was only in your
message posted to debian-gcc, which wasn't part of this thread?  (It's
also possible that I just missed it somewhere.)

That message only mentions GUIX, which I don't know very much about, but
my recollection (maybe wrong?) is that it's a NIX variant that is doing
special tricks to support immutable package trees and
roll-forward/roll-back upgrades.  I can see why that might be motivation
to build incompatible binaries in order to preserve some other invariant
they're trying for as the point of their distribution (in particular, I
suspect they're pinning binaries to a specific version of the dynamic
loader as part of the whole immutable tree strategy).  That's a perfectly
fine decision in a distribution that's trying to do something very
different and is a bit of a science experiment, but I don't think that
describes Debian.

(Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, major
player in Linux distributions, and I'm not sure how much they care about
compatibility with anyone else.)

> Besides, we are not talking about sacred religious texts - the point is
> making things work. If they do, is it _really_
> non-compliant/incompatible?

I understand your point in making this argument, but please understand
that this sort of willingness to change things if one didn't think they
would cause problems didn't work very well, and was part of what led to
the development of standardized ABIs in the first place.  Those of us who
have been around longer than Linux have ABIs have a bit of a strong
reaction here (I think this is also what you're seeing from Steve),
because we remember the bad old days.  I still have compatibility code
around to handle the fact that gcc on IRIX miscompiled calls to inet_ntoa
because gcc didn't correctly implement the IRIX ABI.

People are very bad at judging whether their new idea would be *really*
incompatible.  This is why these days everyone tries to follow the ABI
pretty closely.

And in any case, changing PT_INTERP is trivially and obviously
incompatible; the binary will simply not run on a system that doesn't have
that path.  So it's not like we have to carefully judge nuance here.  Your
argument, so far as I can tell, is basically "but no one will ever want to
run those binaries on a non-/usr-merged system anyway," which is basically
conceding the incompatibility point since the ABI doesn't require merged
/usr.

There's also some other history here: Debian is not super-happy with the
PT_INTERP because ideally we'd prefer it use a path compatible with our
multiarch approach.  I believe we raised that and no one had any interest
in trying to change anything, so we lived with the limitations that
creates.  (And I think that was the right decision.)

> Thanks, that's the first actual real example mentioned so far. And it's
> an interesting one: taking a $random Debian package and using it on a
> completely different, non-Debian system. Is that a supported use case?
> If so, does that mean that I can go ahead and raise a Severity: serious
> bug on any package that doesn't work in such a way?

I feel like you're distorting my argument here to try to make some sort of
slippery slope argument, and it's coming across as possibly more
aggressive than you had intended.

The world does not divide neatly into supported and unsupported use cases.
There are a lot of things I do to computers that I expect to work in some
situations but not in others.  That includes, say, having a Debian chroot
on some other OS and running binaries from that chroot without going into
the chroot.  Often that will absolutely not work.  Sometimes it will work,
and it's convenient that it will work for some obscure recovery situations
or other weird one-off use cases.  I've also copied files from working
systems to broken systems running a different distribution before, and
there's a list of caveats as long as my arm, but sometimes it's a quick
fix for something.

But mostly my reaction is because breaking the ABI is a Really Big Deal.
Constructing the Linux ABI and getting the details actually published was
a hard-fought, arduous endeavor.  I doubt anyone enjoyed it; it's the sort
of annoying compatibility work that provides tons of small, subtle
benefits and takes a great deal of truly thankless work, and people often
don't realize all the tiny ways that it has made the world a better place,
or the range of weird compatibility problems that can arise from messing
with it.  Diverging from it is not something to do lightly, precisely
*because* it's often extremely difficult to understand what the effects
could be or what might break.

While I appreciate how it would make bootstrapping Debian somewhat more

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

I'm *trying* to assume good faith here, but I'm running out of energy
to do so.

On Mon, May 15, 2023 at 01:42:27AM +0100, Luca Boccassi wrote:
>On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
>> Incidentally, that remains true even if we only do that in distribution
>> packages.  I certainly have copied binaries from a Debian package to other
>> Linux systems before for various reasons and expected them to run.  Sure,
>> this might not work for other reasons outside of our control, but that's
>> no reason to be gratuitously incompatible by breaking the ABI,
>> particularly for what seem to be annoyances of our own creation with known
>> workarounds.
>
>Thanks, that's the first actual real example mentioned so far. And
>it's an interesting one: taking a $random Debian package and using it
>on a completely different, non-Debian system. Is that a supported use
>case? If so, does that mean that I can go ahead and raise a Severity:
>serious bug on any package that doesn't work in such a way?

Russ has described copying *binaries* out of packages and running them
elsewhere. I've done that too, from time to time. This is one of the
things made possible by the ABI contract being followed.

You are the one proposing to break that contract, thereby
*guaranteeing* this will fail on systems where otherwise it could
work. I think the onus is on *you* to justify why this is a valid and
useful thing to do. Your apparent lack of care for agreed standards
here is horrifying.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"We're the technical experts.  We were hired so that management could
 ignore our recommendations and tell us how to do our jobs."  -- Mike Andrews

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
>On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
>
>> The x86-64 ABI is set. Feel free to make the case to the next
>> architecture designer that their new ABI should have the dynamic linker
>> in `/usr/lib`. That would *not* have the same downsides, as long as
>> everyone agrees on a path.
>
>In practice it is not, though. There are other distributions that
>change PT_INTERP for their own purposes, they've already been listed
>in this thread. And I am still not hearing any concrete, factual use
>case that would be impaired by such a change. I'm beginning to
>seriously think there aren't any? Is that really the case?

The ABI has been agreed and set down in documentation that *just
about* everybody has been following since its inception. This includes
the most basic set of definitions of what an x86-64 program must look
like, including the interpreter path. If this path is changed, then
*at the most basic level* we'd be making programs that are not valid
by the ABI we've agreed to. This is an *external interface contract*,
not something we should ever consider changing without significant
cross- and inter-project discussion.

Pointing at gentoo or nixos as examples of projects that have decided
to break compatibility doesn't cut it, I'm afraid. They're well known
for changing fundamental things around Linux and (basically) not
caring about interoperability. That attitude is *not* Debian's.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"The problem with defending the purity of the English language is that
 English is about as pure as a cribhouse whore. We don't just borrow words; on
 occasion, English has pursued other languages down alleyways to beat them
 unconscious and rifle their pockets for new vocabulary."  -- James D. Nicoll

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> My understanding is that this specific thread is about a mention that we
> may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some
> similar path.
>
> If PT_INTERP points to a file that doesn't exist, the program is obviously
> not going to run.  The Linux x86_64 ABI says it must point to
> /lib64/ld-linux-x86-64.so.2.  If we build binaries that use some other
> value, then we are not building ABI-compliant binaries and they may not
> run on other systems.  This is the whole point of an ABI.

This is not about locally compiled software or such, only packages
(and maybe even just a subset of them).

> An obvious specific example of such a system would be one that didn't
> merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
> path, but that's just one obvious example.  There may be others; the whole
> point of an ABI is that you do not change things like this, not even if
> you can't personally imagine why your change wouldn't be harmful.  There's
> a whole process for changing an ABI that involves everyone else agreeing
> as well, and unless one goes through that process, the ABI is what it is.
> Debian not building ABI-compliant binaries would be highly surprising.

That's self-evidently not true, as there are other distributions where
that already happens, it's been already mentioned. Besides, we are not
talking about sacred religious texts - the point is making things
work. If they do, is it _really_ non-compliant/incompatible?

> Incidentally, that remains true even if we only do that in distribution
> packages.  I certainly have copied binaries from a Debian package to other
> Linux systems before for various reasons and expected them to run.  Sure,
> this might not work for other reasons outside of our control, but that's
> no reason to be gratuitously incompatible by breaking the ABI,
> particularly for what seem to be annoyances of our own creation with known
> workarounds.

Thanks, that's the first actual real example mentioned so far. And
it's an interesting one: taking a $random Debian package and using it
on a completely different, non-Debian system. Is that a supported use
case? If so, does that mean that I can go ahead and raise a Severity:
serious bug on any package that doesn't work in such a way?

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 01:07, Peter Pentchev  wrote:
>
> On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> > On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
> > >
> > > On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > > > The loader is still available via the old path, so external/third
> > > > party/local/other software works unchanged. This should negatively
> > > > only affect our 1st party packages, when running on a non-merged
> > > > distro.
> > > > And are _all_ our packages really 100% compatible with other distros
> > > > at all? Are they even supposed to be?
> > >
> > > People build things on Debian that are not Debian packages. People
> > > compile binaries on Debian, and expect them to work on any system that
> > > has sufficiently new libraries.
> > >
> > > This is *not* about Debian packages failing to work on other
> > > distributions; this is about *software compiled on Debian* faliing to
> > > work in other environments.
> >
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> If an ELF executable, compiled on Debian, records its interpreter as
> /usr/lib/ld-linux.so.2, what happens when one tries to run it on
> a non-usr-merged system? Even one with a recent enough glibc version?

This is not about locally built ELF executables, no difference in those.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:

> Why would "software compiled on Debian" fail to work in other
> environments? Well, there are many reasons actually, people invented
> containers/flatpaks/snaps exactly for that reason. But nothing do with
> anything discussed here though, as far as I can tell?

My understanding is that this specific thread is about a mention that we
may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some
similar path.

If PT_INTERP points to a file that doesn't exist, the program is obviously
not going to run.  The Linux x86_64 ABI says it must point to
/lib64/ld-linux-x86-64.so.2.  If we build binaries that use some other
value, then we are not building ABI-compliant binaries and they may not
run on other systems.  This is the whole point of an ABI.

An obvious specific example of such a system would be one that didn't
merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
path, but that's just one obvious example.  There may be others; the whole
point of an ABI is that you do not change things like this, not even if
you can't personally imagine why your change wouldn't be harmful.  There's
a whole process for changing an ABI that involves everyone else agreeing
as well, and unless one goes through that process, the ABI is what it is.
Debian not building ABI-compliant binaries would be highly surprising.

Incidentally, that remains true even if we only do that in distribution
packages.  I certainly have copied binaries from a Debian package to other
Linux systems before for various reasons and expected them to run.  Sure,
this might not work for other reasons outside of our control, but that's
no reason to be gratuitously incompatible by breaking the ABI,
particularly for what seem to be annoyances of our own creation with known
workarounds.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Peter Pentchev]

On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
> >
> > On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > > The loader is still available via the old path, so external/third
> > > party/local/other software works unchanged. This should negatively
> > > only affect our 1st party packages, when running on a non-merged
> > > distro.
> > > And are _all_ our packages really 100% compatible with other distros
> > > at all? Are they even supposed to be?
> >
> > People build things on Debian that are not Debian packages. People
> > compile binaries on Debian, and expect them to work on any system that
> > has sufficiently new libraries.
> >
> > This is *not* about Debian packages failing to work on other
> > distributions; this is about *software compiled on Debian* faliing to
> > work in other environments.
> 
> Why would "software compiled on Debian" fail to work in other
> environments? Well, there are many reasons actually, people invented
> containers/flatpaks/snaps exactly for that reason. But nothing do with
> anything discussed here though, as far as I can tell?

If an ELF executable, compiled on Debian, records its interpreter as
/usr/lib/ld-linux.so.2, what happens when one tries to run it on
a non-usr-merged system? Even one with a recent enough glibc version?

G'luck,
Peter
 
-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
>
> On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > The loader is still available via the old path, so external/third
> > party/local/other software works unchanged. This should negatively
> > only affect our 1st party packages, when running on a non-merged
> > distro.
> > And are _all_ our packages really 100% compatible with other distros
> > at all? Are they even supposed to be?
>
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.
>
> This is *not* about Debian packages failing to work on other
> distributions; this is about *software compiled on Debian* faliing to
> work in other environments.

Why would "software compiled on Debian" fail to work in other
environments? Well, there are many reasons actually, people invented
containers/flatpaks/snaps exactly for that reason. But nothing do with
anything discussed here though, as far as I can tell?

> If you build a dynamically linked binary that only depends on glibc, you
> can expect it to be reasonably portable, to any system that uses glibc
> and has a sufficiently new version.
>
> Debian stable is, in fact, one of the common environments people use to
> compile binaries for distribution.

"sufficiently new version" is doing a lot of work there. We have
shlibs dependencies for a reason. In fact, the most common environment
used to distribute binaries is the EOL Ubuntu 16.04, slowly switching
to the soon-to-be-EOL 18.04. glibc is not forward-compatible, new
symbols are added all the time and are used all the time, and you
don't jump on a brand new distribution to do that kind of work, that
would be self-defeating.

> > So, what I am asking is, what actual, real difference does it make if,
> > by default (and with an override available for example), packages
> > built on Debian for Debian record the ld path to point to its (actual)
> > location on Debian, via say a compiler spec file that is injected in a
> > deb build?
>
> Making binaries built *on* Debian different than binaries built *for*
> Debian would introduce a needless additional source of complexity,
> compared to just compiling code the same way in both cases.

That's not how it works today already. There are several significant
differences between just running "gcc sources.c" and building a
package via debhelper on a buildd, they are not the same thing at all,
and they haven't been since forever, there are dozens of
compiler/linker options that the Debian package build environment
sets. Or will you now also ask the distribution to rollback multiarch,
hardening, SOURCE_DATE_EPOCH, -ffile-prefix-map and all the other
reproducibility options, and so on? These and many more are all
"needless additional sources of complexity, compared to just compiling
code the same way" too. Because guess what, there are people who
couldn't possibly care less about
multiarch/security/reproducibility/etc, and there will also be a
subset of users who considers a subset of those compiler options
"needless". So are you going to push to have all of that reverted? And
also are you going to propose a Policy change that forbids adding any
new compiler/linker option to the package build process?

> To frame this in different terms: consider that one of the major goals
> of systemd has been to harmonize across distributions and eliminate
> needless variations that don't serve much actual purpose (e.g.
> variations in config file paths for the same config file). Consider how
> much effort systemd went to work with distributions, understand and deal
> with the *important* variations, and try to convince them to abandon the
> *unimportant* variations. Now imagine if someone came along and said
> "let's patch systemd to put unit files in /purple/; it'll work with
> everything in our distribution".

Pretty sure the Nix folks are already doing pretty much that. And if
it works for their case, all the power to them.

> Or, imagine if someone said "let's inject an argument to gzip, only for
> building the .gz files sihpped in our packages of course, to modify the
> gzip header and remove a few of the extraneous additional fields; it'll
> be fine, because we've patched our gzip to parse it"

Not really related, archives are _intended_ to be opened anywhere for
any reason. Do you have any actual related use case that would no
longer work? Because that would be the easiest and most convincing
counter-factual that could be provided.

> The x86-64 ABI is set. Feel free to make the case to the next
> architecture designer that their new ABI should have the dynamic linker
> in `/usr/lib`. That would *not* have the same downsides, as long as
> everyone agrees on a path.

In practice it is not, though. There are other distributions that
change PT_INTERP for their own purposes, they've already been listed
in this thread. And I am still not 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Josh Triplett]

On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> The loader is still available via the old path, so external/third
> party/local/other software works unchanged. This should negatively
> only affect our 1st party packages, when running on a non-merged
> distro.
> And are _all_ our packages really 100% compatible with other distros
> at all? Are they even supposed to be?

People build things on Debian that are not Debian packages. People
compile binaries on Debian, and expect them to work on any system that
has sufficiently new libraries.

This is *not* about Debian packages failing to work on other
distributions; this is about *software compiled on Debian* faliing to
work in other environments.

If you build a dynamically linked binary that only depends on glibc, you
can expect it to be reasonably portable, to any system that uses glibc
and has a sufficiently new version.

Debian stable is, in fact, one of the common environments people use to
compile binaries for distribution.

> So, what I am asking is, what actual, real difference does it make if,
> by default (and with an override available for example), packages
> built on Debian for Debian record the ld path to point to its (actual)
> location on Debian, via say a compiler spec file that is injected in a
> deb build?

Making binaries built *on* Debian different than binaries built *for*
Debian would introduce a needless additional source of complexity,
compared to just compiling code the same way in both cases.

To frame this in different terms: consider that one of the major goals
of systemd has been to harmonize across distributions and eliminate
needless variations that don't serve much actual purpose (e.g.
variations in config file paths for the same config file). Consider how
much effort systemd went to work with distributions, understand and deal
with the *important* variations, and try to convince them to abandon the
*unimportant* variations. Now imagine if someone came along and said
"let's patch systemd to put unit files in /purple/; it'll work with
everything in our distribution".

Or, imagine if someone said "let's inject an argument to gzip, only for
building the .gz files sihpped in our packages of course, to modify the
gzip header and remove a few of the extraneous additional fields; it'll
be fine, because we've patched our gzip to parse it"

The x86-64 ABI is set. Feel free to make the case to the next
architecture designer that their new ABI should have the dynamic linker
in `/usr/lib`. That would *not* have the same downsides, as long as
everyone agrees on a path.

- Josh Triplett

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Andreas Metzler]

On 2023-05-12 Ansgar  wrote:
[...]
> The core issue as I see it is as follows:
[...]
> Do you think this summary of the issue is right?

I think Simon's reading of the situation as posted in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035904#30
makes a lot of sense.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 15:30, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> >On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
> >>
> >> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
> >> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >> >>>
> >> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >> >>> >
> >> >>> >The core issue as I see it is as follows:
> >> >>> >
> >> >>> >- Debian has decided to support only merged-/usr, including possibly
> >> >>> >  moving /bin/sh to /usr/bin/sh or using 
> >> >>> > /usr/lib*/ld-linux-x86-64.so.2
> >> >>> >  as the interpreter in binaries.
> >> >>>
> >> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >> >>> seen. The interpreter must *not* be changed willy-nilly.
> >> >>
> >> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >> >>seemingly crazy options, as in, "what would _actually_ explode if we
> >> >>do this or do that?", on this very d-devel thread. I posted a longer
> >> >>version here some days ago:
> >> >>
> >> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
> >> >
> >> >Oh holy fuck.
> >> >
> >> >You're talking about changing ABI by doing this. That *is* utterly
> >> >crazy. No.
> >>
> >> People have asked me to expand on this further...
> >>
> >> I've been involved in defining ABI before, specifically for
> >> armhf. It's not a quick and easy process. It needs buy-in from all
> >> sides to make things work, and people *really* value the
> >> interoperability that it enables.
> >>
> >> The interpreter path is one of the most important parts of the ABI
> >> spec, the bit that makes binaries compatible between all the various
> >> stakeholders: compiler/tools people, distros, software vendors,
> >> etc. Lots of the rest of the details downstream of this can be
> >> changed, and people do this all the time - compare multilib to
> >> multi-arch for example. That all works fine *so long as* the runtime
> >> linker can be located and started OK.
> >
> >The loader is still available via the old path, so external/third
> >party/local/other software works unchanged. This should negatively
> >only affect our 1st party packages, when running on a non-merged
> >distro.
>
> So why the hell do you want to break this in the first place? Does a
> symlink in the "wrong" place offend you for some reason? For that you
> want to change a core assumption in *every single binary* in Debian?
> Believe me, I've been here in the past when we made changes in armhf
> to accommodate earlier mistakes. That was just for one
> architecture. What possible benefit do you see in this change?

As it was mentioned on the list, because it makes bootstrapping
self-contained, that's a real and concrete benefit that some
developers like Helmut care greatly about, and that's why we are
talking about it. To me, it sounds very attractive to have a
self-contained and canonicalized distro-wide configuration. If the
canonical location where certain files are stored in /usr/bin or
/usr/lib, it seems sensible to me to configure Debian software to look
for it where we actually put it, while maintaining compatibility for
external/local software so that it keeps working. And it is also
unclear so far what would outright break - the externally defined ABI
in terms of where the loader can be accessed at, would still be
respected. Hence why questions are being asked. Nobody's being forced
to do anything, this is just a discussion.

> >And are _all_ our packages really 100% compatible with other distros
> >at all? Are they even supposed to be?
> >
> >For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
> >machine, when I try to run it, it fails:
> >
> >root@focal:/tmp# wget
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >--2023-05-12 12:46:17--
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
> >Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... 
> >connected.
> >HTTP request sent, awaiting response... 200 OK
> >Length: 27572 (27K) [application/vnd.debian.binary-package]
> >Saving to: 'efibootmgr_17-2_amd64.deb'
> >
> >efibootmgr_17-2_amd64.deb
> >100%[===>]  26.93K
> >--.-KB/sin 0.04s
> >
> >2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved 
> >[27572/27572]
> >
> >root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
> >root@focal:/tmp# ./ebm/bin/efibootmgr
> >./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
> >`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)
> >
> >Should I file a severity: serious bug against efibootmgr because it is
> >not interoperable?
>
> You're wilfully missing the point, and you know it.

I'm trying to determine 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Holger Levsen]

On Fri, May 12, 2023 at 03:29:29PM +0100, Steve McIntyre wrote:
> >> >Oh holy fuck.
> So why the hell do you want to break this in the first place? 
> You're wilfully missing the point, and you know it.
> I have better things to do than argue about this. I refuse to engage
> with this right now. You're talking about breaking things for *no*
> discernible benefit that I've seen any discussion about.
 
language please. and also assume good faith.

thanks.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Das Leben ist schön. Von 'einfach' war nie die Rede. (@lernzyklus)


signature.asc
Description: PGP signature

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
>On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
>>
>> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
>> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
>> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>> >>>
>> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>> >>> >
>> >>> >The core issue as I see it is as follows:
>> >>> >
>> >>> >- Debian has decided to support only merged-/usr, including possibly
>> >>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>> >>> >  as the interpreter in binaries.
>> >>>
>> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
>> >>> seen. The interpreter must *not* be changed willy-nilly.
>> >>
>> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
>> >>seemingly crazy options, as in, "what would _actually_ explode if we
>> >>do this or do that?", on this very d-devel thread. I posted a longer
>> >>version here some days ago:
>> >>
>> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>> >
>> >Oh holy fuck.
>> >
>> >You're talking about changing ABI by doing this. That *is* utterly
>> >crazy. No.
>>
>> People have asked me to expand on this further...
>>
>> I've been involved in defining ABI before, specifically for
>> armhf. It's not a quick and easy process. It needs buy-in from all
>> sides to make things work, and people *really* value the
>> interoperability that it enables.
>>
>> The interpreter path is one of the most important parts of the ABI
>> spec, the bit that makes binaries compatible between all the various
>> stakeholders: compiler/tools people, distros, software vendors,
>> etc. Lots of the rest of the details downstream of this can be
>> changed, and people do this all the time - compare multilib to
>> multi-arch for example. That all works fine *so long as* the runtime
>> linker can be located and started OK.
>
>The loader is still available via the old path, so external/third
>party/local/other software works unchanged. This should negatively
>only affect our 1st party packages, when running on a non-merged
>distro.

So why the hell do you want to break this in the first place? Does a
symlink in the "wrong" place offend you for some reason? For that you
want to change a core assumption in *every single binary* in Debian?
Believe me, I've been here in the past when we made changes in armhf
to accommodate earlier mistakes. That was just for one
architecture. What possible benefit do you see in this change?

>And are _all_ our packages really 100% compatible with other distros
>at all? Are they even supposed to be?
>
>For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
>machine, when I try to run it, it fails:
>
>root@focal:/tmp# wget
>http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
>--2023-05-12 12:46:17--
>http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
>Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
>Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... 
>connected.
>HTTP request sent, awaiting response... 200 OK
>Length: 27572 (27K) [application/vnd.debian.binary-package]
>Saving to: 'efibootmgr_17-2_amd64.deb'
>
>efibootmgr_17-2_amd64.deb
>100%[===>]  26.93K
>--.-KB/sin 0.04s
>
>2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved 
>[27572/27572]
>
>root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
>root@focal:/tmp# ./ebm/bin/efibootmgr
>./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
>`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)
>
>Should I file a severity: serious bug against efibootmgr because it is
>not interoperable?

You're wilfully missing the point, and you know it.

>The answer is obviously not, because it would be absurd to expect a
>binary compiled against libraries from one distro to "just work" on an
>entirely different distro. Glibc itself is not forward compatible, and
>is allowed to add new symbols, that are not present in older versions,
>and packages are allowed to depend on them. Aren't those also ABI
>breakages? What about all the libraries that bump soname? What about
>binaries that rely on newer kernel interfaces, or IPC interfaces?
>
>So, what I am asking is, what actual, real difference does it make if,
>by default (and with an override available for example), packages
>built on Debian for Debian record the ld path to point to its (actual)
>location on Debian, via say a compiler spec file that is injected in a
>deb build?
>There very likely is some real difference and impact, and I am
>genuinely and honestly asking what it could be. If nothing else, it's
>an interesting topic, even if likely nothing comes out of it.

I have better things to do than argue about this. I refuse to engage
with this right now. You're talking about 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >>>
> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >>> >
> >>> >The core issue as I see it is as follows:
> >>> >
> >>> >- Debian has decided to support only merged-/usr, including possibly
> >>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >>> >  as the interpreter in binaries.
> >>>
> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >>> seen. The interpreter must *not* be changed willy-nilly.
> >>
> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >>seemingly crazy options, as in, "what would _actually_ explode if we
> >>do this or do that?", on this very d-devel thread. I posted a longer
> >>version here some days ago:
> >>
> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
> >
> >Oh holy fuck.
> >
> >You're talking about changing ABI by doing this. That *is* utterly
> >crazy. No.
>
> People have asked me to expand on this further...
>
> I've been involved in defining ABI before, specifically for
> armhf. It's not a quick and easy process. It needs buy-in from all
> sides to make things work, and people *really* value the
> interoperability that it enables.
>
> The interpreter path is one of the most important parts of the ABI
> spec, the bit that makes binaries compatible between all the various
> stakeholders: compiler/tools people, distros, software vendors,
> etc. Lots of the rest of the details downstream of this can be
> changed, and people do this all the time - compare multilib to
> multi-arch for example. That all works fine *so long as* the runtime
> linker can be located and started OK.

The loader is still available via the old path, so external/third
party/local/other software works unchanged. This should negatively
only affect our 1st party packages, when running on a non-merged
distro.
And are _all_ our packages really 100% compatible with other distros
at all? Are they even supposed to be?

For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
machine, when I try to run it, it fails:

root@focal:/tmp# wget
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
--2023-05-12 12:46:17--
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27572 (27K) [application/vnd.debian.binary-package]
Saving to: 'efibootmgr_17-2_amd64.deb'

efibootmgr_17-2_amd64.deb
100%[===>]  26.93K
--.-KB/sin 0.04s

2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved [27572/27572]

root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
root@focal:/tmp# ./ebm/bin/efibootmgr
./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)

Should I file a severity: serious bug against efibootmgr because it is
not interoperable?

The answer is obviously not, because it would be absurd to expect a
binary compiled against libraries from one distro to "just work" on an
entirely different distro. Glibc itself is not forward compatible, and
is allowed to add new symbols, that are not present in older versions,
and packages are allowed to depend on them. Aren't those also ABI
breakages? What about all the libraries that bump soname? What about
binaries that rely on newer kernel interfaces, or IPC interfaces?

So, what I am asking is, what actual, real difference does it make if,
by default (and with an override available for example), packages
built on Debian for Debian record the ld path to point to its (actual)
location on Debian, via say a compiler spec file that is injected in a
deb build?
There very likely is some real difference and impact, and I am
genuinely and honestly asking what it could be. If nothing else, it's
an interesting topic, even if likely nothing comes out of it.

> Changing the interpreter path would mean moving to a Debian-specific
> ABI, breaking that compatibility. Hand-waving that away with (and I
> quote):
>
>   "The vast majority of distros today ship the loader in /usr/lib as
>   /lib is just a symlink, so it would be interoperable."
>
> is appalling arrogance. No. You do *not* get to break ABI with that
> argument. The point of the ABI spec is that *everybody* follows
> it. You don't change it just because you think it'll make your life a
> little easier when bootstrapping a system.

AFAIK there are at least 3 distros where the default interpreter path
is changed to follow distro-specific customizations: Gentoo, Nix,
Guix. So evidently, some people *do* get to "break 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
>On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
>>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>>>
>>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>>> >
>>> >The core issue as I see it is as follows:
>>> >
>>> >- Debian has decided to support only merged-/usr, including possibly
>>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>>> >  as the interpreter in binaries.
>>>
>>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
>>> seen. The interpreter must *not* be changed willy-nilly.
>>
>>Nothing's happening 'willy-nilly'. We are discussing a bunch of
>>seemingly crazy options, as in, "what would _actually_ explode if we
>>do this or do that?", on this very d-devel thread. I posted a longer
>>version here some days ago:
>>
>>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>
>Oh holy fuck.
>
>You're talking about changing ABI by doing this. That *is* utterly
>crazy. No.

People have asked me to expand on this further...

I've been involved in defining ABI before, specifically for
armhf. It's not a quick and easy process. It needs buy-in from all
sides to make things work, and people *really* value the
interoperability that it enables.

The interpreter path is one of the most important parts of the ABI
spec, the bit that makes binaries compatible between all the various
stakeholders: compiler/tools people, distros, software vendors,
etc. Lots of the rest of the details downstream of this can be
changed, and people do this all the time - compare multilib to
multi-arch for example. That all works fine *so long as* the runtime
linker can be located and started OK.

Changing the interpreter path would mean moving to a Debian-specific
ABI, breaking that compatibility. Hand-waving that away with (and I
quote):

  "The vast majority of distros today ship the loader in /usr/lib as
  /lib is just a symlink, so it would be interoperable."

is appalling arrogance. No. You do *not* get to break ABI with that
argument. The point of the ABI spec is that *everybody* follows
it. You don't change it just because you think it'll make your life a
little easier when bootstrapping a system.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Welcome my son, welcome to the machine.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 11:40, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >>
> >> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >> >
> >> >The core issue as I see it is as follows:
> >> >
> >> >- Debian has decided to support only merged-/usr, including possibly
> >> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >> >  as the interpreter in binaries.
> >>
> >> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >> seen. The interpreter must *not* be changed willy-nilly.
> >
> >Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >seemingly crazy options, as in, "what would _actually_ explode if we
> >do this or do that?", on this very d-devel thread. I posted a longer
> >version here some days ago:
> >
> >https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>
> Oh holy fuck.
>
> You're talking about changing ABI by doing this. That *is* utterly
> crazy. No.

It's a thought experiment on a mailing list. If we can't even have
those anymore, something went very wrong somewhere.

You seem to be aware of things that wouldn't work anymore (I think?).
If you have a couple of minutes to spare, may I please ask you to
reply to that thread with such examples? I am genuinely interested in
understanding and talking about it. Thank you.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>>
>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>> >
>> >The core issue as I see it is as follows:
>> >
>> >- Debian has decided to support only merged-/usr, including possibly
>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>> >  as the interpreter in binaries.
>>
>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
>> seen. The interpreter must *not* be changed willy-nilly.
>
>Nothing's happening 'willy-nilly'. We are discussing a bunch of
>seemingly crazy options, as in, "what would _actually_ explode if we
>do this or do that?", on this very d-devel thread. I posted a longer
>version here some days ago:
>
>https://lists.debian.org/debian-gcc/2023/05/msg00030.html

Oh holy fuck.

You're talking about changing ABI by doing this. That *is* utterly
crazy. No.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"...In the UNIX world, people tend to interpret `non-technical user'
 as meaning someone who's only ever written one device driver." -- Daniel Pead

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >
> >The core issue as I see it is as follows:
> >
> >- Debian has decided to support only merged-/usr, including possibly
> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >  as the interpreter in binaries.
>
> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> seen. The interpreter must *not* be changed willy-nilly.

Nothing's happening 'willy-nilly'. We are discussing a bunch of
seemingly crazy options, as in, "what would _actually_ explode if we
do this or do that?", on this very d-devel thread. I posted a longer
version here some days ago:

https://lists.debian.org/debian-gcc/2023/05/msg00030.html

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>
>The core issue as I see it is as follows:
>
>- Debian has decided to support only merged-/usr, including possibly
>  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>  as the interpreter in binaries.

WTF? *Nobody* has been talking about breaking ABI like this, that I've
seen. The interpreter must *not* be changed willy-nilly.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"I've only once written 'SQL is my bitch' in a comment. But that code 
 is in use on a military site..." -- Simon Booth

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Ansgar]

On Wed, 2023-05-10 at 19:01 -0700, Sean Whitton wrote:
> On Wed 10 May 2023 at 11:47PM +02, Ansgar wrote:
> > Cool, then let's ask tech-ctte.
> > 
> > Dear ctte, please consider overruling the dpkg maintainer to
> > include
> > the patch from #994388[1].
> > 
> > Thanks,
> > Ansgar
> > 
> >   [1]: https://bugs.debian.org/994388#397
> 
> This would require a new, maintainer-overruling vote.
> Our existing decisions do not apply, so far as I can tell.

Yes, I agree.

> I have written a separate message to the bug and to debian-dpkg with
> a proposal to avoid having to have such a vote.

That seems to be about an implementation detail on how to apply the
patch. I don't think that is the core of the issue?

The core issue as I see it is as follows:

- Debian has decided to support only merged-/usr, including possibly
  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
  as the interpreter in binaries.

- This change breaks on non-merged-/usr systems, including derivatives
  that do not revert *all* relevant changes. (Do you know one that
  does this or plans to do so?)

- dpkg recommends derivative users to move to non-merged-/usr.

I think this contradiction is not good and the core conflict. For me a
distribution should have some coherence. It is not just a distribution
of unrelated parts (like linux, libc, dpkg, dash, ...), but also
integrates them to work together.

And this also means not one package telling users to do X which breaks
other packages. Or (if other packages would do similar things as dpkg)
one package asking users to do X and the other asking users to do the
opposite of X. Just imagine dpkg asking users to move to split-/usr and
then another package starting to warn users to move back to merged-
/usr. Would that be a good state? I think not which is why this bug
exists.

Do you think this summary of the issue is right?

Is there some consensus about how this issue should be solved or do we
need a longer discussion to explore the solution space?

Ansgar

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Sean Whitton]

Hello,

On Wed 10 May 2023 at 11:47PM +02, Ansgar wrote:

> Cool, then let's ask tech-ctte.
>
> Dear ctte, please consider overruling the dpkg maintainer to include
> the patch from #994388[1].
>
> Thanks,
> Ansgar
>
>   [1]: https://bugs.debian.org/994388#397

This would require a new, maintainer-overruling vote.
Our existing decisions do not apply, so far as I can tell.

I have written a separate message to the bug and to debian-dpkg with a
proposal to avoid having to have such a vote.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035904: dpkg currently warning about merged-usr systems (revisited) (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Ansgar]

Package: tech-ctte
X-Debbugs-Cc: Russ Allbery , Sean Whitton 
, Helmut Grohne , Luca Boccassi 
, debian-d...@lists.debian.org, debian-devel@lists.debian.org

On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
> Ansgar  writes:
> > Debian going out of its way to tell derivative users to switch back from
> > merged-/usr to split-/usr is the *opposite* of trying to make things as
> > smooth for them as possible.
> 
> Yes, I agree with that part and I think I objected to that at the time.
> Nonetheless, one bad decision doesn't mean that it is Debian policy that
> we don't care about derivatives or their users.  I think we made a mistake
> there which is not in alignment with our ideals or our goals.  We should
> try to reverse that mistake, not double down on it.

Cool, then let's ask tech-ctte.

Dear ctte, please consider overruling the dpkg maintainer to include
the patch from #994388[1].

Thanks,
Ansgar

  [1]: https://bugs.debian.org/994388#397