Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Ansgar  writes:

> But the subject of this issue talks about "script interpreters", not
> just `/bin/sh` (which I guess is safe to assume would be one of the
> "handful").

> It is unclear what files the Jackson symlink farm proposal would leave
> in /bin.  Would /bin/python3 stay?  Or would it stay first, but dropped
> at some later point?  What about /bin/perl, /bin/zsh, /bin/env, ...?

Oh, okay, I see what you're syaing now.  This is a bit beyond the scope of
the areas of Policy touched by Luca's proposal, but I can see why it would
indeed arise under Ian's proposal.  We've introduced new /bin interfaces
for every binary in /usr/bin, and if we went down that path we'd remove
most of those interfaces again.  That is indeed an argument for (c) for
*most* things, just not the areas touched by this diff (with the possible
exception of /bin/csh; I'm not sure if that would qualify for an exception
or not, these days).

So yes, I agree that the resolution of this bug would significantly affect
what we want to say in Policy about /usr-merge in general, even if not
what we say about /bin/sh.

I still don't feel like we need to wait for the TC bug to be resolved,
since there is a standing TC decision to make /bin a symlink to /usr/bin
and we can always change our wording later if that decision changes, but
we need to wait for the buildd /usr-merge anyway, so either way I don't
think we're ready to merge patches for this bug right now.

-- 
Russ Allbery (r...@debian.org)  

Bug#1051371: Post-/usr-merge paths for script interpreters

[Ansgar]

On Sun, 2023-09-17 at 11:54 +0200, Bill Allombert wrote:
> /bin/perl, /bin/env, /bin/python3 did not exist in the old scheme, so
> there is no point in creating them now.

No, in Debian's current filesystem layout (Debian stable and later;
partly Debian oldoldstable and later) all of these exist.  They are
also used, both by software shipping in Debian and outside.

Dropping them would break existing software.  I'm assuming the Jackson
symlink proposal would retain them for this reason instead of breaking
software for no good reason.

Ansgar

Bug#1051371: Post-/usr-merge paths for script interpreters

[Bill Allombert]

On Sun, Sep 17, 2023 at 08:52:17AM +0200, Ansgar wrote:
> > Control: unblock 1051371 by 1050001
> > 
> > Ansgar  writes:
> > 
> > > However, there is a proposal by Jackson for an alternative filesystem
> > > layout based on symlink farms in consideration by the technical
> > > committee.  This advocates removing compat symlinks in /bin, /sbin over
> > > time[1], thus requiring (c).
> > 
> > This is not a correct summary of Ian's proposal.  In the message that you
> > linked, Ian says:
> > 
> >     /bin and /lib etc. remain directories (so there is no aliasing).  All
> >     actual files are shipped in /usr.  / contains compatibility symlinks
> >     pointing into /usr, for those files/APIs/programs where this is needed
> >     (which is far from all of them).  Eventualloy, over time, the set of
> >     compatibility links is reduced to a mere handful.
> > 
> > I am absolutely certain that Ian would consider /bin/sh to be one of the
> > programs for which a compatibility symlink is needed, and one of the
> > remaining handful of links that would exist indefinitely into the future.
> > Indeed, he mentions /bin/sh explicitly later in that message.
> 
> But the subject of this issue talks about "script interpreters", not
> just `/bin/sh` (which I guess is safe to assume would be one of the
> "handful").
> 
> It is unclear what files the Jackson symlink farm proposal would leave
> in /bin.  Would /bin/python3 stay?  Or would it stay first, but dropped
> at some later point?  What about /bin/perl, /bin/zsh, /bin/env, ...?

/bin/perl, /bin/env, /bin/python3 did not exist in the old scheme, so there is
no point in creating them now.

/bin/zsh and /bin/sed existed, though.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1051371: Post-/usr-merge paths for script interpreters

[Ansgar]

Hi,

On Sat, 2023-09-16 at 12:58 -0700, Russ Allbery wrote:
> Control: unblock 1051371 by 1050001
> 
> Ansgar  writes:
> 
> > However, there is a proposal by Jackson for an alternative filesystem
> > layout based on symlink farms in consideration by the technical
> > committee.  This advocates removing compat symlinks in /bin, /sbin over
> > time[1], thus requiring (c).
> 
> This is not a correct summary of Ian's proposal.  In the message that you
> linked, Ian says:
> 
>     /bin and /lib etc. remain directories (so there is no aliasing).  All
>     actual files are shipped in /usr.  / contains compatibility symlinks
>     pointing into /usr, for those files/APIs/programs where this is needed
>     (which is far from all of them).  Eventualloy, over time, the set of
>     compatibility links is reduced to a mere handful.
> 
> I am absolutely certain that Ian would consider /bin/sh to be one of the
> programs for which a compatibility symlink is needed, and one of the
> remaining handful of links that would exist indefinitely into the future.
> Indeed, he mentions /bin/sh explicitly later in that message.

But the subject of this issue talks about "script interpreters", not
just `/bin/sh` (which I guess is safe to assume would be one of the
"handful").

It is unclear what files the Jackson symlink farm proposal would leave
in /bin.  Would /bin/python3 stay?  Or would it stay first, but dropped
at some later point?  What about /bin/perl, /bin/zsh, /bin/env, ...?

As far as I understand most compat symlinks should eventually be
dropped (but which is unclear). Therefore doing (c) is safe, but
anything else might use symlinks that eventually disappear.

(And really: what about calling /sbin/ifconfig, ... as well?  Here (c)
is again the only safe solution with the Jackson symlink farming
proposal.)

It was asked to clarify the plan for this multiple times, sadly the
proposers of the newly proposed filesystem layout have refused to give
any answer so far.

Ansgar

Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Bill Allombert  writes:

> One of the issue in the past is that reproducible build was broken
> because different build environment lead to different paths. We at least
> need to address that.

I believe the reproducible build problem specifically will be largely
fixed by /usr-merging the buildds so that they look like all other Debian
systems.  I suspect the problems that you ran into in the past were
precisely because some systems on which the package was built were
/usr-merged and others were not.  But you make a good point that just
because the /bin and /usr/bin paths both work does not mean that package
build systems can pick randomly between them, since that undermines build
reproducibility.  They need to pick one or the other consistently.

I do think packages should be allowed to do a PATH search, and it's up to
the people doing a reproducible build to make sure the PATH stays
consistent from one build to the next.

-- 
Russ Allbery (r...@debian.org)  

Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Sam Hartman  writes:

> My problem with (b) is that I value interfaces and that especially for
> /bin/sh I do think that /bin/sh is more portable as an interface path
> than /usr/bin/sh.

To be clear, I personally agree with this, and I am sure that I will use
/bin/sh until the end of time, even if every extant UNIX system decides to
create a /usr/bin/sh alias for it (one way or the other).

But from an interface perspective, I think it adds some clarity to think
of it this way: When people started merging /bin and /usr/bin, they added
a new /usr/bin/sh interface for invoking the Bourne shell.  They may not
have intended to do this, but that's what they did, and there was nothing
inherent in the system to prevent people from using that interface.  So
people started doing so.

I'm sure that I'm not the only one who has encountered #!/usr/bin/sh
scripts in the wild, usually because the person who wrote the script was
using some Red Hat or Fedora distribution and didn't really think about
it.  Either they were used to the /usr/bin pattern from Perl and Python or
they ran "which sh" and, as Bill points out, that returns /usr/bin/sh,
etc.  Up until now, those scripts didn't work on Debian systems, which
caused some minor annoyances.  Now they do.

What has essentially happened given the number of distributions that have
done /usr-merge is that Linux has added /usr/bin/sh as a new interface
that is interchangeable with /bin/sh, and now nearly every current Linux
system you run across will also support that interface.  This is annoying
for UNIXes that don't support that interface (I don't know what the BSDs
are doing, for instance), but it's fairly invisible to users unless they
care about portability to non-Linux systems.  I care about portability to
non-Linux systems in at least some situations, but a lot of people don't.

(Incidentally, this interface argument is also why I have some nit-picks
about the wording of Luca's proposed patch, because the guarantees in
Policy apply specifically to /bin/sh and /usr/bin/sh, not to, say,
/usr/local/bin/sh or /usr/xpg4/bin/sh or whatever.  This is an admittedly
very minor point.)

> I care a lot that we use /bin/sh in our documentation and examples
> (especially in policy).

I agree with this.  If we write example scripts, I think they should use
#!/bin/sh.  I don't think my (b) proposal is arguing against that; (b)
says that both paths work and you can use either of them, and Policy will
chose to use /bin/sh at least for the time being because, well, I'm one of
the Policy Editors and I care about things like that.  :)  I think Policy
would only use /usr/bin/sh in documentation and examples if we adopted
(c), which based on the discussion so far seems unlikely.

> I care a lot that we point out that the reality of the situation is
> people will see other paths.

This is the part that I'm really focusing on at the moment, because I
think it's what people are really asking and what directly addresses the
concern about bug reports asking maintainers to switch from /usr/bin/sh to
/bin/sh.  If Debian is going to contain scripts that use /usr/bin/sh, this
is a change and we should say so explicitly.

Bill pointed out precisely the way that I expect this to happen: a lot of
packages use PATH searches in their build systems to find common tools,
and although I think this is ill-advised, some of them do that for sh
(mostly because of old Solaris breakage).  As soon as the buildds are
/usr-merged, those packages are going to start using /usr/bin/sh instead
of /bin/sh because it's first in the PATH, assuming the buildds use the
same PATH setup as /etc/login.defs (I don't know if this is the case).
And the question is do we tell maintainers that they should fix this, or
do we accept that it will happen and not worry about it?

I personally probably would find a way to force /bin/sh somewhere because
I am a perfectionist, but it requires overriding the upstream build system
and maintaining that code going forward and a lot of maintainers aren't
going to want to bother.  I think Policy should provide some guidance to
both maintainers and bug reporters here about what we expect.

> At least for things traditionally in /bin I do not want to encourage
> those other paths, but I also don't think it is often a good use of
> project resources to "fix" those other paths.

Precisely.  To me, that's the heart of the (b) position, although people
advocating for (b) are going to vary from "prefer /bin" to "prefer
/usr/bin" and (b) is sort of inherently a compromise position about what
we're going to *do* among people who have different preferences about
style.

> In some cases (for example what version of a path autoconf detects), I
> think that patching individual packages to detect a particular path
> would be net harmful.

Yeah, it sounds like you're thinking of exactly the case I was thinking
of.

> So I want to argue for (a) with no enforcement mechanism in packages.

> 1) Policy should enc

Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Control: unblock 1051371 by 1050001

Ansgar  writes:

> However, there is a proposal by Jackson for an alternative filesystem
> layout based on symlink farms in consideration by the technical
> committee.  This advocates removing compat symlinks in /bin, /sbin over
> time[1], thus requiring (c).

This is not a correct summary of Ian's proposal.  In the message that you
linked, Ian says:

/bin and /lib etc. remain directories (so there is no aliasing).  All
actual files are shipped in /usr.  / contains compatibility symlinks
pointing into /usr, for those files/APIs/programs where this is needed
(which is far from all of them).  Eventualloy, over time, the set of
compatibility links is reduced to a mere handful.

I am absolutely certain that Ian would consider /bin/sh to be one of the
programs for which a compatibility symlink is needed, and one of the
remaining handful of links that would exist indefinitely into the future.
Indeed, he mentions /bin/sh explicitly later in that message.

Given that, I believe Ian's proposal is orthogonal to this bug.  For
/bin/sh and /usr/bin/sh, it would create the same aliasing and thus would
create the same question about how to talk about those paths in Policy.  I
therefore don't think resolution of this bug blocks on the TC bug.

-- 
Russ Allbery (r...@debian.org)  

Processed: Re: Bug#1051371: Post-/usr-merge paths for script interpreters

[Debian Bug Tracking System]

Processing control commands:

> unblock 1051371 by 1050001
Bug #1051371 [debian-policy] Post-/usr-merge paths for script interpreters
1051371 was not blocked by any bugs.
1051371 was not blocking any bugs.
Removed blocking bug(s) of 1051371: 1050001

-- 
1051371: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Processed: Re: Bug#1051371: Post-/usr-merge paths for script interpreters

[Debian Bug Tracking System]

Processing control commands:

> unblock 1051371 by 1050001
Bug #1051371 [debian-policy] Post-/usr-merge paths for script interpreters
1051371 was blocked by: 1050001
1051371 was not blocking any bugs.
Removed blocking bug(s) of 1051371: 1050001

-- 
1050001: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050001
1051371: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Luca Boccassi  writes:
> On Wed, 13 Sept 2023 at 04:48, Russ Allbery  wrote:

>> Simon pointed out that this bug is not yet ready to act on, which was
>> very helpful.  Thank you.  However, presumably the buildds will be
>> /usr-merged at some point in the not-too-distant future, and we do need
>> to decide what to do after that point.

> While that could be said for the original revision, in my view that's
> not really the case for the latest that I posted?

> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

> So I would prefer if this was a clone rather than a retitle/repurpose.
> Unless I'm missing something, the changes linked above should be
> uncontroversial and simply remove excessively normative language in what
> are essentially examples that should have been taken as such - and that
> currently are not. So, could that be taken forward independently of the
> problem you define below?

I believe it would be an error to move Policy away from explicitly saying
/bin/sh as long as we have unmerged systems.  For as long as we have to
support unmerged systems, which includes the buildds because quite a
surprising range of packages end up needing to be installed on buildds
during package builds, packages must use /bin/sh and may not use
/usr/bin/sh.

This is not currently explicit in Policy because previously it wasn't
necessary.  Packages using /usr/bin/sh would simply not work, thus forcing
the issue.  But right now, if we were writing Policy from scratch, we
would need to explicitly say that using /bin/sh is a must in order to
avoid breaking the buildds, since /usr/bin/sh would appear to work locally
and then potentially cause weird problems during package builds.  (Ideally
build failure, but a lot of strange things can happen when paths are
missing during a build.)

Presumably this is getting fixed in short order, so it's not worth the
intermediate Policy change to add that language only to remove it again.
But similarly, I don't think it's correct to relax Policy language about
the /bin/sh path for as long as using /usr/bin/sh is potentially a
release-critical bug.

Obviously this all becomes moot as soon as the buildds are /usr-merged.

> I think b would work out fine, but if we want to start being normative
> then it probably would make more sense to go for the new layout rather
> than the old. It would seem strange to have lived with the old layout
> and no rule, and suddenly add a rule for the old layout just as it has
> been phased out, no?

The reason why this is not strange to me (which is not to say that this is
my personal preference) is that previously we had a rule requiring /bin/sh
enforced by a much harsher mechanism than Policy:  using /usr/bin/sh would
simply break.  So we *did* have a de facto rule, which we implicitly
dropped by doing /usr-merge, and the debate is whether to replace it with
a de jure rule.

-- 
Russ Allbery (r...@debian.org)  

Bug#1051371: Post-/usr-merge paths for script interpreters

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> On Wed, 13 Sept 2023 at 04:48, Russ Allbery  wrote:
>> 
>> Control: retitle -1 Post-/usr-merge paths for script interpreters
>> 
>> Simon pointed out that this bug is not yet ready to act on, which
>> was very helpful.  Thank you.  However, presumably the buildds
>> will be /usr-merged at some point in the not-too-distant future,
>> and we do need to decide what to do after that point.

Luca> While that could be said for the original revision, in my view
Luca> that's not really the case for the latest that I posted?

Luca> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

Luca> So I would prefer if this was a clone rather than a
Luca> retitle/repurpose.  Unless I'm missing something, the changes
Luca> linked above should be uncontroversial and simply remove
Luca> excessively normative language in what are essentially
Luca> examples that should have been taken as such - and that
Luca> currently are not. So, could that be taken forward
Luca> independently of the problem you define below?

I agree the above is uncontroversial and would support including it in
policy now.
I don't think it needs seconds because it is non-normative.

(As an aside, reading the summary, I expected to find the patch
something I was not entirely happy with.  I was planning to hold my
nose, and neither support the patch nor object.  However, since you
brought it up again, I read the full patch and find I like the patch a
lot better than your summary of it:-)


signature.asc
Description: PGP signature

Bug#1051371: Post-/usr-merge paths for script interpreters

[Bill Allombert]

On Wed, Sep 13, 2023 at 10:47:48AM +0200, Bill Allombert wrote:
> On Tue, Sep 12, 2023 at 08:48:04PM -0700, Russ Allbery wrote:
> > Control: retitle -1 Post-/usr-merge paths for script interpreters
> > 
> > Simon pointed out that this bug is not yet ready to act on, which was very
> > helpful.  Thank you.  However, presumably the buildds will be /usr-merged
> > at some point in the not-too-distant future, and we do need to decide what
> > to do after that point.
> > 
> > I think the root problem behind this bug is that it is revealing we have
> > not made a decision about /bin and /usr/bin path references in Debian
> > after /usr-merge.  Various people, myself included, made assumptions about
> > what the policy would be, but we never actually decided anything that I am
> > aware of and people's assumptions are not matching.  I think we need to
> > talk about this directly, after which what to do with this bug will
> > probably become obvious.
> 
> Russ, there is a quite related point I do not think the TC addressed 
> directly, 
> but I can easily be mistaken: the default PATH.
> It is currently defined in /etc/login.defs (unless my copy of this file is 
> out of date):
> as
> ENV_SUPATH  
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> ENV_PATHPATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
> 
> so in pratice, tools like 'which' will always favor /usr/bin over /bin
> 
> $ which sh
> /usr/bin/sh
> 
> One of the issue in the past is that reproducible build was broken because 
> different
> build environment lead to different paths. We at least need to address that.

To be specific, I needed to add 
SHELL=/bin/sh GREP=/bin/grep SED=/bin/sed DD=/bin/dd
to configure to get reproducible build. (This was in December 2018).

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1051371: Post-/usr-merge paths for script interpreters

[Luca Boccassi]

On Wed, 13 Sept 2023 at 04:48, Russ Allbery  wrote:
>
> Control: retitle -1 Post-/usr-merge paths for script interpreters
>
> Simon pointed out that this bug is not yet ready to act on, which was very
> helpful.  Thank you.  However, presumably the buildds will be /usr-merged
> at some point in the not-too-distant future, and we do need to decide what
> to do after that point.

While that could be said for the original revision, in my view that's
not really the case for the latest that I posted?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

So I would prefer if this was a clone rather than a retitle/repurpose.
Unless I'm missing something, the changes linked above should be
uncontroversial and simply remove excessively normative language in
what are essentially examples that should have been taken as such -
and that currently are not. So, could that be taken forward
independently of the problem you define below?

> I think the root problem behind this bug is that it is revealing we have
> not made a decision about /bin and /usr/bin path references in Debian
> after /usr-merge.  Various people, myself included, made assumptions about
> what the policy would be, but we never actually decided anything that I am
> aware of and people's assumptions are not matching.  I think we need to
> talk about this directly, after which what to do with this bug will
> probably become obvious.
>
> So far as I can tell, there are three main possibilities:
>
> (a) Although /bin and /usr/bin are merged (and likewise for the other
> merged paths), Debian will continue to require (or at least recommend)
> use of /bin paths for things such as /bin/sh that historically used
> those paths.
>
> (b) Since /bin and /usr/bin (and likewise for the other paths) are merged,
> /bin/sh and /usr/bin/sh are equivalent.  Packages can use whichever
> path they want, and Debian will end up with a mix of both references.
>
> (c) Although /bin and /lib technically work due to the aliasing, they are
> deprecated and everything in Debian should stop using those paths.
> All paths should point to /usr/bin and /usr/lib now.
>
> Although Luca made a few arguments in the direction of (c), my
> understanding of his current patch is that it essentially implements (b)
> for script interpreters, although without encoding into Policy an explicit
> decision between these three options (quite understandably because he was
> trying to deal with a narrower issue).  Several other people were, I
> think, arguing for (a), but I'm not sure if they would continue to do so
> when it's put in these terms.
>
> Policy currently says nothing significant about this (hence most of the
> text so-far discussed being informative) because, up until now, (a) was
> the de facto policy.  If you didn't follow (a), you would get not-found
> errors and your package would have an RC bug because it wouldn't work.
>
> If we do nothing, we'll get (b).  I think that's reasonably obvious, since
> there is no technical factor forcing either (a) or (c).  Both paths will
> work without any noticable difference, so some people will use /bin and
> some people will use /usr/bin.
>
> That said, I would rather make an explicit choice rather than decide by
> default, since otherwise this seems like the kind of thing where people
> are going to get conflicting advice, which is frustrating for everyone.
>
> If someone wants to argue for (a) or (c), I think the biggest problem with
> either of them is an enforcement mechanism.  How are we going to check
> whether packages are following the rules?  Lintian and a bunch of grep
> patterns is sort of an unsatisfying 90% solution, and people will ignore
> it or just not use Lintian.  There is also no technical reason why both
> paths will not work, so people are going to get grumpy about being told
> what to do and some people will view this as makework.  I think any
> proposal to pick (a) or (c) needs to wrestle with that.
>
> I will also say that it will be very hard to convince me that Debian
> should give Policy advice like (a) or (c) but not actually enforce it
> anywhere.  We have a long history to look at for how those sorts of
> mandates go.  Conscientious packagers who read Policy carefully follow the
> rules and go to extra effort to do so, but other people will never see
> that advice or not pay attention to it.  That means that the effort is
> mostly wasted, because no one can rely on the invariant that either (a) or
> (c) is attempting to achieve.  I am not a big fan of asking people to do
> extra work without some clear benefit from doing that work, which mostly
> requires uniformity.
>
> One last point about this decision: although there are a few style
> recommendations in it, Policy is not a style guide.  The point of Policy
> is to describe the things we should or must do, or at least that the
> project as a whole wants to encourage, that have a concrete effect on the
> quality of the distribu

Bug#1051371: Post-/usr-merge paths for script interpreters

[Sam Hartman]

> "Russ" == Russ Allbery  writes:
Russ> with a narrower issue).  Several other people were, I think,
Russ> arguing for (a), but I'm not sure if they would continue to do
Russ> so when it's put in these terms.

It's hard for me to express what I was advocating for in the terms you
have adopted.
I think you've  advanced the discussion significantly by making it clear
where the difficulty is, which will help me focus my argument.


Russ> If someone wants to argue for (a) or (c), I think the biggest
Russ> problem with either of them is an enforcement mechanism.

My problem with (b) is that I value interfaces and that  especially for
/bin/sh I do think that /bin/sh is more portable as an interface path
than /usr/bin/sh.
I care a lot that we use /bin/sh in our documentation and examples
(especially in policy).
I care a lot that we point out that the reality of the situation is
people  will see other paths.
At least for things traditionally in /bin I do not want to encourage
those other paths, but I also don't think it is often a good use of
project resources to "fix" those other paths.
In some cases (for example what version of a path autoconf detects), I
think that patching individual packages to detect a particular path
would be net harmful.

So I want to argue for (a) with no  enforcement mechanism in packages.

1) Policy should encourage /bin paths for binaries traditionally in
/bin. (At a minimum I'd like to see this for /bin/sh and /bin/bash).
That at most makes it a minor bug if you don't follow that
encouragement.

2) The examples in policy should use the standard interface paths.
(This is the thing I care most about).

3) I'd like to see policy point out that /usr/bin paths will end up
being used, and packages SHOULD work regardless of which path is used.

I think there are some complexities here.  Imagine someone writes a tool
to determine if something is a shell script.  If it's security
sensitive, I think it is important that it work both for /bin/sh and
/usr/bin/sh.

If it's a syntax highlighting program and it only detects
/bin/sh as a valid shell script, I'd be okay if the maintainer didn't
want to fix it but instead said "fix your shell scripts if they say
/usr/bin/sh."  (I think the maintainer would be doing a better job if
they supported both).  But I would not be comfortable with a syntax
highlighting tool pushing people toward /usr/bin/sh.



signature.asc
Description: PGP signature

Bug#1051371: Post-/usr-merge paths for script interpreters

[Bill Allombert]

On Tue, Sep 12, 2023 at 08:48:04PM -0700, Russ Allbery wrote:
> Control: retitle -1 Post-/usr-merge paths for script interpreters
> 
> Simon pointed out that this bug is not yet ready to act on, which was very
> helpful.  Thank you.  However, presumably the buildds will be /usr-merged
> at some point in the not-too-distant future, and we do need to decide what
> to do after that point.
> 
> I think the root problem behind this bug is that it is revealing we have
> not made a decision about /bin and /usr/bin path references in Debian
> after /usr-merge.  Various people, myself included, made assumptions about
> what the policy would be, but we never actually decided anything that I am
> aware of and people's assumptions are not matching.  I think we need to
> talk about this directly, after which what to do with this bug will
> probably become obvious.

Russ, there is a quite related point I do not think the TC addressed directly, 
but I can easily be mistaken: the default PATH.
It is currently defined in /etc/login.defs (unless my copy of this file is out 
of date):
as
ENV_SUPATH  
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV_PATHPATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

so in pratice, tools like 'which' will always favor /usr/bin over /bin

$ which sh
/usr/bin/sh

One of the issue in the past is that reproducible build was broken because 
different
build environment lead to different paths. We at least need to address that.

Personnally, I favor keeping /bin/sh for consistency, but that is aside.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Bug#1051371: Post-/usr-merge paths for script interpreters

[Ansgar]

Control: block 1051371 by 1050001

Hi,

On Tue, 2023-09-12 at 20:48 -0700, Russ Allbery wrote:

> I think the root problem behind this bug is that it is revealing we have
> not made a decision about /bin and /usr/bin path references in Debian
> after /usr-merge.  Various people, myself included, made assumptions about
> what the policy would be, but we never actually decided anything that I am
> aware of and people's assumptions are not matching.  I think we need to
> talk about this directly, after which what to do with this bug will
> probably become obvious.
> 
> So far as I can tell, there are three main possibilities:
> 
> (a) Although /bin and /usr/bin are merged (and likewise for the other
>     merged paths), Debian will continue to require (or at least recommend)
>     use of /bin paths for things such as /bin/sh that historically used
>     those paths.
> 
> (b) Since /bin and /usr/bin (and likewise for the other paths) are merged,
>     /bin/sh and /usr/bin/sh are equivalent.  Packages can use whichever
>     path they want, and Debian will end up with a mix of both references.
> 
> (c) Although /bin and /lib technically work due to the aliasing, they are
>     deprecated and everything in Debian should stop using those paths.
>     All paths should point to /usr/bin and /usr/lib now.

As far as I understand people wanting merged-/usr want (b) (I do).
There have been people advocating (a) in this bug.

However, there is a proposal by Jackson for an alternative filesystem
layout based on symlink farms in consideration by the technical
committee.  This advocates removing compat symlinks in /bin, /sbin over
time[1], thus requiring (c).

The technical committee should therefore probably be aware of this
policy issue in their consideration of #1050001; the resolution of
which might also cover this issue (#1051371).

Ansgar

  [1]: https://bugs.debian.org/1050001#33

Processed: Re: Bug#1051371: Post-/usr-merge paths for script interpreters

[Debian Bug Tracking System]

Processing control commands:

> block 1051371 by 1050001
Bug #1051371 [debian-policy] Post-/usr-merge paths for script interpreters
1051371 was blocked by: 1050001
1051371 was not blocking any bugs.
Ignoring request to alter blocking bugs of bug #1051371 to the same blocks 
previously set

-- 
1051371: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Processed: Re: Bug#1051371: Post-/usr-merge paths for script interpreters

[Debian Bug Tracking System]

Processing control commands:

> block 1051371 by 1050001
Bug #1051371 [debian-policy] Post-/usr-merge paths for script interpreters
1051371 was not blocked by any bugs.
1051371 was not blocking any bugs.
Added blocking bug(s) of 1051371: 1050001

-- 
1050001: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050001
1051371: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Control: retitle -1 Post-/usr-merge paths for script interpreters

Simon pointed out that this bug is not yet ready to act on, which was very
helpful.  Thank you.  However, presumably the buildds will be /usr-merged
at some point in the not-too-distant future, and we do need to decide what
to do after that point.

I think the root problem behind this bug is that it is revealing we have
not made a decision about /bin and /usr/bin path references in Debian
after /usr-merge.  Various people, myself included, made assumptions about
what the policy would be, but we never actually decided anything that I am
aware of and people's assumptions are not matching.  I think we need to
talk about this directly, after which what to do with this bug will
probably become obvious.

So far as I can tell, there are three main possibilities:

(a) Although /bin and /usr/bin are merged (and likewise for the other
merged paths), Debian will continue to require (or at least recommend)
use of /bin paths for things such as /bin/sh that historically used
those paths.

(b) Since /bin and /usr/bin (and likewise for the other paths) are merged,
/bin/sh and /usr/bin/sh are equivalent.  Packages can use whichever
path they want, and Debian will end up with a mix of both references.

(c) Although /bin and /lib technically work due to the aliasing, they are
deprecated and everything in Debian should stop using those paths.
All paths should point to /usr/bin and /usr/lib now.

Although Luca made a few arguments in the direction of (c), my
understanding of his current patch is that it essentially implements (b)
for script interpreters, although without encoding into Policy an explicit
decision between these three options (quite understandably because he was
trying to deal with a narrower issue).  Several other people were, I
think, arguing for (a), but I'm not sure if they would continue to do so
when it's put in these terms.

Policy currently says nothing significant about this (hence most of the
text so-far discussed being informative) because, up until now, (a) was
the de facto policy.  If you didn't follow (a), you would get not-found
errors and your package would have an RC bug because it wouldn't work.

If we do nothing, we'll get (b).  I think that's reasonably obvious, since
there is no technical factor forcing either (a) or (c).  Both paths will
work without any noticable difference, so some people will use /bin and
some people will use /usr/bin.

That said, I would rather make an explicit choice rather than decide by
default, since otherwise this seems like the kind of thing where people
are going to get conflicting advice, which is frustrating for everyone.

If someone wants to argue for (a) or (c), I think the biggest problem with
either of them is an enforcement mechanism.  How are we going to check
whether packages are following the rules?  Lintian and a bunch of grep
patterns is sort of an unsatisfying 90% solution, and people will ignore
it or just not use Lintian.  There is also no technical reason why both
paths will not work, so people are going to get grumpy about being told
what to do and some people will view this as makework.  I think any
proposal to pick (a) or (c) needs to wrestle with that.

I will also say that it will be very hard to convince me that Debian
should give Policy advice like (a) or (c) but not actually enforce it
anywhere.  We have a long history to look at for how those sorts of
mandates go.  Conscientious packagers who read Policy carefully follow the
rules and go to extra effort to do so, but other people will never see
that advice or not pay attention to it.  That means that the effort is
mostly wasted, because no one can rely on the invariant that either (a) or
(c) is attempting to achieve.  I am not a big fan of asking people to do
extra work without some clear benefit from doing that work, which mostly
requires uniformity.

One last point about this decision: although there are a few style
recommendations in it, Policy is not a style guide.  The point of Policy
is to describe the things we should or must do, or at least that the
project as a whole wants to encourage, that have a concrete effect on the
quality of the distribution.  I'm reluctant to add more style advice to
Policy, particularly about things that are not specific to Debian.  If
we're going to require or recommend something, I think we need to have a
concrete goal in mind for what that requirement or recommendation is going
to achieve.

-- 
Russ Allbery (r...@debian.org)  

Processed: Re: Bug#1051371: Post-/usr-merge paths for script interpreters

[Debian Bug Tracking System]

Processing control commands:

> retitle -1 Post-/usr-merge paths for script interpreters
Bug #1051371 [debian-policy] debian-policy: stop referring to specific paths in 
scripts shebang examples
Changed Bug title to 'Post-/usr-merge paths for script interpreters' from 
'debian-policy: stop referring to specific paths in scripts shebang examples'.

-- 
1051371: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems