Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Luca Boccassi  writes:
> On Wed, 13 Sept 2023 at 04:48, Russ Allbery  wrote:

>> Simon pointed out that this bug is not yet ready to act on, which was
>> very helpful.  Thank you.  However, presumably the buildds will be
>> /usr-merged at some point in the not-too-distant future, and we do need
>> to decide what to do after that point.

> While that could be said for the original revision, in my view that's
> not really the case for the latest that I posted?

> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

> So I would prefer if this was a clone rather than a retitle/repurpose.
> Unless I'm missing something, the changes linked above should be
> uncontroversial and simply remove excessively normative language in what
> are essentially examples that should have been taken as such - and that
> currently are not. So, could that be taken forward independently of the
> problem you define below?

I believe it would be an error to move Policy away from explicitly saying
/bin/sh as long as we have unmerged systems.  For as long as we have to
support unmerged systems, which includes the buildds because quite a
surprising range of packages end up needing to be installed on buildds
during package builds, packages must use /bin/sh and may not use
/usr/bin/sh.

This is not currently explicit in Policy because previously it wasn't
necessary.  Packages using /usr/bin/sh would simply not work, thus forcing
the issue.  But right now, if we were writing Policy from scratch, we
would need to explicitly say that using /bin/sh is a must in order to
avoid breaking the buildds, since /usr/bin/sh would appear to work locally
and then potentially cause weird problems during package builds.  (Ideally
build failure, but a lot of strange things can happen when paths are
missing during a build.)

Presumably this is getting fixed in short order, so it's not worth the
intermediate Policy change to add that language only to remove it again.
But similarly, I don't think it's correct to relax Policy language about
the /bin/sh path for as long as using /usr/bin/sh is potentially a
release-critical bug.

Obviously this all becomes moot as soon as the buildds are /usr-merged.

> I think b would work out fine, but if we want to start being normative
> then it probably would make more sense to go for the new layout rather
> than the old. It would seem strange to have lived with the old layout
> and no rule, and suddenly add a rule for the old layout just as it has
> been phased out, no?

The reason why this is not strange to me (which is not to say that this is
my personal preference) is that previously we had a rule requiring /bin/sh
enforced by a much harsher mechanism than Policy:  using /usr/bin/sh would
simply break.  So we *did* have a de facto rule, which we implicitly
dropped by doing /usr-merge, and the debate is whether to replace it with
a de jure rule.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Guillem Jover  writes:

> Not shipping these empty directories in the .deb seems like a regression
> or a disservice to me. Even for things that might get deleted because
> things like our policy or the FHS allows for it (say stuff under
> /var/cache), as «dpkg --verify» can be useful. Because of course, these
> in addition, can be defined via tmpfiles.d, so that they can possibly be
> recreated if needed (until dpkg provides its own interfaces to do that).

Luca, are there any drawbacks for your purposes in both shipping the
directories in the deb *and* defining them with tmpfiles.d for those cases
where it is possible to ship them in the deb (no dynamic owner or group,
for instance)?  At first glance, it feels like this should be fine, since
tmpfiles.d will recreate the directories and dpkg will then be happy with
them.

It does potentially create problems if dpkg and tmpfiles.d have different
ideas about what the ownership or permissions of the directories should
be, but at present I don't think such conflicts would create a lot of
practical problems (tmpfiles.d should essentialy always win), so I think
it's a fairly minor point.

It's a bit more complicated to specify in Policy because it's not possible
to include the directory in the deb file in cases where it needs to have
ownership set based on users or groups created dynamically by the
maintainer scripts, but hopefully not overly complicated.

This has the valuable benefit, as Guillem points out, of retaining dpkg
database awareness of the association between that directory and a package
until such time as dpkg is aware of files defined in tmpfiles.d (directly
or indirectly via debhelper magic to register the tmpfiles.d targets with
a new dpkg dynamic file database; the latter is my guess about where we're
headed based on previous discussions).

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Guillem Jover]

Hi!

On Tue, 2023-09-12 at 22:17:44 -0700, Russ Allbery wrote:
> Russ Allbery  writes:
> > Russ Allbery  writes:
> >> Maybe the right way to do this is just have two examples, one as the
> >> default and another if you're using tmpfiles.d functionality added in a
> >> specific version of systemd that's newer than the version shipped with
> >> the stable version of Debian prior to the one you're targeting.
> 
> > Here's an updated version with that change plus some other minor fixes.
> 
> Er, right, helps to rebase first.  Here's the actual patch.

> diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> index b34c183..fa3e5be 100644
> --- a/policy/ch-files.rst
> +++ b/policy/ch-files.rst
> @@ -722,6 +722,70 @@ The name of the files and directories installed by 
> binary packages
>  outside the system PATH must be encoded in UTF-8 and should be
>  restricted to ASCII when it is possible to do so.
>  
> +.. _s-tmpfiles.d:
> +
> +Volatile and temporary files (``tmpfiles.d``)
> +-
> +
> +Some packages require empty directories in ``/var`` or ``/etc``, or

Not shipping these empty directories in the .deb seems like a
regression or a disservice to me. Even for things that might get
deleted because things like our policy or the FHS allows for it (say
stuff under /var/cache), as «dpkg --verify» can be useful. Because of
course, these in addition, can be defined via tmpfiles.d, so that they
can possibly be recreated if needed (until dpkg provides its own
interfaces to do that).

> +symlinks or files with trivial content in ``/var``, to implement their
> +functionality.  Examples include directories under ``/var/cache`` that are
> +writable by the package as cache areas, an initially-empty directory in
> +``/etc`` intended for local overrides added by the local system
> +administrator, or a file in ``/var`` that should default to a symlink
> +elsewhere on the system but may be changed later.
> +
> +Rather than include these symlinks, files, or directories in the binary
> +package or create them in package maintainer scripts, packages should use
> +the ``tmpfiles.d`` mechanism to specify the files and directories that
> +should be created.  This allows associating these files and directories
> +with specific packages (not currently possible when creating them in
> +maintainer scripts),

Well, this association would then only be indirect, instead of being able
to get at them via say «dpkg-query --search» or «dpkg-query --listfiles».

>   and allows local administrators to delete the
> +contents of directories such as ``/var/cache`` with the assurance that
> +``tmpfiles.d`` can recreate the necessary file structure without
> +reinstalling packages or re-running maintainer scripts.

Thanks,
Guillem

Bug#1051371: Post-/usr-merge paths for script interpreters

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> On Wed, 13 Sept 2023 at 04:48, Russ Allbery  wrote:
>> 
>> Control: retitle -1 Post-/usr-merge paths for script interpreters
>> 
>> Simon pointed out that this bug is not yet ready to act on, which
>> was very helpful.  Thank you.  However, presumably the buildds
>> will be /usr-merged at some point in the not-too-distant future,
>> and we do need to decide what to do after that point.

Luca> While that could be said for the original revision, in my view
Luca> that's not really the case for the latest that I posted?

Luca> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

Luca> So I would prefer if this was a clone rather than a
Luca> retitle/repurpose.  Unless I'm missing something, the changes
Luca> linked above should be uncontroversial and simply remove
Luca> excessively normative language in what are essentially
Luca> examples that should have been taken as such - and that
Luca> currently are not. So, could that be taken forward
Luca> independently of the problem you define below?

I agree the above is uncontroversial and would support including it in
policy now.
I don't think it needs seconds because it is non-normative.

(As an aside, reading the summary, I expected to find the patch
something I was not entirely happy with.  I was planning to hold my
nose, and neither support the patch nor object.  However, since you
brought it up again, I read the full patch and find I like the patch a
lot better than your summary of it:-)


signature.asc
Description: PGP signature

Bug#1051371: Post-/usr-merge paths for script interpreters

[Bill Allombert]

On Wed, Sep 13, 2023 at 10:47:48AM +0200, Bill Allombert wrote:
> On Tue, Sep 12, 2023 at 08:48:04PM -0700, Russ Allbery wrote:
> > Control: retitle -1 Post-/usr-merge paths for script interpreters
> > 
> > Simon pointed out that this bug is not yet ready to act on, which was very
> > helpful.  Thank you.  However, presumably the buildds will be /usr-merged
> > at some point in the not-too-distant future, and we do need to decide what
> > to do after that point.
> > 
> > I think the root problem behind this bug is that it is revealing we have
> > not made a decision about /bin and /usr/bin path references in Debian
> > after /usr-merge.  Various people, myself included, made assumptions about
> > what the policy would be, but we never actually decided anything that I am
> > aware of and people's assumptions are not matching.  I think we need to
> > talk about this directly, after which what to do with this bug will
> > probably become obvious.
> 
> Russ, there is a quite related point I do not think the TC addressed 
> directly, 
> but I can easily be mistaken: the default PATH.
> It is currently defined in /etc/login.defs (unless my copy of this file is 
> out of date):
> as
> ENV_SUPATH  
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> ENV_PATHPATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
> 
> so in pratice, tools like 'which' will always favor /usr/bin over /bin
> 
> $ which sh
> /usr/bin/sh
> 
> One of the issue in the past is that reproducible build was broken because 
> different
> build environment lead to different paths. We at least need to address that.

To be specific, I needed to add 
SHELL=/bin/sh GREP=/bin/grep SED=/bin/sed DD=/bin/dd
to configure to get reproducible build. (This was in December 2018).

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1051371: Post-/usr-merge paths for script interpreters

[Luca Boccassi]

On Wed, 13 Sept 2023 at 04:48, Russ Allbery  wrote:
>
> Control: retitle -1 Post-/usr-merge paths for script interpreters
>
> Simon pointed out that this bug is not yet ready to act on, which was very
> helpful.  Thank you.  However, presumably the buildds will be /usr-merged
> at some point in the not-too-distant future, and we do need to decide what
> to do after that point.

While that could be said for the original revision, in my view that's
not really the case for the latest that I posted?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

So I would prefer if this was a clone rather than a retitle/repurpose.
Unless I'm missing something, the changes linked above should be
uncontroversial and simply remove excessively normative language in
what are essentially examples that should have been taken as such -
and that currently are not. So, could that be taken forward
independently of the problem you define below?

> I think the root problem behind this bug is that it is revealing we have
> not made a decision about /bin and /usr/bin path references in Debian
> after /usr-merge.  Various people, myself included, made assumptions about
> what the policy would be, but we never actually decided anything that I am
> aware of and people's assumptions are not matching.  I think we need to
> talk about this directly, after which what to do with this bug will
> probably become obvious.
>
> So far as I can tell, there are three main possibilities:
>
> (a) Although /bin and /usr/bin are merged (and likewise for the other
> merged paths), Debian will continue to require (or at least recommend)
> use of /bin paths for things such as /bin/sh that historically used
> those paths.
>
> (b) Since /bin and /usr/bin (and likewise for the other paths) are merged,
> /bin/sh and /usr/bin/sh are equivalent.  Packages can use whichever
> path they want, and Debian will end up with a mix of both references.
>
> (c) Although /bin and /lib technically work due to the aliasing, they are
> deprecated and everything in Debian should stop using those paths.
> All paths should point to /usr/bin and /usr/lib now.
>
> Although Luca made a few arguments in the direction of (c), my
> understanding of his current patch is that it essentially implements (b)
> for script interpreters, although without encoding into Policy an explicit
> decision between these three options (quite understandably because he was
> trying to deal with a narrower issue).  Several other people were, I
> think, arguing for (a), but I'm not sure if they would continue to do so
> when it's put in these terms.
>
> Policy currently says nothing significant about this (hence most of the
> text so-far discussed being informative) because, up until now, (a) was
> the de facto policy.  If you didn't follow (a), you would get not-found
> errors and your package would have an RC bug because it wouldn't work.
>
> If we do nothing, we'll get (b).  I think that's reasonably obvious, since
> there is no technical factor forcing either (a) or (c).  Both paths will
> work without any noticable difference, so some people will use /bin and
> some people will use /usr/bin.
>
> That said, I would rather make an explicit choice rather than decide by
> default, since otherwise this seems like the kind of thing where people
> are going to get conflicting advice, which is frustrating for everyone.
>
> If someone wants to argue for (a) or (c), I think the biggest problem with
> either of them is an enforcement mechanism.  How are we going to check
> whether packages are following the rules?  Lintian and a bunch of grep
> patterns is sort of an unsatisfying 90% solution, and people will ignore
> it or just not use Lintian.  There is also no technical reason why both
> paths will not work, so people are going to get grumpy about being told
> what to do and some people will view this as makework.  I think any
> proposal to pick (a) or (c) needs to wrestle with that.
>
> I will also say that it will be very hard to convince me that Debian
> should give Policy advice like (a) or (c) but not actually enforce it
> anywhere.  We have a long history to look at for how those sorts of
> mandates go.  Conscientious packagers who read Policy carefully follow the
> rules and go to extra effort to do so, but other people will never see
> that advice or not pay attention to it.  That means that the effort is
> mostly wasted, because no one can rely on the invariant that either (a) or
> (c) is attempting to achieve.  I am not a big fan of asking people to do
> extra work without some clear benefit from doing that work, which mostly
> requires uniformity.
>
> One last point about this decision: although there are a few style
> recommendations in it, Policy is not a style guide.  The point of Policy
> is to describe the things we should or must do, or at least that the
> project as a whole wants to encourage, that have a concrete effect on the
> quality of the 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Wed, 13 Sept 2023 at 15:37, Sam Hartman  wrote:
>
> > "Russ" == Russ Allbery  writes:
>
> I don't know if this needs seconds, but I reviewed all the text and it
> looks good.
> If seconds are required, I second.

Same, in case ownership passes to Russ, seconded/approved/you have my
sword/etc etc