Re: /usr-merge and DEP17 update: what happens next and how you can help
Hi, On 10/10/23 03:16, Helmut Grohne wrote: For one thing, dh_installsystemd generates maintainer scripts for restarting services. Before version 13.11.6, it did not recognize the /usr location. If you were to backport such a package, bookworm's debhelper would not generate the relevant maintainer scripts. You can mitigate this by issuing "Build-Depends: debhelper (>= 13.11.6~)". Thus, you'll be using a backported debhelper (unless the backporter carelessly deletes this dependency). If that would be the only reason to require a backported debhelper, then it would probably be better to make the file move conditional on the target distribution (from "dpkg-parsechangelog -S distribution") -- the intention of that is immediately readable, while a versioned build dependency needs to be explained or a backporter might want to carelessly delete it. gcc does something similar, although that uses "lsb_release -cs", which is technically not entirely correct since it looks at the build system, not the current package, but for the purpose of backports is good enough and avoids corner cases like "stable-proposed-updates" not matching "bookworm" or "bookworm-backports". Simon
Re: /usr-merge and DEP17 update: what happens next and how you can help
On Mon, Oct 09, 2023 at 08:16:32PM +0200, Helmut Grohne wrote: > On Mon, Oct 09, 2023 at 02:10:27PM +0200, Andrea Bolognani wrote: > > Am I missing something? > > Yes, you are and what you are missing really is not obvious, so thanks > for asking! > > For one thing, dh_installsystemd generates maintainer scripts for > restarting services. Before version 13.11.6, it did not recognize the > /usr location. If you were to backport such a package, bookworm's > debhelper would not generate the relevant maintainer scripts. You can > mitigate this by issuing "Build-Depends: debhelper (>= 13.11.6~)". Thus, > you'll be using a backported debhelper (unless the backporter carelessly > deletes this dependency). You mentioned this constraint in your original email, so while I didn't mention it explicitly I was planning on adding the necessary Build-Depends. I was also assuming that a good enough version of debhelper would be backported to bookworm. > For another, we have this generic file loss problem (DEP17 P1). If - in > addition to moving units to /usr - you also restructure your package > between bookworm and trixie (move units between binary packages), then > an upgrade scenario may delete those files even in the presence of > correct Breaks+Replaces. As long as you are sure that you do not rename > any binary packages nor move any units between packages from bookworm to > trixie, this won't apply. Such renames or moves are hard to predict > though. I'm actually hoping that I will be able to get around to a pretty big refactoring of the libvirt package before trixie, so this is kind of a deal breaker for me :) > So if you understand these limitations and are prepared to handle them > for backports, cleaning things up now is fine. If you are not, deferring > that cleanup until after trixie and using dh_movetousr in the interim, > may be the simpler option. Yup, given the situation dh_movetousr definitely feels like the way to go. Thank you for taking the time to explain the situation and, once again, for all your tireless work in this area :) -- Andrea Bolognani Resistance is futile, you will be garbage collected. signature.asc Description: PGP signature
Re: /usr-merge and DEP17 update: what happens next and how you can help
Hi Andrea,
On Mon, Oct 09, 2023 at 02:10:27PM +0200, Andrea Bolognani wrote:
> For libvirt, the upstream build system actually installs systemd
> units under /usr/lib, and we move things around in debian/rules so
> that they end up under /lib in the Debian package:
>
> SRV_MONOLITHIC = libvirt-guests virtlogd virtlockd \
>libvirtd libvirtd-tcp libvirtd-tls virt-guest-shutdown
>
> set -e; for f in $(SRV_MONOLITHIC); do \
> dh_install -p libvirt-daemon-system \
> usr/lib/systemd/system/$${f}* \
> lib/systemd/system/; \
> done
>
> I wouldn't be surprised if other packages did something similar.
This definitely is more common, yes.
> In this case, instead of throwing dh_movetousr into the mix, wouldn't
> it be more sensible to drop the rename part and just follow the
> upstream build system?
In the long run, I definitely agree. In the short term, there are
downsides.
> I guess this could theoretically be problematic for backports, as the
> dh_movetousr approach would guarantee that units still end up in /lib
> on bookworm and older but this wouldn't. On the other hand, hasn't
> systemd been able to load units both from /lib and /usr/lib for
> several releases now? So I would expect that to work somewhat
> transparently.
This is correct. systemd handles both locations since very long.
> Am I missing something? I have to admit that, while I've tried to
> keep tabs on the discussion and all the great work you and other have
> been doing to push things forward, I never quite managed to fully
> absorb the problem space.
Yes, you are and what you are missing really is not obvious, so thanks
for asking!
For one thing, dh_installsystemd generates maintainer scripts for
restarting services. Before version 13.11.6, it did not recognize the
/usr location. If you were to backport such a package, bookworm's
debhelper would not generate the relevant maintainer scripts. You can
mitigate this by issuing "Build-Depends: debhelper (>= 13.11.6~)". Thus,
you'll be using a backported debhelper (unless the backporter carelessly
deletes this dependency).
For another, we have this generic file loss problem (DEP17 P1). If - in
addition to moving units to /usr - you also restructure your package
between bookworm and trixie (move units between binary packages), then
an upgrade scenario may delete those files even in the presence of
correct Breaks+Replaces. As long as you are sure that you do not rename
any binary packages nor move any units between packages from bookworm to
trixie, this won't apply. Such renames or moves are hard to predict
though.
So if you understand these limitations and are prepared to handle them
for backports, cleaning things up now is fine. If you are not, deferring
that cleanup until after trixie and using dh_movetousr in the interim,
may be the simpler option.
Helmut
Re: /usr-merge and DEP17 update: what happens next and how you can help
On 2023-10-09 14:10 +0200, Andrea Bolognani wrote:
> For libvirt, the upstream build system actually installs systemd
> units under /usr/lib, and we move things around in debian/rules so
> that they end up under /lib in the Debian package:
>
> SRV_MONOLITHIC = libvirt-guests virtlogd virtlockd \
>libvirtd libvirtd-tcp libvirtd-tls virt-guest-shutdown
>
> set -e; for f in $(SRV_MONOLITHIC); do \
> dh_install -p libvirt-daemon-system \
> usr/lib/systemd/system/$${f}* \
> lib/systemd/system/; \
> done
>
> I wouldn't be surprised if other packages did something similar.
>
> In this case, instead of throwing dh_movetousr into the mix, wouldn't
> it be more sensible to drop the rename part and just follow the
> upstream build system?
Makes sense to me, but there is one caveat.
> I guess this could theoretically be problematic for backports, as the
> dh_movetousr approach would guarantee that units still end up in /lib
> on bookworm and older but this wouldn't. On the other hand, hasn't
> systemd been able to load units both from /lib and /usr/lib for
> several releases now? So I would expect that to work somewhat
> transparently.
While systemd has supported units both under /lib/systemd/system and
/usr/lib/systemd/system for years, dh_installsystemd has only gained
support for the latter in the latest debhelper upload.
So if you install systemd units there, adding a build-dependency on
debhelper (>= 13.11.6~) is probably advisable, lest backports run into
#1041159[1].
Cheers,
Sven
1. https://bugs.debian.org/1041159
Re: /usr-merge and DEP17 update: what happens next and how you can help
On Sun, Oct 08, 2023 at 10:25:44PM +0200, Helmut Grohne wrote:
> * For many other cases, I propose leaving the upstream install layout
>as is and performing the conversion using a new debhelper component
>that will be called dh_movetousr
>
[...]
>
> * movetousr.ddlist lists packages that probably need to run
>dh_movetousr for some reason:
> + There are other files than systemd units shipped in aliased
> locations.
> + debian/*.install places units to /lib. In this case, consider
> installing units by placing them to debian/*.service if possible
> to benefit from the automatic conversion.
> + An upstream build system hard codes the systemd unit location to
> /lib.
> + ...
>
[...]
>
> Debian Libvirt Maintainers
>libvirt
For libvirt, the upstream build system actually installs systemd
units under /usr/lib, and we move things around in debian/rules so
that they end up under /lib in the Debian package:
SRV_MONOLITHIC = libvirt-guests virtlogd virtlockd \
libvirtd libvirtd-tcp libvirtd-tls virt-guest-shutdown
set -e; for f in $(SRV_MONOLITHIC); do \
dh_install -p libvirt-daemon-system \
usr/lib/systemd/system/$${f}* \
lib/systemd/system/; \
done
I wouldn't be surprised if other packages did something similar.
In this case, instead of throwing dh_movetousr into the mix, wouldn't
it be more sensible to drop the rename part and just follow the
upstream build system?
I guess this could theoretically be problematic for backports, as the
dh_movetousr approach would guarantee that units still end up in /lib
on bookworm and older but this wouldn't. On the other hand, hasn't
systemd been able to load units both from /lib and /usr/lib for
several releases now? So I would expect that to work somewhat
transparently.
Am I missing something? I have to admit that, while I've tried to
keep tabs on the discussion and all the great work you and other have
been doing to push things forward, I never quite managed to fully
absorb the problem space.
--
Andrea Bolognani
Resistance is futile, you will be garbage collected.
signature.asc
Description: PGP signature
/usr-merge and DEP17 update: what happens next and how you can help
Hi, Quite a bit has happened and we're more and more moving from discussion into action. I'd like to use this opportunity to thank all the invisible voices who've given me useful feedback. Your private messages, BoF feedback, and other forms have reached me even if I did not answer all of them individually. Please bear with one more mail that also is too long. ;) The file move moratorium issued by the CTTE still is in effect technically. It is issued as a recommendation and that means, we technically can violate it given reasons. I believe there are reasons and hope we can formally lift it soon. A major blocker to lifting it has been that buildd chroots have been unmerged, but those will hopefully be merged as you read this, see https://lists.debian.org/debian-devel/2023/10/msg00024.html. On a practical level, just lifting the moratorium would break lots of stuff. Instead, it needs to be lifted progressively. So the moratorium will no longer be a CTTE moratorium, but a transition one and will cover less and less aspects over time. Its state shall be tracked on https://wiki.debian.org/UsrMerge. Since my last episode, I've focused on moving systemd units. The reason is that they constitute a big chunk of the problem (if you judge it by the number of affected packages) and moving them technically poses little problems in the face of unmerged buildds and unmerged d-i. In particular, debhelper supports generating maintainer scripts for units installed to /usr since version 13.11.6. As for moving files, I propose three concurrent ways: * If your units are stored as debian/*.service and others and installed by dh_installsystemd or dh_systemd_start, please just wait. I want to change these helpers, see https://salsa.debian.org/debian/debhelper/-/merge_requests/113. Most affected packages can be converted using a binNMU. * If your units are installed to the location given by `pkg-config --variable systemdsystemunitdir systemd`, please also wait. I want to change systemd.pc, see https://salsa.debian.org/systemd-team/systemd/-/merge_requests/218, but this currently causes too many FTBFS. * For many other cases, I propose leaving the upstream install layout as is and performing the conversion using a new debhelper component that will be called dh_movetousr and can be enabled by depending on dh-sequence-movetousr, see https://salsa.debian.org/debian/debhelper/-/merge_requests/112. Note that this helper intentionally is opt-in. It should not be blindly applied to all packages, but used where we understand that it does not cause breakage. In a number of cases, using it will require mitigations from DEP17 to be implemented as well. These changes have two significant implications for affected packages that you should be aware of. For one thing, all of these violate the CTTE moratorium and when affected packages are restructured, they may incur the file loss scenario (DEP17 P1) that the moratorium was meant to prevent. Indeed, the upload of dhcpcd5 to bookworm accidentally did that. To avoid this from causing file loss, please upload restructuring changes (i.e. those where packages are renamed or files move between packages) to experimental first and wait three days. In case of problems, https://salsa.debian.org/helmutg/dumat shall automatically file an RC bug for your package. At the time of this writing, I still proof read every bug before filing, but the intention is to let this proceed without human intervention. The other aspect is that such moves may break backports. Therefore backports should continue shipping their units in the same location as the relevant base suite (most probably not /usr). To ease this, backports of debhelper shall revert the change in installation location for dh_installsystemd and dh_systemd_start and they shall also turn dh_movetousr into a noop. Also the systemd.pc change shall not be backported. So if you use one these mechanisms, backports should just work. However, if you change e.g. --prefix=/ to --prefix=/usr to move all files, please pay attention to revert this for backports. I know you love dd-lists, so I have prepared some: * nochange.ddlist lists 406 packages where a rebuild is sufficient. This is almost half of the source packages that ship files in aliased locations! * movetousr.ddlist lists packages that probably need to run dh_movetousr for some reason: + There are other files than systemd units shipped in aliased locations. + debian/*.install places units to /lib. In this case, consider installing units by placing them to debian/*.service if possible to benefit from the automatic conversion. + An upstream build system hard codes the systemd unit location to /lib. + ... Once we got the ball rolling for systemd units, my next focus will be transitively essential packages and risky corner cases. I shall be sending patches for affected packages and move all files in the
