Need help with Multi-Arch in systemd
Hi,
a couple of days ago, the following bug report was filed against systemd
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990547
I'm quoting the relevant parts
I noticed that systemd-machined was failing to start on an arm64
box. This box had armhf enabled, and turns out systemd had been
cross-graded over to systemd:armhf[*]. It still had
systemd-container:arm64 installed, so now I had an arm64
/lib/systemd/systemd-machine, but an armhf
/lib/systemd/libsystemd-shared-244.so.
libsystemd-shared-244.so is from the systemd package. Since systemd is
marked Multi-Arch: foreign, systemd:armhf was able to incorrectly
satisfy systemd-container:arm64's dependency on systemd.
The systemd package provides a package private library
/lib/systemd/libsystemd-shared-{source:Version}.so, which is used by
binaries that are built from src:systemd. Some of those binaries are
split into separate binary packages, like systemd-container.
Since both systemd is marked as Multi-Arch: foreign, it can happen that
we get this architecture mismatch.
Asking on #debian-systemd, Marco d'Itri suggested, that we move
libsystemd-shared into a separate binary package. This would only help,
if we moved libsystemd-shared into a Multi-Arch location (which means,
we'd have to carry a patch against upstream). It would also mean, that
on every new upstream release, systemd would have to go through NEW.
So I'm not very keen on doing that.
Option 2 would be to drop Multi-Arch: foreign from systemd. This would
mean, that packages like libpam-systemd/libnss-systemd can no longer be
installed for a foreign architecture (even though those modules only use
architecture independent interfaces of systemd).
Option 3 is something I came up with after reading the Multi-Arch
related wiki pages:
https://salsa.debian.org/systemd-team/systemd/-/commit/c379461a14dcd13e0b625bd8faabbcf7fb3d19d6
It would mean marking systemd as Multi-Arch: allowed. And packages that
only use architecture interfaces of systemd would have to use the :any
annotation.
I would appreciate feedback, if option 3 is proper solution or not, or
if there are other, better alternatives. If we go with option 3, should
I inform all rdeps of systemd to update their dependency accordingly,
i.e. do a MBF?
Currently, I only see interpreters like perl, use M-A: allowed, so I'm
not sure if I'm misusing this feature.
Regards,
Michael
OpenPGP_signature
Description: OpenPGP digital signature
Re: Need help with Multi-Arch in systemd
Hi Michael,
thanks for reaching out!
On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote:
> a couple of days ago, the following bug report was filed against systemd
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990547
Imo, this is bug should be serious. In essence, it is a missing
dependency and we track missing dependencies as serious.
At the same time, I find it so unlikely to happen in practice combined
with the difficulty of finding a good solution that I'd propose
bullseye-ignore.
> Asking on #debian-systemd, Marco d'Itri suggested, that we move
> libsystemd-shared into a separate binary package. This would only help, if
> we moved libsystemd-shared into a Multi-Arch location (which means, we'd
> have to carry a patch against upstream). It would also mean, that on every
> new upstream release, systemd would have to go through NEW.
> So I'm not very keen on doing that.
I concur with Marco here and I argue that splitting the library is my
preferred solution. I confirm that you'd need to move the library for
this to work. For the other points I do not follow. I think it would be
ok to move the library using code inside debian/. Is there any reason
why you ruled out that option?
I also disagree with the need to go through NEW more than once. The new
package could quite simply be named libsystemd-private and lack a
.symbols and .shlibs file. Internal users would always use (=
${binary:Version}) anyway. The package description would declare that
any external dependency on libsystemd-private is a bug. Prior art:
libbinutils/binutils-dev.
One could argue that packaging a shared library like that is a policy
violation. If that is the case, then the current bundling is a policy
violation as well. So meh. It really sounds like a fair compromise to
me.
I am actually wondering now whether we should have a separate archive
section for "private" packages. I would define it as follows:
A private package is an implementation detail used by a single source.
Binary packages built from a one source package must not depend on
private packages built from another. Dependencies on private packages
should use tight version requirements (e.g. (= ${binary:Version}).
I suppose that a number of *-common and *-data packages as well as
gcc-N-base could be moved to such a section. User interfaces for package
managers could hide private packages by default.
Regardless of whether we add an archive section, lintian could carry a
list of private packages and enforce the no-dependency rule.
> Option 2 would be to drop Multi-Arch: foreign from systemd. This would mean,
> that packages like libpam-systemd/libnss-systemd can no longer be installed
> for a foreign architecture (even though those modules only use architecture
> independent interfaces of systemd).
That would break a pile of use cases and cause a lot of pain downstream.
Please don't.
> Option 3 is something I came up with after reading the Multi-Arch related
> wiki pages:
> https://salsa.debian.org/systemd-team/systemd/-/commit/c379461a14dcd13e0b625bd8faabbcf7fb3d19d6
> It would mean marking systemd as Multi-Arch: allowed. And packages that only
> use architecture interfaces of systemd would have to use the :any
> annotation.
Yeah, this technically works, but it causes a lot of maintenance effort,
because you get to hunt down every single systemd dependency in the
archive for years. I think our time is worth more than the cost of
introducing a new binary package into the archive.
> I would appreciate feedback, if option 3 is proper solution or not, or if
> there are other, better alternatives. If we go with option 3, should I
> inform all rdeps of systemd to update their dependency accordingly, i.e. do
> a MBF?
The problem I see here is less with annotating all deps once. It is more
the constant effort it incurs. It is not obvious that you should
annotate your systemd dependency :any. People will forget that when
adding systemd dependencies. What I take issue with is the permanent
cost that we keep in the long run.
Hope this helps
Helmut
Re: Need help with Multi-Arch in systemd
Am 09.07.21 um 10:28 schrieb Helmut Grohne: I concur with Marco here and I argue that splitting the library is my preferred solution. I confirm that you'd need to move the library for this to work. For the other points I do not follow. I think it would be ok to move the library using code inside debian/. Is there any reason why you ruled out that option? I also disagree with the need to go through NEW more than once. The new package could quite simply be named libsystemd-private and lack a .symbols and .shlibs file. Splitting out libsystemd-shared (once) will make PID1 very brittle. It can lead to situations where old systemd + new libsystemd-private is installed. If the installation is aborted at this point, you have an unbootable system. This is why I think it's not a viable approach. OpenPGP_signature Description: OpenPGP digital signature
Re: Need help with Multi-Arch in systemd
Am 09.07.21 um 10:28 schrieb Helmut Grohne: On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote: Option 2 would be to drop Multi-Arch: foreign from systemd. This would mean, that packages like libpam-systemd/libnss-systemd can no longer be installed for a foreign architecture (even though those modules only use architecture independent interfaces of systemd). That would break a pile of use cases and cause a lot of pain downstream. Please don't. Can you iterate on that a bit? I was looking for use cases, but couldn't come up with them. Option 3 is something I came up with after reading the Multi-Arch related wiki pages: https://salsa.debian.org/systemd-team/systemd/-/commit/c379461a14dcd13e0b625bd8faabbcf7fb3d19d6 It would mean marking systemd as Multi-Arch: allowed. And packages that only use architecture interfaces of systemd would have to use the :any annotation. Yeah, this technically works, but it causes a lot of maintenance effort, because you get to hunt down every single systemd dependency in the archive for years. I think our time is worth more than the cost of introducing a new binary package into the archive. As said, option 1 is not something I really consider a viable option for the aforementioned reasons. I would appreciate feedback, if option 3 is proper solution or not, or if there are other, better alternatives. If we go with option 3, should I inform all rdeps of systemd to update their dependency accordingly, i.e. do a MBF? The problem I see here is less with annotating all deps once. It is more the constant effort it incurs. It is not obvious that you should annotate your systemd dependency :any. People will forget that when adding systemd dependencies. What I take issue with is the permanent cost that we keep in the long run. So, packages need to decide whether they use architecture independent interfaces of systemd or not and annotate it accordingly. But they only need to do that once. Isn't this the same as with any python, ruby or perl dependency? Maybe having something like the inverse of M-A: allowed would be useful? Instead of having packages explicitly opt-in via a :any annotation, they can opt out and request the same architecture even if a package is marked as M-A: foreign? So the select few cases which use an architecture dependent interface of M-A: foreign package would use something like pkg:arch ? OpenPGP_signature Description: OpenPGP digital signature
Re: Need help with Multi-Arch in systemd
* Michael Biebl [2021-07-09 11:01]: Splitting out libsystemd-shared (once) will make PID1 very brittle. It can lead to situations where old systemd + new libsystemd-private is installed. If the installation is aborted at this point, you have an unbootable system. Helmut suggested a tightly versioned dependency, so such a state would never be consistent. And given that system updates are not atomic transactions, I'm pretty confident you can interrupt *any* update of a boot-critical package at a point that leaves the system unbootable. Cheers Timo -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: Need help with Multi-Arch in systemd
Am 09.07.2021 um 11:37 schrieb Timo Röhling: * Michael Biebl [2021-07-09 11:01]: Splitting out libsystemd-shared (once) will make PID1 very brittle. It can lead to situations where old systemd + new libsystemd-private is installed. If the installation is aborted at this point, you have an unbootable system. Helmut suggested a tightly versioned dependency, so such a state would never be consistent. That tightly versioned dependency doesn't help unfortunately. There is still a time window between the new libsystemd-shared and the new systemd being unpacked. Michael
Re: Need help with Multi-Arch in systemd
* Michael Biebl [2021-07-09 12:29]: That tightly versioned dependency doesn't help unfortunately. There is still a time window between the new libsystemd-shared and the new systemd being unpacked. There's also a time window between files in /usr/bin and files in /usr/lib being replaced. If you reboot your system with an unfinished/interrupted upgrade, you cannot assume a consistent or bootable state. Packaging all stuff in the same package *may* reduce the above time window somewhat, but it will not eliminate the underlying problem, so it is a non-solution. There may be other reasons to object Helmut's proposal, but this is not it. Cheers Timo -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: Need help with Multi-Arch in systemd
Am 09.07.2021 um 13:01 schrieb Timo Röhling: * Michael Biebl [2021-07-09 12:29]: That tightly versioned dependency doesn't help unfortunately. There is still a time window between the new libsystemd-shared and the new systemd being unpacked. There's also a time window between files in /usr/bin and files in /usr/lib being replaced. If you reboot your system with an unfinished/interrupted upgrade, you cannot assume a consistent or bootable state. Packaging all stuff in the same package *may* reduce the above time window somewhat, but it will not eliminate the underlying problem, so it is a non-solution. There may be other reasons to object Helmut's proposal, but this is not it. I don't agree. The time window between dpkg unpacking a single package and ending up in a inconsistent state is neglibible to apt updating multiple packages during a dist-upgrade. Let me be blunt here: I'm not willing to compromise on robustness here.
Re: Need help with Multi-Arch in systemd
On Fri, 2021-07-09 at 12:29:19 +0200, Michael Biebl wrote: > Am 09.07.2021 um 11:37 schrieb Timo Röhling: > > * Michael Biebl [2021-07-09 11:01]: > > > Splitting out libsystemd-shared (once) will make PID1 very brittle. > > > It can lead to situations where old systemd + new libsystemd-private > > > is installed. If the installation is aborted at this point, you have > > > an unbootable system. > > Helmut suggested a tightly versioned dependency, so such a state > > would never be consistent. > That tightly versioned dependency doesn't help unfortunately. There is still > a time window between the new libsystemd-shared and the new systemd being > unpacked. If the private library has no backwards or forward compatibility (due to the SONAME used) the time window where the library does not match the expectations of the stuff linked to it might indeed be too big. But the reported bug is just a symptom of a bigger issue. This problem already exists for any of the other binary packages built against this private shared library, there's a time-window where they will not work if the installation gets interrupted or fails, compared with public shared libraries. This implies to me the correct solution is a name-versioned package, even though that seems cumbersome given our current archive handling practices this is IMO the only correct solution. The alternative of using Multi-Arch:allowed is a workaround for this packaging problem. Another solution might be to request upstream to stabilize this library? Yet another solution, which seems suboptimal, might be to statically link it, but the shared library seems rather big, but perhaps if the programs linked only use parts of it, that might not be too bad? Thanks, Guillem
Re: Need help with Multi-Arch in systemd
Hi Michael,
in your other mail, you gave a very good reason for keeping the systemd
binary and libsystemd-shared.so in the same binary package. That
improves my understanding of why you favour option 3.
On Fri, Jul 09, 2021 at 11:31:15AM +0200, Michael Biebl wrote:
> Am 09.07.21 um 10:28 schrieb Helmut Grohne:
> >
> > On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote:
>
> > > Option 2 would be to drop Multi-Arch: foreign from systemd. This would
> > > mean,
> > > that packages like libpam-systemd/libnss-systemd can no longer be
> > > installed
> > > for a foreign architecture (even though those modules only use
> > > architecture
> > > independent interfaces of systemd).
> >
> > That would break a pile of use cases and cause a lot of pain downstream.
> > Please don't.
>
> Can you iterate on that a bit? I was looking for use cases, but couldn't
> come up with them.
I guess you already know what comes next: cross building. 4700 packages
include systemd in their Build-Depends. Many of them would immediately
become cross bd-uninstallable. For others, the installation set would be
satisfied with a foreign systemd, which would quite simply fail to work.
Any kind of mixed-arch system where you want to install a daemon that
depends on systemd for a foreign architecture would also be affected.
Someone attempting to do so might accidentally cross grade systemd in
doing so.
> As said, option 1 is not something I really consider a viable option for the
> aforementioned reasons.
>From my pov, the only reason left against 1 is partial upgrades breaking
the installation. It's a strong one anyhow.
> So, packages need to decide whether they use architecture independent
> interfaces of systemd or not and annotate it accordingly. But they only need
> to do that once. Isn't this the same as with any python, ruby or perl
> dependency?
Yes. For perl and python, we haven't even annotated half of the relevant
Depends with :any yet. Experience tells, the annotation is not gonna
happen.
> Maybe having something like the inverse of M-A: allowed would be useful?
> Instead of having packages explicitly opt-in via a :any annotation, they can
> opt out and request the same architecture even if a package is marked as
> M-A: foreign?
> So the select few cases which use an architecture dependent interface of
> M-A: foreign package would use something like pkg:arch ?
If I were to design Multi-Arch today, I would likely agree. The
transition cost to get there seems prohibitively high though. I've tried
hard to change the spec and I did not succeed (on the Multi-Arch
interpreter problem). Yet, we devised a workaround for that issue. So
you made me thinking, can we somehow implement this with our current
spec? The most important requirements seem to be:
* libsystemd-shared.so and /sbin/systemd need to reside in the same
binary package.
* It shall be possible to depend on libsystemd-shared.so for a
particular architecture.
* A dependency on "systemd" should request a native systemd.
Now let's do something stupid. Rename systemd to systemd-core (taking
all files with it, please refrain from discussing the name unless you
seriously consider doing this). Mark it Multi-Arch: allowed. Add a new,
empty binary package systemd. It is Multi-Arch: foreign and depends on
systemd-core:any. This approach would technically satisfy all three
requirements, but it feels a little crazy to me.
Another possibility (kudos to David Kalnischkies) would be exploiting
something I might call a loophole in the Multi-Arch spec. While we don't
currently use arch-qualified dependencies in the archive, the spec
considers them and dpkg and apt support them (somewhat). So in theory,
we could say that systemd-container Depends: systemd:${DEB_HOST_ARCH}.
I'm not sure whether all resolvers agree on this, but I'd expect this
dependency to pull the right instance (despite being marked Multi-Arch:
foreign). Before jumping onto this, please consider David's warning:
here be dragons.
I'm relaying another question from David: Assuming we'd solve the
libsystemd-shared.so issue, would systemd-container:arch1 actually work
with systemd:arch2 or are there more aspects where the same architecture
is assumed?
As we already are going crazy in the search for solutions to this, let
me propose a scary one. I have no idea whether this works in practice.
Split out libsystemd-shared.so into systemd-private like before, but at
the same time ship it in systemd. That causes a conflict of course, so
systemd-private Conflicts: systemd:${DEB_HOST_ARCH} and systemd
Conflicts: systemd-private:${DEB_HOST_ARCH} as well as Provides:
systemd-private:${DEB_HOST_ARCH}. These Conflicts must be arch-qualified
as otherwise they're interpreted as any instance. We already have
arch-qualified Conflicts in the archive, so this aspect likely works.
I'm less sure about arch-qualified Provides though. In any case, the
(common) native case would be systemd without systemd-private
Re: Need help with Multi-Arch in systemd
* Michael Biebl [2021-07-09 13:24]: Let me be blunt here: I'm not willing to compromise on robustness here. Well, I have had my say and ultimately, it's your package and your decision, of course. Cheers Timo -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: Need help with Multi-Arch in systemd
Hi Guillem,
thanks for your feedback
Am 09.07.21 um 13:46 schrieb Guillem Jover:
If the private library has no backwards or forward compatibility (due
to the SONAME used) the time window where the library does not match
the expectations of the stuff linked to it might indeed be too big.
The libsystemd-shared library is considered an implementation detail and
indeed has no guaranteed backwards or forward compatibility.
The soname is bumped when the first rc1 release is made (e.g v249-rc1 →
libsystemd-shared-249.so) and there might even be incompatible changes
between the rc1 and the final release.
But the reported bug is just a symptom of a bigger issue. This problem
already exists for any of the other binary packages built against this
private shared library, there's a time-window where they will not work
if the installation gets interrupted or fails, compared with public
shared libraries.
This observation is correct I'd say. Currently we have the following
split-off binary packages which ship binaries that link against
libsystemd-shared:
systemd-container (4 binaries in $PATH, 7 services, 11 total)
systemd-coredump (1 binary in $PATH, 1 service, 2 total)
systemd-journal-remote (3 services, 3 total)
systemd-timesyncd (1 service, 1 total)
(I'll exclude systemd-tests, as this is a special case)
The main difference here, is that none of those binaries is really
critical for the runtime of the system.
An unbootable system though is one of the worst things that can happen.
Which is why I'm really reluctant to split off libsystemd-shared from
systemd and hopefully also explains why in general I'm not super keen on
chopping up src:systemd unnecessarily.
This implies to me the correct solution is a name-versioned package,
even though that seems cumbersome given our current archive handling
practices this is IMO the only correct solution.
A name-versioned package would certainly be better then an unversioned
libsystemd-shared package. It still would make PID 1 a bit more brittle
though (see e.g. my comment regard rc releases).
It would also need patching of the build system, to override the rpath,
which would have to be multi-arch aware.
$ find . -name meson.build | xargs grep rpath | wc -l
123
This would be a significant patch with a lot of ongoing churn and
maintenance effort. I'm not sure if I'm willing or even able to do that.
Another solution might be to request upstream to stabilize this library?
The point of libsystemd-shared is to be an internal implementation
detail where ABI can be broken. There are certainly no plans to change
that upstream and I don't think I can convince upstream otherwise on
this matter.
Yet another solution, which seems suboptimal, might be to statically
link it, but the shared library seems rather big, but perhaps if the
programs linked only use parts of it, that might not be too bad?
So, option 4).
I tried that for v249. It's doable, but as this there no upstream
support for that, it requires a patch which is significant and would
require ongoing maintenance effort as well.
Regarding the numbers:
systemd-container (11 binaries)
Installed-Size: [-1278-] {+5644+}
systemd-coredump (2 binaries)
Installed-Size: [-276-] {+1106+}
systemd-journal-remote (3 binaries)
Installed-Size: [-336-] {+1218+}
systemd-timesyncd (1 binary)
Installed-Size: [-209-] {+595+}
I think the increase in size is significant.
More importantly, I'd need help maintaining this patch going forward
I acknowledge, that option 2 and 3 makes Helmut's work on cross-building
harder, although I think that option 3 would be preferrable to 2, as it
as least leaves the door open for making cross-building possible.
So, unless we get a :arch annotation that can be used for M-A:foreign
packages, maybe the best option is
option 5)
do nothing?
I mean, we shipped the M-A: foreign notation for systemd since 2014.
If there is one bug report regarding such an architecture mismatch in 7
years, maybe the most pragmatic approach is to simply ignore it, given
the downsides of the alternatives? I'm not sure what the best course of
action is here.
Regards,
Michael
OpenPGP_signature
Description: OpenPGP digital signature
Re: Need help with Multi-Arch in systemd
Quoting Helmut Grohne (2021-07-09 14:24:01)
> Another possibility (kudos to David Kalnischkies) would be exploiting
> something I might call a loophole in the Multi-Arch spec. While we don't
> currently use arch-qualified dependencies in the archive, the spec considers
> them and dpkg and apt support them (somewhat). So in theory, we could say
> that systemd-container Depends: systemd:${DEB_HOST_ARCH}. I'm not sure
> whether all resolvers agree on this, but I'd expect this dependency to pull
> the right instance (despite being marked Multi-Arch: foreign). Before jumping
> onto this, please consider David's warning: here be dragons.
No, not all solvers agree on this. Imagine this situation:
Package: pkga
Architecture: amd64
Depends: pkgb:i386
Multi-Arch: no
Package: pkgb
Architecture: i386
Multi-Arch: foreign
dose3, apt and dpkg agree that pkga can be installed because its dependency is
satisfied by pkgb. But if we change pkgb:i386 to pkgb:amd64 to end up with
this:
Package: pkga
Architecture: amd64
Depends: pkgb:amd64
Multi-Arch: no
Package: pkgb
Architecture: i386
Multi-Arch: foreign
Then dose3 can satisfy the dependency of pkg while apt and dpkg cannot. This is
because when building the cudf representation of above two packages, dose3
considers pkg to provide pkgb for all architectures, including amd64, because
it is marked as m-a:foreign.
Thanks!
cheers, josch
signature.asc
Description: signature
Re: Need help with Multi-Arch in systemd
> "Helmut" == Helmut Grohne writes: Helmut> Looks like this leaves us between a rock and a hard Helmut> place. None of the options seems particularly attractive to Helmut> me at this point. We seem to be in a brainstorming space here, throwing out half baked ideas because none of us has a great answer. So, I'm going to join the party. Are there games we could play with prerm scripts in a systemd-private package to make sure that the old systemd private library sticks around until the upgrade finishes when upgrading systemd-private? --Sam
Re: Need help with Multi-Arch in systemd
On Fri, Jul 09, 2021 at 10:28:57AM +0200, Helmut Grohne wrote:
>...
> I also disagree with the need to go through NEW more than once. The new
> package could quite simply be named libsystemd-private and lack a
> .symbols and .shlibs file. Internal users would always use (=
> ${binary:Version}) anyway. The package description would declare that
> any external dependency on libsystemd-private is a bug. Prior art:
> libbinutils/binutils-dev.
>...
The prior art of libbinutils/binutils-dev is that instead of shared
linking, users are linking to the static version to the private library
(see e.g. #964536).
I don't understand what benefits this switch to static linking gives,
except for making security support harder.
> Helmut
cu
Adrian
Re: Need help with Multi-Arch in systemd
Hi Michael, I'm not yet fully convinced that we're out of options, but let's for a moment assume we were. On Fri, Jul 09, 2021 at 10:26:43PM +0200, Michael Biebl wrote: > So, unless we get a :arch annotation that can be used for M-A:foreign > packages, maybe the best option is Given Johannes' reply, I think that :arch annotations can be a strict improvement to the status quo. dose will ignore them, so you aren't worse off with it. apt and dpkg agree to honour them, so they'd practically solve the symptoms (despite leading dose into non-solutions to certain satisfiability issues that are mostly relevant to cross building only). The one question here is whether those are the semantics that we want for this corner case of multiarch. A while ago, David changed apt to make it stop behaving like dose and instead make it behave like dpkg. That looks like we had consensus back then. Did we document it anywhere? > option 5) > do nothing? > > I mean, we shipped the M-A: foreign notation for systemd since 2014. > If there is one bug report regarding such an architecture mismatch in 7 > years, maybe the most pragmatic approach is to simply ignore it, given the > downsides of the alternatives? I'm not sure what the best course of action > is here. If :arch qualifiers are an option, I think they are a strict improvement over doing nothing. That's something I couldn't convincingly say about any other option. In any case, I don't think fixing this in time for bullseye is reasonable, so we do have some time figuring out whether :arch is a solution. Helmut
Re: Need help with Multi-Arch in systemd
Hi Michael, Michael Biebl ezt írta (időpont: 2021. júl. 9., P, 22:42): > > Hi Guillem, > > thanks for your feedback > > Am 09.07.21 um 13:46 schrieb Guillem Jover: > > If the private library has no backwards or forward compatibility (due > > to the SONAME used) the time window where the library does not match > > the expectations of the stuff linked to it might indeed be too big. > > The libsystemd-shared library is considered an implementation detail and > indeed has no guaranteed backwards or forward compatibility. > The soname is bumped when the first rc1 release is made (e.g v249-rc1 → > libsystemd-shared-249.so) and there might even be incompatible changes > between the rc1 and the final release. > > > But the reported bug is just a symptom of a bigger issue. This problem > > already exists for any of the other binary packages built against this > > private shared library, there's a time-window where they will not work > > if the installation gets interrupted or fails, compared with public > > shared libraries. > > This observation is correct I'd say. Currently we have the following > split-off binary packages which ship binaries that link against > libsystemd-shared: > > systemd-container (4 binaries in $PATH, 7 services, 11 total) > systemd-coredump (1 binary in $PATH, 1 service, 2 total) > systemd-journal-remote (3 services, 3 total) > systemd-timesyncd (1 service, 1 total) > > (I'll exclude systemd-tests, as this is a special case) > > The main difference here, is that none of those binaries is really > critical for the runtime of the system. > An unbootable system though is one of the worst things that can happen. > Which is why I'm really reluctant to split off libsystemd-shared from > systemd and hopefully also explains why in general I'm not super keen on > chopping up src:systemd unnecessarily. > > > This implies to me the correct solution is a name-versioned package, > > even though that seems cumbersome given our current archive handling > > practices this is IMO the only correct solution. > > A name-versioned package would certainly be better then an unversioned > libsystemd-shared package. It still would make PID 1 a bit more brittle > though (see e.g. my comment regard rc releases). > It would also need patching of the build system, to override the rpath, > which would have to be multi-arch aware. > > $ find . -name meson.build | xargs grep rpath | wc -l > 123 > > This would be a significant patch with a lot of ongoing churn and > maintenance effort. I'm not sure if I'm willing or even able to do that. IMO what Guillem recommends, i.e. the name-versioned libsystemd package with versioned shared library name is still the cleanest and so far the only reasonably reliable solution. IMO going through NEW in every few months could be an acceptable cost especially since I was quite happy with the pace at which NEW is processed recently. I don't maintain systemd in Ubuntu anymore, but if I were, I'd be happy to accept this increased maintenance burden for the sake of having a clean solution for the Multi-Arch problem. If release candidates will be uploaded only to experimental then the incompatible changes won't cause problems. Even if RC-s are uploaded to unstable and a rarely occurring incompatible change takes place then the library package name can be bumped again. After we discussed this topic on #debian devel on 2020-05-05 I even started implementing the solution but did not finish it because it was not an important problem from Ubuntu's POV. I think the rpath usage search counted a lot of hits which don't have to be changed and maybe upstream would be willing to accept the patch. Cheers, Balint
Re: Need help with Multi-Arch in systemd
Helmut Grohne wrote: > So you made me thinking, can we somehow implement this with our > current spec? The most important requirements seem to be: > > * libsystemd-shared.so and /sbin/systemd need to reside in the same >binary package. > * It shall be possible to depend on libsystemd-shared.so for a >particular architecture. > * A dependency on "systemd" should request a native systemd. > > Now let's do something stupid. Rename systemd to systemd-core (taking > all files with it, please refrain from discussing the name unless you > seriously consider doing this). Mark it Multi-Arch: allowed. Add a new, > empty binary package systemd. It is Multi-Arch: foreign and depends on > systemd-core:any. This approach would technically satisfy all three > requirements, but it feels a little crazy to me. This seems like a rather reasonable approach, actually. It's a little unusual, but it has all the advantages of making systemd multi-arch aware while not creating the trap of making a `systemd` dependency do the wrong thing, because `systemd` has the right multi-arch dependency on `systemd-core:any`. The handful of packages that really do need a same-arch dependency on systemd (those that are part of systemd itself and need libsystemd-shared) could depend on `systemd-core` directly, and everything else can continue depending on systemd with no transition required. And there's only one trip through NEW, once.
Re: Need help with Multi-Arch in systemd
On Thu, 08 Jul 2021 at 23:03:48 +0200, Michael Biebl wrote: > [a separate libsystemd-shared-249 .deb] would also mean, that on every > new upstream release, systemd would have to go through NEW It seems like we're rejecting a good technical solution because social/organisational factors block it (namely, new binary packages triggering manual review from the ftp team even if there has not been any significant change in how the package is organised, resulting in manual review being artificially frequent for libraries that happen to break ABI a lot, but infrequent for packages that aren't libraries or don't break ABI). This seems like a side-effect of the ftp team's two gatekeeping roles (legal review and managing the namespaces of source and binary package names) both having the dak override file as their implementation, rather than necessarily anything that was designed or intended. Would it be feasible for dak to have a list of binary package name regexes mapped to a source package and a section/priority, and auto-accept packages from the given source package that match the regex, assigning the given section/priority, without manual action? That would let the ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/ in libs/optional, for example. It seems like this would also be good for src:linux, where ABI breaks are often tied to security fixes that should enter the archive ASAP. smcv
Re: Need help with Multi-Arch in systemd
* Simon McVittie [2021-07-14 11:59]: Would it be feasible for dak to have a list of binary package name regexes mapped to a source package and a section/priority, and auto-accept packages from the given source package that match the regex, assigning the given section/priority, without manual action? That would let the ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/ in libs/optional, for example. +1 Especially considering that library packages are subject to transitions anyway, so they already receive much more scrutiny than, say, an updated Python module. Cheers Timo -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: Need help with Multi-Arch in systemd
On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote: > Asking on #debian-systemd, Marco d'Itri suggested, that we move > libsystemd-shared into a separate binary package. This would only help, if > we moved libsystemd-shared into a Multi-Arch location (which means, we'd > have to carry a patch against upstream). It would also mean, that on every > new upstream release, systemd would have to go through NEW. > So I'm not very keen on doing that. Why do you think? libsystemd-shared is a perfectly valid package name and it does not change. However, if you call it libsystemd-shared-249, it is supposed to be stable. Bastian -- Is truth not truth for all? -- Natira, "For the World is Hollow and I have Touched the Sky", stardate 5476.4.
Re: Need help with Multi-Arch in systemd
On Wed, Jul 14, 2021 at 11:59:11AM +0100, Simon McVittie wrote: > It seems like this would also be good for src:linux, where ABI breaks > are often tied to security fixes that should enter the archive ASAP. As security updates are hand approved, accepting by NEW does not help that much. Bastian -- Peace was the way. -- Kirk, "The City on the Edge of Forever", stardate unknown
Re: Need help with Multi-Arch in systemd
On Wed, 2021-07-14 at 11:59:11 +0100, Simon McVittie wrote: > On Thu, 08 Jul 2021 at 23:03:48 +0200, Michael Biebl wrote: > > [a separate libsystemd-shared-249 .deb] would also mean, that on every > > new upstream release, systemd would have to go through NEW > > It seems like we're rejecting a good technical solution because > social/organisational factors block it (namely, new binary packages > triggering manual review from the ftp team even if there has not been > any significant change in how the package is organised, resulting in > manual review being artificially frequent for libraries that happen to > break ABI a lot, but infrequent for packages that aren't libraries or > don't break ABI). Yes. I was mentioning exactly this the other day on the #debian-bootstrap IRC channel. In addition having this automatic support could make life easier for many other potential packages. > This seems like a side-effect of the ftp team's two gatekeeping roles > (legal review and managing the namespaces of source and binary package > names) both having the dak override file as their implementation, rather > than necessarily anything that was designed or intended. Yes, plus section and priority-spaces. But then, I don't see why a binary package rename should trigger a new legal audit. > Would it be feasible for dak to have a list of binary package name > regexes mapped to a source package and a section/priority, and auto-accept > packages from the given source package that match the regex, assigning > the given section/priority, without manual action? That would let the > ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/ > in libs/optional, for example. What I had in mind was that DAK would gain support for automatic ACCEPT of binary package renames due to SONAME version bumps, something like this: * If the new bin:lib: - replaces an existing bin:lib from the same source, where is lower than . - contains a shared library mapping to that package name. - is in section */libs or */oldlibs. * Then → auto-ACCEPT, pre-filling the new section/priority from the old binary package. * Otherwise → NEW. I guess it could potentially be further extended later on to cover other safe non shared library cases. But if that's too much to ask, either due to implementation or policy concerns, I'd take an explicit allowlist letting specific cases through, such as the systemd one, instead of having to settle for either suboptimal or wrong solutions for the problem at hand, due to the currently required workflow being too cumbersome. Thanks, Guillem
Re: Need help with Multi-Arch in systemd
Am 09.07.21 um 14:24 schrieb Helmut Grohne:
Now let's do something stupid. Rename systemd to systemd-core (taking
all files with it, please refrain from discussing the name unless you
seriously consider doing this). Mark it Multi-Arch: allowed. Add a new,
empty binary package systemd. It is Multi-Arch: foreign and depends on
systemd-core:any. This approach would technically satisfy all three
requirements, but it feels a little crazy to me.
[..]
And I fear we have another related issue that we swept under the carpet
thus far. man systemd.exec explains SystemCallArchitectures=native.
You are right. Thinking more about this, splitting out libsystemd-shared
as a Multi-Arch: same library will not help with
SystemCallArchitectures=native, which is used by the services in
systemd-{container,journal-remote,...}.
So splitting out libsystemd-shared, while in theory a nice solution, is
not the right one in this case. We really need to ensure that systemd
and systemd-* are of of the same architecture.
I still think that my idea of having a ":arch" annotation as counterpart
to ":any" would be the most elegant way to achieve this. But since this
is only an idea and not implemented, it's not something I can make use
of. Do you think there is a chance we could convince dpkg and apt
maintainers to add support for that?
Alternatively, your idea of systemd-core/systemd got me thinking.
While I don't like the prospect of moving all (conf)files and state from
one package to another, maybe we can turn this idea around.
We introduce an empty systemd-native package, which is
Architecture:linux-any but *not* Multi-Arch: foreign/same/allowed.
systemd and all systemd-* packages would depend on systemd-native.
This would link the architecture of systemd and systemd-* together and
ensure they are the same.
Would this work for your cross-compile use-case?
Michael
OpenPGP_signature
Description: OpenPGP digital signature
Re: Need help with Multi-Arch in systemd
Hi Michael,
On Thu, Jul 15, 2021 at 12:22:59AM +0200, Michael Biebl wrote:
> You are right. Thinking more about this, splitting out libsystemd-shared as
> a Multi-Arch: same library will not help with
> SystemCallArchitectures=native, which is used by the services in
> systemd-{container,journal-remote,...}.
That is correct, but it actually goes beyond systemd-* using systemd.
Any other service can face the very same problem.
> So splitting out libsystemd-shared, while in theory a nice solution, is not
> the right one in this case. We really need to ensure that systemd and
> systemd-* are of of the same architecture.
We have two related issues at hand. One is the libsystemd-shared
architecture matching and the other is the SystemCallArchitectures
matching. You appear to imply that both issues need to be addressed by
one common solution. Sure that would be nice, but is it required? Maybe
these issues are not that related after all.
Let us for a moment assume that we could magically make
SystemCallArchitectures work by locking users to the same architecture
as systemd. That would be bad in terms for mixed multiarch systems and
cross grading, because you essentially lock all daemons to the same
architecture as systemd. You fix the problem, by removing flexibility.
I also proposed a solution here, which is avoiding
SystemCallArchitectures=native. Initially, that sounds like a
maintenance nightmare until you notice that it can be solved on a
technical level. We already have dh_installsystemd. How bad would it be
to have dh_installsystemd change .service files and automatically
replace =native with a concrete architecture (in arch:any packages
only)? systemd would no longer detect the architecture of services from
its own one but rather use the one documented by the package. The mixing
of architectures that our earlier solution broke would now partially
work. We'd fix one package and binNMU the pile.
So while these problems are related, I think that forcing the
architectures to equal is a suboptimal solution for
SystemCallArchitectures.
> I still think that my idea of having a ":arch" annotation as counterpart to
> ":any" would be the most elegant way to achieve this. But since this is only
> an idea and not implemented, it's not something I can make use of. Do you
> think there is a chance we could convince dpkg and apt maintainers to add
> support for that?
If you replace :arch with :$DEB_HOST_ARCH, it already works today with
apt and dpkg, but not dose. The question is not whether we can implement
that (it already is), but whether we want to endorse these semantics or
not.
> Alternatively, your idea of systemd-core/systemd got me thinking.
> While I don't like the prospect of moving all (conf)files and state from one
> package to another, maybe we can turn this idea around.
>
> We introduce an empty systemd-native package, which is
> Architecture:linux-any but *not* Multi-Arch: foreign/same/allowed.
>
> systemd and all systemd-* packages would depend on systemd-native.
> This would link the architecture of systemd and systemd-* together and
> ensure they are the same.
I think this technically works. We also have prior art for this. blas
temporarily added such a package as a measure to fix #760936.
> Would this work for your cross-compile use-case?
I don't think this would negatively affect cross compiling. It could
affect people who try to run mixed-architecture systems though.
Given Simon's and Guillem's replies, I presently favour fixing the NEW
processing to package the library separately and fix
SystemCallArchitectures using dh_installsystemd, because it is
technically sound and does not negatively impact multiarch use cases.
Should I file a bug about SystemCallArchitectures such that we can track
it somewhere?
Helmut
Re: Need help with Multi-Arch in systemd
On 16194 March 1977, Simon McVittie wrote: Would it be feasible for dak to have a list of binary package name regexes mapped to a source package and a section/priority, and auto-accept packages from the given source package that match the regex, assigning the given section/priority, without manual action? That would let the ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/ in libs/optional, for example. At some point somewhere I think it was already said, but in general: We love MRs, we are at https://salsa.debian.org/ftp-team/dak/ and something doing kind of auto-NEW processing is NOT out of the question. Exact details have to be hashed out, but this is the wrong thread for that. While sometimes NEW can be annoying, it regularly catches bugs even if "only a small change in packaging" happened, so it does have some reason why its there. But yes, there are some candidates (hello kernel) that can make use of something with less humans involved. Your example above doesn't sound too bad as a starter, though haven't put any further thought into it yet. -- bye, Joerg
Re: Need help with Multi-Arch in systemd
On Thu, Jul 15, 2021 at 08:08:13AM +0200, Helmut Grohne wrote: > I also proposed a solution here, which is avoiding > SystemCallArchitectures=native. Initially, that sounds like a > maintenance nightmare until you notice that it can be solved on a > technical level. We already have dh_installsystemd. How bad would it be > to have dh_installsystemd change .service files and automatically > replace =native with a concrete architecture (in arch:any packages > only)? systemd would no longer detect the architecture of services from > its own one but rather use the one documented by the package. The mixing > of architectures that our earlier solution broke would now partially > work. We'd fix one package and binNMU the pile. I've discussed this with #systemd on libera.chat. That resulted in a new approach: Letting systemd detect the architecture of the binary being executed. We've reported that as a feature request at https://github.com/systemd/systemd/issues/20277. I therefore propose to temporarily ignore the SystemCallArchitectures issue on the Debian side and see what upstream thinks about it. The libsystemd-shared issue can be tackled independently. Helmut
automatic NEW processing [was Re: Need help with Multi-Arch in systemd]
Am 14.07.21 um 12:59 schrieb Simon McVittie: Would it be feasible for dak to have a list of binary package name regexes mapped to a source package and a section/priority, and auto-accept packages from the given source package that match the regex, assigning the given section/priority, without manual action? That would let the ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/ in libs/optional, for example. It seems like this would also be good for src:linux, where ABI breaks are often tied to security fixes that should enter the archive ASAP. If something fully automated like this would be implemented, I would have much less concerns with this option. As it stands today, NEW processing is simply to unpredictable. It can range from taking a a few hours/days to several months. Regards, Michael OpenPGP_signature Description: OpenPGP digital signature
Re: automatic NEW processing [was Re: Need help with Multi-Arch in systemd]
Am 14.07.21 um 13:47 schrieb Michael Biebl: Am 14.07.21 um 12:59 schrieb Simon McVittie: Would it be feasible for dak to have a list of binary package name regexes mapped to a source package and a section/priority, and auto-accept packages from the given source package that match the regex, assigning the given section/priority, without manual action? That would let the ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/ in libs/optional, for example. It seems like this would also be good for src:linux, where ABI breaks are often tied to security fixes that should enter the archive ASAP. If something fully automated like this would be implemented, I would have much less concerns with this option. Fwiw, I like your proposal and would like to see it implemented regardless of the issue at hand as I think it could be generally useful. Thanks, Simon! OpenPGP_signature Description: OpenPGP digital signature
Re: automatic NEW processing [was Re: Need help with Multi-Arch in systemd]
On 14.07.21 13:47, Michael Biebl wrote: Am 14.07.21 um 12:59 schrieb Simon McVittie: Would it be feasible for dak to have a list of binary package name regexes mapped to a source package and a section/priority, and auto-accept packages from the given source package that match the regex, assigning the given section/priority, without manual action? That would let the ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/ in libs/optional, for example. It seems like this would also be good for src:linux, where ABI breaks are often tied to security fixes that should enter the archive ASAP. If something fully automated like this would be implemented, I would have much less concerns with this option. As it stands today, NEW processing is simply to unpredictable. It can range from taking a a few hours/days to several months. And yet it should not dictate technical solutions. We basically see the same thing with nvidia-graphics-drivers that break your running applications when the libraries are upgraded and you don't reboot. Arguably the proper solution is to version them with the full major/minor version. But I can see how that's a total hassle with NEW processing for both for the maintainer and to the FTP team. I do recall that the FTP masters would've been generally open to have such an auto-approver (but maybe I'm wrong), but that no-one stepped up yet to code it up? Kind regards Philipp Kern
Re: automatic NEW processing [was Re: Need help with Multi-Arch in systemd]
On Wed, 14 Jul 2021 18:48:24 +0200 Philipp Kern wrote: > On 14.07.21 13:47, Michael Biebl wrote: > > Am 14.07.21 um 12:59 schrieb Simon McVittie: > I do recall that the FTP masters would've been generally open to have > such an auto-approver (but maybe I'm wrong), but that no-one stepped up > yet to code it up? A few of us came up with some proof of concept designs/models, but we ultimately dropped the effort when it became clear the team wasn't interested in such tools. We considered continuing with a tool that would work for individual users reviewing their own work, but there just wasn't enough interest for that either. I'd be happy to help resurrect (the personal-use version of) the project/effort if there's any chance I won't be working on it by myself...
Re: automatic NEW processing [was Re: Need help with Multi-Arch in systemd]
On 16200 March 1977, Michael Lustfield wrote: I do recall that the FTP masters would've been generally open to have such an auto-approver (but maybe I'm wrong), but that no-one stepped up yet to code it up? A few of us came up with some proof of concept designs/models, but we ultimately dropped the effort when it became clear the team wasn't interested in such tools. We considered continuing with a tool that would work for individual users reviewing their own work, but there just wasn't enough interest for that either. I'd be happy to help resurrect (the personal-use version of) the project/effort if there's any chance I won't be working on it by myself... The place for that feature to end up in is inside dak, obviously. That feature should read its config (is it enabled? for the current package? with which config for it?) from the database (projectb). And then just hook itself into the part of the uploading that redirects packages to NEW. And then one could look if it gets implemented a bit more generic and not NEW specific, but able to do something like this for any policy queue. (As in, backports NEW, p-u, whatever). So, configurable per queue. For the actual feature and the affected packages. Any MR going in that direction will be great. -- bye, Joerg
