Hi Simon,
thanks for following up so quickly.
On Thu, Aug 01, 2024 at 10:22:39AM +0100, Simon McVittie wrote:
> Is the multiarch hinter a cross team thing, or someone else's? I find
> it frustrating that it doesn't seem to have an overrides mechanism,
> so when it has a recommendation that is well-intentioned but wrong,
> the affected package can't silence it (like we can for e.g. Lintian or
> blhc false positives).
The hinter is operated by me on delfin.d.o. Thus far the amount of false
recommendations wasn't that much and whenever a false recommendation
surfaced I changed the algorithm to rule out the entire category instead
of adding an individual override mechanism.
I am happy to implement an override mechanism if that's what is needed
and I am happy to share the maintenance with interested parties. When I
wrote that item it was more of a thing to remember to carry out rather
than a blocker.
> > Package: cross-exe-wrapper
> > Architecture: any
> > Multi-Arch: same
> > Depends: native-architecture | qemu-user, native-architecture | arch-test
> > # wrapper below
>
> I'd recommend making this include its own /bin/true implementation
> (/usr/lib/*/gobject-introspection/deb-can-run-* in gobject-introspection
> are an implementation of this), instead of making arch-test
> load-bearing.
That would work as well. I note that arch-test sets QEMU_VERSION=meow
and WINEPREFIX=/dev/null to avoid detecting emulation. The benefit would
be that the use of emulation becomes explicit in the build log rather
than implicit. Other than that aspect, I agree.
In using arch-test, I attempted describing an MVP that has sufficient
detail to be easy to understand without overloading it and arch-test
felt like a reasonable compromise to express the intention. I fully
expect the final solution to have subtly different shape.
> Ideally this would have a special case for the common i386-on-amd64 to
> make it not require qemu-user, if that's possible (#1070773). Perhaps
> something like this
>
> Depends: native-architecture | dpkg-architecture-is-amd64 [i386] | qemu-user
I think native-architecture could reasonably be part of
src:architecture-properties if we want to pursue that approach. If we
opt for that, architecture-properties can also have an arch-any package
called native-properties depending on native-architecture and providing
dpkg-architecture-is-*, so this is technically implementable without
dpkg support.
> where dpkg Provides: dpkg-architecture-${DEB_HOST_ARCH}? But that doesn't
> have to be part of a minimum-viable-product implementation.
Looping dpkg in seems very sensible to me, because the dpkg package is
what defines the native architecture. In lifting it to a Provides there,
we make this aspect dependable without depending on the subtle semantics
of Arch:all M-A:no packages.
Arguably, this makes using the native-architecture approach (with its
subtle semantics) feasible. If we were to ever change the Arch:all
semantics in dpkg, we'd also have to add these Provides. Then we could
turn the native-architecture package from Arch:all to Arch:any and have
it depend on dpkg-architecture-is-${DEB_HOST_ARCH}.
> Concretely, let's say we're building on an amd64 system and we want to
> compile a GLib-dependent package for amd64 and riscv64 hosts. We have:
>
> libglib2.0-dev:amd64
> -> cross-exe-wrapper:amd64
> -> native-architecture:all (success)
>
> libglib2.0-dev:riscv64
> -> cross-exe-wrapper:riscv64
> -> ~~native-architecture:all~~ (no, uninstallable)
> -> qemu-user (success)
I confirm.
> It seems pretty subtle that the only thing making this work as intended
> is that dpkg:amd64 will refuse to install native-architecture:all to
> satisfy a dependency of a riscv64 package. That's a relatively obscure
> corner of multiarch!
I fully concur and that is part of the reason why I have been arguing in
favour of changing semantics in a way that removes this subtlety.
> I do wonder whether it would be cleaner to have dpkg add a
>
> Provides: dpkg-architecture-is-${local:DEB-HOST-ARCH}
>
> and then have cross-exe-wrapper depend on
>
> Depends: dpkg-architecture-is-${local:DEB-HOST-ARCH} |
> dpkg-architecture-is-amd64 [i386] | qemu-user
>
> (where ${local:DEB-HOST-ARCH} is a substvar, in both cases set up in
> the same way that ${local:DEB-HOST-ARCH-OS} works in src:glib2.0)
Given the above, I concur that this is feasible. It is feasible both
when leaving dpkg unchanged (by exploiting subtle semantics) and when
changing subtle semantics in dpkg (by adding concurrently adding the
Provides).
> > The actual wrapper script's name should be architecture-dependent such
> > as $host_triplet-cross-exe-wrapper.
>
> That's ${DEB_HOST_GNU_TYPE}-cross-exe-wrapper (not the multiarch tuple),
> in the same pattern we use for cross gcc and cross pkgconf and so on, right?
Yes, I should have been more explicit about this.
> > It can roughly look like:
> >
> > #!/bin/sh
> >
> > if test "$(dpkg --print-architecture)" = #DEB_HOST_ARCH# || arch-test -n
> > #DEB_HOST_ARCH# >/dev/null; then
> > exec "$@"
> > else
> > exec qemu-#DEB_HOST_ARCH# -- "$@"
> > fi
>
> I'm not sure that qemu-user actually accepts "--" (which is why in the
> straw-man specification I wrote, if I remember correctly I said something
> like "the wrapped executable should be an absolute path and must not
> start with -" to bypass that concern), but otherwise yes, that's more
> or less what I had in mind.
>
> Maybe it would be good to put a linux32 prefix on the command if targeting
> a known 32-bit counterpart architecture (and not using qemu), like i386
> on amd64? (exec linux32 "$@")
Do you mind if I call this details? Technically speaking this is
spot-on, but I think these aspects are generally solvable with low risk
of introducing show stoppers.
> > Let me note that I have been arguing in favour of changing the semantics
> > or Arch:all in ways that would break the aforementioned setup.
>
> That's already the part of this that seems most fragile to me. If the dpkg
> maintainer was willing to add the dpkg-architecture-is-* setup that I
> suggested, I think that would give us a way to avoid needing to rely on
> this behaviour?
I concur.
> > I recently encountered the
> > following packages that would fail to install their Build-Depends due to
> > choosing a host architecture Python interpreter and failing its
> > postinst:
> > [29 packages]
>
> That's not actually all that many. I wonder whether it would be pragmatic
> to add python3:native to their B-D and move on?
It's actually way more. I am regularly looking at cross build failures
and these were the ones I recently encountered. The message I tried to
get across was that my semi-random sample lacks a package that would not
benefit from using the barbarian switch.
I could relatively easily switch crossqa workers to configure barbarian
architectures if we want to pursue this.
> Or teach crossqa or sbuild to always --add-depends=python3:native when
> cross-building? In the worst case scenario that's adding python3 when
> it isn't actually needed, but increasingly many packages use Meson which
> pulls in python3 anyway, so maybe that's not so bad?
I quite strongly prefer to avoid non-minimal build environments.
> Certainly if any of those are critical-path for unblocking whole subtrees
> of currently-non-cross-compilable packages, or if they're important targets
> for cross-compiling for any other reason, adding python3:native to their
> B-D should be an easy workaround.
I don't consider such an approach reasonable. Due to the number of
affected packages, it would add a constant maintenance effort to an
understaffed (single-person?) team (cross porters). With the current
staffing, the only viable way forward is building lasting solutions even
if it takes us longer to get there.
> > There are two main downsides of going forward with barbarian as far as I
> > can see. One is that we restrict the solution space and outright reject
> > ever installing a host's instance of a M-A:foreign package in a cross
> > build environment. I don't know why that would ever be necessary, but
> > then we can no longer do it. The other is that we need to fix all the
> > builders to make cross building unstable work. In particular, bookworm's
> > sbuild will no longer be able to cross build unstable.
>
> My instinct is to say that neither of these is particularly bad.
Thank you. To me this sounds like we should actually pursue both
approaches in parallel. The barbarian change has a patch in sbuild.git
and is fairly easily deployable to crossqa and I now expect it to
provide significant improvements. If it turns out to have bad
side-effects we can still consider reverting it. It does not solve the
amd64/i386 use case #1070773 that is solved by the cross-exe-wrapper.
I anticipate that more than gobject-introspection will need the
cross-exe-wrapper. Therefore, I'd prefer to decouple it. Any ideas for a
good place for it? I am wondering whether architecture-properties would
be willing to carry this.
Helmut
