Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, 26 Feb 2024 at 17:58, Simon McVittie wrote: > I did consider that, but if I'm understanding what you propose > correctly, that would involve going through these steps for every > possible cross-tool: > > * inventing a new environment variable similar to CC and PKG_CONFIG, for > example perhaps G_IR_SCANNER in this case > * making debhelper (or dpkg) add it to the environment > * making `meson env2mfile` (or the older debcrossgen) read it from the > environment and add it to the generated cross-file That is not how it would work at all. Meson's env2mfile has a command line option that tells it to generate things for Debian package building. The idea is that then you can add any custom detection logic you could possibly want. In this way custom logic can be added in the main project in an actual programming language rather than trying to replicate the same via envvars and magic hacks.
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Sun, 25 Feb 2024 at 03:21:23 +0200, Jussi Pakkanen wrote:
> Meson comes with a tool that converts native and cross environments
> into cross files. It is run with `meson env2mfile`. It even has a mode
> to generate Debian package building scripts. Updating that to handle
> this case would probably be the smartest thing to do.
I did consider that, but if I'm understanding what you propose
correctly, that would involve going through these steps for every
possible cross-tool:
* inventing a new environment variable similar to CC and PKG_CONFIG, for
example perhaps G_IR_SCANNER in this case
* making debhelper (or dpkg) add it to the environment
* making `meson env2mfile` (or the older debcrossgen) read it from the
environment and add it to the generated cross-file
and that seems like something that would scale poorly with the number
of cross-tools that exist? We would have to coordinate changes between
debhelper and meson every time a new cross-tool was identified.
Or do you mean that `meson env2mfile` would have a long list of pkg-config
queries that it would do using ${PKG_CONFIG} in order to populate the
cross-file, like for example in this case, if gobject-introspection-1.0.pc
was found, it would output the equivalent of the shell-like pseudocode
"g-ir-scanner = '$(${PKG_CONFIG} --variable=g_ir_scanner
gobject-introspection-1.0)'"
and so on? That would avoid needing to invent new environment variables
and modify debhelper, but it would still make updating `meson env2mfile`
into a single point of failure for the ability to use any cross-tool.
I see two possible routes that would scale better.
One is what Helmut suggested: if running a cross-tool might make sense,
ask the host pkg-config rather than the build pkg-config for the path to
that tool, and rely on the packaging having been set up to make that work
(as I already did for gobject-introspection). For the finite number of
tools discovered internally by meson (like g-ir-scanner in the gnome
module) this would still need to be a meson change, but for tools run
by third-party packages as a custom_target or similar, it would be up
to the third-party package to do this (although Meson could perhaps
encourage it in documentation).
The other is to do something with the common convention of an
architecture prefix. In Autotools-world, there's a convention that
if running a cross-tool might make sense, then the configure script
looks for ${host_tuple}-${tool} before ${tool}, where ${tool} is
something like pkg-config or g-ir-scanner, and ${host_tuple} is the
host architecture given to the configure script (in Debian this
is ${DEB_HOST_GNU_TYPE}). Similarly, when using plain Makefiles,
it's relatively common to ask for ${CROSS_COMPILE}${tool}, where
${CROSS_COMPILE} is normally empty, but can be set to the host tuple
followed by "-" when cross-compiling. Either of these will successfully
find our /usr/bin/x86_64-linux-gnu-g-ir-scanner and so on.
smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Fri, 23 Feb 2024 at 14:27, Simon McVittie wrote: > On Wed, 21 Feb 2024 at 10:47:33 +0100, Johannes Schauer Marin Rodrigues wrote: > > I've just learned from Simon McVittie in #debian-devel that there is a way > > to > > cross-build packages using meson and gobject-introspection without this > > patch > > by configuring them with --cross-file. Here is an example from libportal: > > > > https://salsa.debian.org/debian/libportal/-/commit/4efec56ef735eac25836acca867fe59812c3dc6d Meson comes with a tool that converts native and cross environments into cross files. It is run with `meson env2mfile`. It even has a mode to generate Debian package building scripts. Updating that to handle this case would probably be the smartest thing to do. Currently Debian package building does not use that by default. We planned to make it use that, but it had some bugs and it had to be taken out for Debian release freeze.
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Wed, 21 Feb 2024 at 10:47:33 +0100, Johannes Schauer Marin Rodrigues wrote: > I've just learned from Simon McVittie in #debian-devel that there is a way to > cross-build packages using meson and gobject-introspection without this patch > by configuring them with --cross-file. Here is an example from libportal: > > https://salsa.debian.org/debian/libportal/-/commit/4efec56ef735eac25836acca867fe59812c3dc6d I did mention this cross-file in an earlier message to this same bug. This and other non-trivial gobject-introspection packaging topics are documented in: file:///usr/share/doc/gobject-introspection/README.Debian.gz (which I hope to keep up to date with best-practice as it changes). smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
Hi, Quoting Helmut Grohne (2024-01-15 13:04:27) > Simon worked a lot on making gobject-introspection workable with cross > compilation and this work has hit unstable now. While this seems to work > fine with autotools, it tends to fail with meson. I've just learned from Simon McVittie in #debian-devel that there is a way to cross-build packages using meson and gobject-introspection without this patch by configuring them with --cross-file. Here is an example from libportal: https://salsa.debian.org/debian/libportal/-/commit/4efec56ef735eac25836acca867fe59812c3dc6d I successfully used the same technique for gst-plugins-bad1.0 and gst-plugins-good1.0. I'm recording this here so this technique doesn't get lost and in case others want to have their packages cross-built before this or a similar patch can get accepted. Thanks! cheers, josch signature.asc Description: signature
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, 15 Jan 2024 at 14:32:44 +0100, Helmut Grohne wrote:
> On Mon, Jan 15, 2024 at 12:38:54PM +, Simon McVittie wrote:
> > in general the
> > host g-ir-scanner will not even be executable during a cross-build
>
> I question this. g-ir-scanner is written in Python and it is fairly
> unlikely that /usr/bin/python3 will end up being the host Python that
> cannot be run. Hence using the g-ir-scanner from the host typically
> should work (or not) the same way as the one from the build
If it was pure Python, then this would be
true, but it loads a private module written in C
(/usr/lib/*/gobject-introspection/giscanner/_giscanner.cpython-311-*.so)
and that module needs to match the architecture of the python3
interpreter.
The result is that g-ir-scanner has two orthogonal architecture
dependencies:
1. It needs to be loading a _giscanner module whose
architecture matches python3: in practice this means adding
/usr/lib/${DEB_BUILD_MULTIARCH}/gobject-introspection to sys.path
2. Meanwhile it also needs to be using a $CC, $PKG_CONFIG, and search
paths that are appropriate for ${DEB_HOST_MULTIARCH}, and eventually
run a temporary host-architecture dumper binary, possibly via qemu
This makes it "the same shape" as gcc or ld, which are build-architecture
executables that have host-architecture object code as their
input/output. Unlike gcc, there is no built-in upstream way to build
a cross-g-ir-scanner analogous to a cross-compiler, and there does not
seem to be any interest in adding that upstream, which is why I had to
do this as a Debian-specific change.
However, reasoning similar to what's quoted above *does* work for tools
like gdbus-codegen, which *is* pure Python, avoiding the first of those
architecture dependencies. (As it happens, gdbus-codegen also operates
at the level of source code rather than binaries, so it avoids the second
architecture dependency too.)
smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, Jan 15, 2024 at 12:38:54PM +, Simon McVittie wrote:
> On Mon, 15 Jan 2024 at 13:04:27 +0100, Helmut Grohne wrote:
> > Among other things, Simon introduced new programs
> > /usr/bin/-g-ir-scanner. When cross building, these should be
> > used and they'll automatically employ qemu-user when needed. He also
> > updated gobject-introspection-1.0.pc to have g_ir_scanner point at this
> > rather than plain /usr/bin/g-ir-scanner. Unfortunately, meson looks up
> > the gobject-introspection-1.0 dependency with native set to true. Hence,
> > it does not pick up the host triplet but the build triplet here and that
> > doesn't go well.
>
> I believe this was done intentionally in Meson, because in general the
> host g-ir-scanner will not even be executable during a cross-build. The
> fact that we have made it work is Debian-specific. If I understand
> upstream's position correctly, the way to force this sort of thing is
> via cross-files.
I question this. g-ir-scanner is written in Python and it is fairly
unlikely that /usr/bin/python3 will end up being the host Python that
cannot be run. Hence using the g-ir-scanner from the host typically
should work (or not) the same way as the one from the build - unless one
has specifically changed it (as you've done for the benefit of Debian).
Possibly, some builds are performed with no gobject-introspection-1.0.pc
being installed for the host architecture. In such cases, we could fall
back from host architecture to build architecture. I don't see a
plausible failure scenario arising from using the g_ir_scanner variable
from the host file when available though.
Independently of whether it breaks stuff or not, the proposed patch may
still go against upstream's position, yes.
> Debian's gobject-introspection does provide a cross-file that does what we
> want (--cross-file=${DEB_HOST_GNU_TYPE}-gobject-introspection.ini). See
> src:graphene for an example of this being used. Ideally, this would
> be used automatically for all cross-builds that involve g-i, to avoid
> needing to make sourceful changes in all packages that run g-i, but
> unfortunately I am not aware of any currently available way to ask an
> appropriate layer (debhelper?) to add a category of non-core cross files
> to all cross-builds.
>
> Perhaps a possible route for the future would be to teach debhelper's
> meson build system to add a --cross-file to all cross-builds, automatically,
> for each file matching /usr/share/meson/cross/${DEB_HOST_GNU_TYPE}-*.ini?
> But I have not yet discussed that idea with the debhelper maintainers.
Let me express an argument against this approach. Being hooked into
debhelper means that things will just work in Debian package builds.
However, Debian is also used as a development platform for building
other things than packages. When you perform your build by running meson
and ninja directly, you'll have to remember to also pass these
crossfiles for every cross build performed on Debian. In order for this
to benefit non-package builds, meson itself would have to automatically
consume those crossfiles.
I actually see this as a problem in the rust ecosystem currently. Debian
package builds are somehow enhanced to dependencies from Debian packages
rather than downloading stuff from cargo. Replicating this offline
experience without building Debian packages is a non-trivial affair.
For this reasons, I still prefer a solution at the meson-level (even if
it is not the solution I am proposing here).
Still, if everyone else thinks that debhelper is the better place, I'm
going out of the way and can supply the relevant debhelper patch.
> I think we are going to need a similar mechanism in future for at
> least Vala's vapigen (which is the reason why e.g. libportal can't be
> cross-compiled by making changes similar to the ones in graphene).
>
> > def find_tool(self, name: str, depname: str, varname: str, required:
> > bool = True,
> > - wanted: T.Optional[str] = None) ->
> > T.Union['build.Executable', ExternalProgram, 'OverrideProgram']:
> > + wanted: T.Optional[str] = None, native: bool = True) ->
> > T.Union['build.Executable', ExternalProgram, 'OverrideProgram']:
>
> Is find_tool intended to be analogous to Autoconf AC_CHECK_TOOL,
> which automatically checks for a cross-tool using the convention that a
> cross-tool that works with binaries for the host architecture ${HOST} (but
> runnable on the build architecture) is usually named ${HOST}-${tool_name},
> falling back to ${tool_name} only if ${HOST}-${tool_name} is not found?
>
> (In Debian terms, ${HOST} is ${DEB_HOST_GNU_TYPE})
>
> Or is find_tool intended to be more like Autoconf AC_CHECK_PROG, which is
> appropriate for tools that have no particular architecture dependency, like
> documentation generators, but not appropriate for compilers, linkers and
> other tools that might have different behaviour and outputs per-architecture?
Thanks for asking the question I
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, 15 Jan 2024 at 13:04:27 +0100, Helmut Grohne wrote:
> Among other things, Simon introduced new programs
> /usr/bin/-g-ir-scanner. When cross building, these should be
> used and they'll automatically employ qemu-user when needed. He also
> updated gobject-introspection-1.0.pc to have g_ir_scanner point at this
> rather than plain /usr/bin/g-ir-scanner. Unfortunately, meson looks up
> the gobject-introspection-1.0 dependency with native set to true. Hence,
> it does not pick up the host triplet but the build triplet here and that
> doesn't go well.
I believe this was done intentionally in Meson, because in general the
host g-ir-scanner will not even be executable during a cross-build. The
fact that we have made it work is Debian-specific. If I understand
upstream's position correctly, the way to force this sort of thing is
via cross-files.
Debian's gobject-introspection does provide a cross-file that does what we
want (--cross-file=${DEB_HOST_GNU_TYPE}-gobject-introspection.ini). See
src:graphene for an example of this being used. Ideally, this would
be used automatically for all cross-builds that involve g-i, to avoid
needing to make sourceful changes in all packages that run g-i, but
unfortunately I am not aware of any currently available way to ask an
appropriate layer (debhelper?) to add a category of non-core cross files
to all cross-builds.
Perhaps a possible route for the future would be to teach debhelper's
meson build system to add a --cross-file to all cross-builds, automatically,
for each file matching /usr/share/meson/cross/${DEB_HOST_GNU_TYPE}-*.ini?
But I have not yet discussed that idea with the debhelper maintainers.
I think we are going to need a similar mechanism in future for at
least Vala's vapigen (which is the reason why e.g. libportal can't be
cross-compiled by making changes similar to the ones in graphene).
> def find_tool(self, name: str, depname: str, varname: str, required:
> bool = True,
> - wanted: T.Optional[str] = None) ->
> T.Union['build.Executable', ExternalProgram, 'OverrideProgram']:
> + wanted: T.Optional[str] = None, native: bool = True) ->
> T.Union['build.Executable', ExternalProgram, 'OverrideProgram']:
Is find_tool intended to be analogous to Autoconf AC_CHECK_TOOL,
which automatically checks for a cross-tool using the convention that a
cross-tool that works with binaries for the host architecture ${HOST} (but
runnable on the build architecture) is usually named ${HOST}-${tool_name},
falling back to ${tool_name} only if ${HOST}-${tool_name} is not found?
(In Debian terms, ${HOST} is ${DEB_HOST_GNU_TYPE})
Or is find_tool intended to be more like Autoconf AC_CHECK_PROG, which is
appropriate for tools that have no particular architecture dependency, like
documentation generators, but not appropriate for compilers, linkers and
other tools that might have different behaviour and outputs per-architecture?
smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
Package: meson
Version: 1.3.1-1
Severity: important
Tags: patch upstream
X-Debbugs-Cc: debian-cr...@lists.debian.org, s...@debian.org
User: debian-cr...@lists.debian.org
Usertags: ftcbfs
Control: affects -1 + src:harfbuzz
Hi,
Simon worked a lot on making gobject-introspection workable with cross
compilation and this work has hit unstable now. While this seems to work
fine with autotools, it tends to fail with meson.
Among other things, Simon introduced new programs
/usr/bin/-g-ir-scanner. When cross building, these should be
used and they'll automatically employ qemu-user when needed. He also
updated gobject-introspection-1.0.pc to have g_ir_scanner point at this
rather than plain /usr/bin/g-ir-scanner. Unfortunately, meson looks up
the gobject-introspection-1.0 dependency with native set to true. Hence,
it does not pick up the host triplet but the build triplet here and that
doesn't go well. We need to change the native argument to false here.
I'm attaching a patch demonstrating the change and it makes building
e.g. harfbuzz work. I expect this change to fix many packages hence
raising the severity to important. I'm not sure whether this is the
desired way to extend meson nor whether this has consequences for
downstreams other than Debian. How can we take care of these matters?
Helmut
--- meson-1.3.1.orig/mesonbuild/modules/__init__.py
+++ meson-1.3.1/mesonbuild/modules/__init__.py
@@ -96,7 +96,7 @@
wanted=wanted, silent=silent, for_machine=for_machine)
def find_tool(self, name: str, depname: str, varname: str, required: bool = True,
- wanted: T.Optional[str] = None) -> T.Union['build.Executable', ExternalProgram, 'OverrideProgram']:
+ wanted: T.Optional[str] = None, native: bool = True) -> T.Union['build.Executable', ExternalProgram, 'OverrideProgram']:
# Look in overrides in case it's built as subproject
progobj = self._interpreter.program_from_overrides([name], [])
if progobj is not None:
@@ -108,7 +108,7 @@
return ExternalProgram.from_entry(name, prog_list)
# Check if pkgconfig has a variable
-dep = self.dependency(depname, native=True, required=False, wanted=wanted)
+dep = self.dependency(depname, native=native, required=False, wanted=wanted)
if dep.found() and dep.type_name == 'pkgconfig':
value = dep.get_variable(pkgconfig=varname)
if value:
--- meson-1.3.1.orig/mesonbuild/modules/gnome.py
+++ meson-1.3.1/mesonbuild/modules/gnome.py
@@ -328,7 +328,7 @@
}
depname = tool_map[tool]
varname = tool.replace('-', '_')
-return state.find_tool(tool, depname, varname)
+return state.find_tool(tool, depname, varname, native=depname != "gobject-introspection-1.0")
@typed_kwargs(
'gnome.post_install',
