Re: autoreconf --force seemingly does not forcibly update everything

[Simon Josefsson via Gnulib discussion list]

Paul Eggert  writes:

> On 4/10/24 13:36, Simon Josefsson via Gnulib discussion list wrote:
>> Is bootstrap intended to be reliable from within a tarball?  I thought
>> the bootstrap script was not included in tarballs because it wasn't
>> designed to be ran that way, and the way it is designed may not give
>> expected results.
>
> It's pretty routinely distributed, I expect under the theory that
> we're being transparent about what sources we use to generate the
> tarball.
>
> Whether it works from a tarball depends on one's definition of
> "works". Certainly more expertise and tools are needed to bootstrap
> than merely to configure + make.

The definition for "works" seems fairly permissive: running ./bootstrap
from, e.g., the coreutils 9.5 tarball dies instantly due to this:

  if test -n "$checkout_only_file" && test ! -r "$checkout_only_file"; then
die "Running this script from a non-checked-out distribution is risky."
  fi

I see that some projects (including coreutils) add bootstrap to
EXTRA_DIST, but I can't find any recommendation in the gnulib manual to
do that so I had assumed it is not something we recommend generally.  I
haven't added it to inetutils, libidn2, gsasl, etc.

/Simon


signature.asc
Description: PGP signature

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Nick Bowler wrote:
> Not including the scripts used to build configure in a source tarball
> is a mistake, particularly for a GPL-licensed package.  The configure
> script itself is clearly object code, and the GPL defines corresponding
> source to include any "scripts to control [its generation]".

But the GPL does not require that the corresponding sources be included
in the *same tarball*.

When a distro distributes binary packages, they don't include the source
code in the same binary .rpm or binary .tar.xz. All they do is to provide
the source code in a location that is straightforward to find.

The same rights hold for upstream package maintainers: When they produce
tarballs, that include a "binary" configure and generated .c files, it is
sufficient that the source code (*.m4 files and .y and .l files) are in
the git repository or submodules, and that the git repository is straight-
forward to find.

Bruno

Re: autoreconf --force seemingly does not forcibly update everything

[Nick Bowler]

On 2024-04-10 16:36, Simon Josefsson wrote:
> Is bootstrap intended to be reliable from within a tarball?  I thought
> the bootstrap script was not included in tarballs because it wasn't
> designed to be ran that way, and the way it is designed may not give
> expected results.  Has this changed, so we should recommend maintainers
> to 'EXTRA_DIST = bootstrap bootstrap-funclib.sh bootstrap.conf' so this
> is even possible?

Not including the scripts used to build configure in a source tarball
is a mistake, particularly for a GPL-licensed package.  The configure
script itself is clearly object code, and the GPL defines corresponding
source to include any "scripts to control [its generation]".

If you cannot successfully regenerate configure from a source tarball,
it is probably missing some of the source code for the configure script,
which is a problem if it was supposed to be a free software package.

Cheers,
  Nick

Re: autoreconf --force seemingly does not forcibly update everything

[Jeffrey Walton]

On Wed, Apr 10, 2024 at 5:44 PM Paul Eggert  wrote:
>
> On 4/10/24 13:36, Simon Josefsson via Gnulib discussion list wrote:
> > Is bootstrap intended to be reliable from within a tarball?  I thought
> > the bootstrap script was not included in tarballs because it wasn't
> > designed to be ran that way, and the way it is designed may not give
> > expected results.
>
> It's pretty routinely distributed, I expect under the theory that we're
> being transparent about what sources we use to generate the tarball.
>
> Whether it works from a tarball depends on one's definition of "works".
> Certainly more expertise and tools are needed to bootstrap than merely
> to configure + make.

Maybe the tarball should include a shell script that can achieve a
reproducible build, similar to what bootstrap does.

Jeff

Re: autoreconf --force seemingly does not forcibly update everything

[Paul Eggert]

On 4/10/24 13:36, Simon Josefsson via Gnulib discussion list wrote:

Is bootstrap intended to be reliable from within a tarball?  I thought
the bootstrap script was not included in tarballs because it wasn't
designed to be ran that way, and the way it is designed may not give
expected results.


It's pretty routinely distributed, I expect under the theory that we're 
being transparent about what sources we use to generate the tarball.


Whether it works from a tarball depends on one's definition of "works". 
Certainly more expertise and tools are needed to bootstrap than merely 
to configure + make.

Re: autoreconf --force seemingly does not forcibly update everything

[Simon Josefsson via Gnulib discussion list]

Bruno Haible  writes:

> Bernhard Voelker wrote:
>>  > Last month, I spent 2 days on prerelease testing of coreutils. If, after
>>  > downloading the carefully prepared tarball from ftp.gnu.org, the first
>>  > thing a distro does is to throw away the *.m4 files and regenerate the
>>  > configure script with their own one,
>>  >* It shows [...]
>> 
>> FWIW: especially the downstream builds of the 'coreutils' package have been
>> using 'autoreconf -fi' for a long time, because the upstream tools do not
>> have full I18N support, and the large I18N patch is in use e.g. at Fedora,
>> openSUSE and Debian probably since >15 years.
>
> Sure, if downstream applies a patch that modifies bootstrap.conf, they need
> to rerun 'bootstrap'.

Is bootstrap intended to be reliable from within a tarball?  I thought
the bootstrap script was not included in tarballs because it wasn't
designed to be ran that way, and the way it is designed may not give
expected results.  Has this changed, so we should recommend maintainers
to 'EXTRA_DIST = bootstrap bootstrap-funclib.sh bootstrap.conf' so this
is even possible?  I recall some project already added something to that
effect, but I'm not sure if that is something gnulib supports?

/Simon


signature.asc
Description: PGP signature

Re: autoreconf --force seemingly does not forcibly update everything

[Bernhard Voelker]

On 4/10/24 4:22 PM, Bruno Haible wrote:

Sure, if downstream applies a patch that modifies bootstrap.conf, they need
to rerun 'bootstrap'. That goes without saying. I hope downstream then also
runs "make check".


Sure, full automated QA is in action on all supported platforms.

Have a nice day,
Berny

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Bernhard Voelker wrote:
>  > Last month, I spent 2 days on prerelease testing of coreutils. If, after
>  > downloading the carefully prepared tarball from ftp.gnu.org, the first
>  > thing a distro does is to throw away the *.m4 files and regenerate the
>  > configure script with their own one,
>  >* It shows [...]
> 
> FWIW: especially the downstream builds of the 'coreutils' package have been
> using 'autoreconf -fi' for a long time, because the upstream tools do not
> have full I18N support, and the large I18N patch is in use e.g. at Fedora,
> openSUSE and Debian probably since >15 years.

Sure, if downstream applies a patch that modifies bootstrap.conf, they need
to rerun 'bootstrap'. That goes without saying. I hope downstream then also
runs "make check".

Bruno

Re: autoreconf --force seemingly does not forcibly update everything

[Bernhard Voelker]

On 4/1/24 11:54 PM, Bruno Haible wrote:
> Last month, I spent 2 days on prerelease testing of coreutils. If, after
> downloading the carefully prepared tarball from ftp.gnu.org, the first
> thing a distro does is to throw away the *.m4 files and regenerate the
> configure script with their own one,
>* It shows [...]

FWIW: especially the downstream builds of the 'coreutils' package have been
using 'autoreconf -fi' for a long time, because the upstream tools do not
have full I18N support, and the large I18N patch is in use e.g. at Fedora,
openSUSE and Debian probably since >15 years.
Nowadays, the coreutils-i18n.patch [1][2] gets smaller and smaller because
upstream is adding I18N step by step in one tool or another.
But we're still not there yet, unfortunately.

[1]
https://src.fedoraproject.org/rpms/coreutils/blob/main/f/coreutils-i18n.patch
[2]
https://build.opensuse.org/projects/openSUSE:Factory/packages/coreutils/files/coreutils-i18n.patch?expand=1

Have a nice day,
Berny

Re: autoreconf --force seemingly does not forcibly update everything

[Nick Bowler]

On 2024-04-09 18:06, Sam James wrote:
> Nick poses that a specific combination of tools is what is tested and
> anything else invalidates it. But how does this work when building on
> a system that was never tested on, or with different flags, or a
> different toolchain?
>
> It's reasonable to say "look, if you do this, please both state it
> clearly and also do some investigation first to see if you can
> reproduce it with my macros", but I don't think it's a crime for
> someone to attempt it either.

To be clear, I don't mean to suggest that modifying a package by
replacing m4 sources with different versions and/or regenerating
configure with a different version of Autoconf is something that
should never be done by downstream distributors.  If doing this
solves some particular problem, then by all means do it, that's
an important part of what free software is all about.

What I have a problem with is the suggestion that distributors should
systematically throw away actually-tested configure scripts by just
discarding any m4 source files that appear to be copied from another
project (Gnulib, in this case), copying in new ones from a possibly
different version of that project, regenerating the configure script
using a possibly different version of Autoconf, and then expecting
that this process will produce high-quality results.

Cheers,
  Nick

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Sam James replied to Bruno Haible who cited Nick Bowler:
> >> If I distribute a release package, what I have tested is exactly what is
> >> in that package.  If you start replacing different versions of m4 macros,
> >> or use some distribution-patched autoconf/automake/libtool or whatever,
> >> then this you have invalidated any and all release testing.
> >
> > +1
> >
> > Last month, I spent 2 days on prerelease testing of coreutils. If, after
> > downloading the carefully prepared tarball from ftp.gnu.org, the first
> > thing a distro does is to throw away the *.m4 files and regenerate the
> > configure script with their own one,
> >   * It shows no respect for the QA work that the upstream developers have
> > put in.
> ...
> Nick poses that a specific combination of tools is what is tested and
> anything else invalidates it.

Correct. When an upstream developer/tester has tested a tarball in N
situations, then that tarball is what - he can guarantee - works. The
more changes a distro applies, the more they do so on their own risk.

> But how does this work when building on a
> system that was never tested on, or with different flags, or a different
> toolchain?

The interface of GNU 'configure' [1][2] accommodates for these cases.
In most of these cases, it is not needed to rebuild 'configure'. Just
set CC, CFLAGS, LDFLAGS etc.
  - For other systems: see [3].
  - System that is still in development: Replace config.guess, config.sub,
and potentially config.rpath with modified variants.
  - Different flags: That's what CFLAGS, CXXFLAGS are for.
  - Different compiler: E.g. clang on Ubuntu 22.04
CC="/inst-clang/17.0.4/bin/clang -Wl,-rpath,/inst-clang/17.0.4/lib"
CXX="/inst-clang/17.0.4/bin/clang++ -I/usr/include/c++/11 
-I/usr/include/x86_64-linux-gnu/c++/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 
-Wl,-rpath,/inst-clang/17.0.4/lib"
  - Different linker: E.g. to link with 'mold' instead of 'ld',
create a directory /alternative-ld/with-mold that contains a symlink
ld -> /.../bin/mold
and use CC="gcc -B/alternative-ld/with-mold".

On exotic systems with non-ELF binary format, modifying libtool.m4 is needed.
But most distros are not in this situation; they use the glibc or musl dynamic
loader, hence they don't need libtool.m4 changes.

> It also has value in the context of software which is no longer
> maintained but needs to work on newer systems.

Granted; that's a different category of situation. Here a distro will
probably not only need to change *.m4 files but also *.c files. And
hopefully submit the changes upstream.

> We don't apply this rule to anything else -- you've never rejected a
> report from me because I have a newer version of a library installed
> like openssl or similar. Why is this different?

As an upstream maintainer, I have chosen (or, well, GNU has chosen for me,
even before I was present) to give *tarballs* to my users, not git
repositories. The main differences are that

  - tarballs contain some generated files for which we want to spare
the user the need to install special tools and get familiar with
them (code generators, doc formatters [texlive, doxygen, ...] etc.)

  - tarballs contain source code from other packages (git submodule,
parts of gnulib, etc.)

  - tarballs contain localizations, which are maintained elsewhere than
in the git repository (e.g. in translationproject.org or weblate
instances).

Experience has shown that this interface (tarballs with configure
script) allows for relatively effective support.

There are an entire bunch of questions from users of a git repository
(from "can you please commit the formatted documentation into git?"
over "how to I pull in the submodules?" to "why do I get this error
from flex?") that are obsoleted by this interface.

Also, too often people have reported problems with older versions of
the tools. I mean, we are at Automake 1.16.5, and if someone wants
to rebuild my package with Automake 1.13.4 because that's what his
distro is carrying, and they encounter problems, it is just a waste
of upstream developer's time. Old bugs in old versions of the tools
have been fixed. As an upstream maintainer, I don't want to support
  - different versions of Automake,
  - different versions of Bison,
  - different versions of texinfo,
  - different versionf of groff,
  - etc.
I have enough work supporting
  - different versions of the OS (glibc, Cygwin, etc.),
  - different versions of GNU make,
  - different versions of gcc and clang,
  - different versions of packages with optional support (--with-* options),
  - ...
Keep the test matrix small!

Bruno

[1] https://www.gnu.org/prep/standards/html_node/Configuration.html
[2] 
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/html_node/Preset-Output-Variables.html
[3] https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration

Re: autoreconf --force seemingly does not forcibly update everything

[Sam James]

Bruno Haible  writes:

> Nick Bowler wrote:
>> If I distribute a release package, what I have tested is exactly what is
>> in that package.  If you start replacing different versions of m4 macros,
>> or use some distribution-patched autoconf/automake/libtool or whatever,
>> then this you have invalidated any and all release testing.
>
> +1
>
> Last month, I spent 2 days on prerelease testing of coreutils. If, after
> downloading the carefully prepared tarball from ftp.gnu.org, the first
> thing a distro does is to throw away the *.m4 files and regenerate the
> configure script with their own one,
>   * It shows no respect for the QA work that the upstream developers have
> put in.

This reads as taking it a bit personally to me.

Nick poses that a specific combination of tools is what is tested and
anything else invalidates it. But how does this work when building on a
system that was never tested on, or with different flags, or a different
toolchain?

It's reasonable to say "look, if you do this, please both state it
clearly and also do some investigation first to see if you can reproduce
it with my macros", but I don't think it's a crime for someone to
attempt it either.

It also has value in the context of software which is no longer
maintained but needs to work on newer systems.

We don't apply this rule to anything else -- you've never rejected a
report from me because I have a newer version of a library installed
like openssl or similar. Why is this different?

>   * It increases the number of bug reports that reach upstream and yet
> are caused by the distro.
>   * In the long run, the upstream maintainers will be less willing to
> handle bug reports from users of this distro.
>

I do sympathise with these points and also that it might be
overwhelming given it removes the ability to fix some elements of the
build environment. Right now, you can at least assert that it builds in
a diverse set of places based on what you tested.

> Bruno

thanks,
sam


signature.asc
Description: PGP signature

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Guillem Jover wrote:
> > It may be unexpected to you, but it is very much intended.
> 
> The only unexpected part (which I perhaps failed to convey) was that
> it is being taken into account with --force.

This is documented in
https://www.gnu.org/software/automake/manual/html_node/Serials.html
where the last paragraph says:

  "Finally, note that the --force option of aclocal has absolutely
   no effect on the files installed by --install. For instance, if you
   have modified your local macros, do not expect --install --force to
   replace the local macros by their system-wide versions. If you want
   to do so, simply erase the local macros you want to revert, and run
   ‘aclocal --install’."

Bruno

Re: autoreconf --force seemingly does not forcibly update everything

[Simon Josefsson via Gnulib discussion list]

Guillem Jover  writes:

> But if as a downstream distribution I explicitly request everything to
> be considered obsolete via --force, then I really do want to get whatever
> is in the system instead of in the upstream package. Because then I
> can fix things centrally in a distribution dependency package, instead
> of having to wait for upstreams to do so, for example, or because then I
> have a higher chance of building from known system files instead of
> local stuff.

I think that is the perception that leads to problems: 'autoreconf -f
-i' will not give you what you are looking for even if we make '-f' pull
in *.m4 files regardless of serial number.  There will be other files
that are not updated from the central system package.  It was never
intended as a re-bootstrapping tool.

Nick Bowler  writes:

> On 2024-04-01 16:43, Guillem Jover wrote:
>> But if as a downstream distribution I explicitly request everything
>> to be considered obsolete via --force, then I really do want to get
>> whatever is in the system instead of in the upstream package.
>
> If I distribute a release package, what I have tested is exactly what is
> in that package.  If you start replacing different versions of m4 macros,
> or use some distribution-patched autoconf/automake/libtool or whatever,
> then this you have invalidated any and all release testing.
>
> This is fine, modifying a package and distributing modified versions
> are freedoms 1 and 3, but if it breaks you keep both pieces.
>
> The aclocal --install feature should be seen as a feature to help update
> dependencies as part of the process of preparing a modified version, not
> something that should ever be routinely performed by system integrators.
>
> GNU/Linux distributions have a long history of buggy backports to the
> autotools.  For a recent example, Gentoo shipped a broken libtool 2.4.6
> which included a patch to make Gentoo installs go faster but if you
> prepared a package with this broken libtool version, the resulting
> package would not build on HP-UX, oops.

Indeed, I think distributions are using autoconf tools in unintended
ways, and when repeatedly being told so, the response has usually been
"we know what we are doing, and want it our way for portability".
Usually because distribution packagers find it easier to run 'autoreconf
-fi' than to patch config.guess etc to support new targets.

That said, co-operation between GNU and Debian has been historically
poor, and we should try to collaborate and keep the lowest common
denominator between the projects as healthy as possible.  So if there is
something that can be improved, let's do that, but let's base it on good
design rather than "I want it my way".  I think the fundamental feature
request -- to re-bootstrap all generated files -- from Guillem is a fair
request, and arguable the GNU project has not been helpful to provide
this historically.  Instead the approach has been "we want various files
to be included in the tarball for portability".  This approach is still
important for porting to new targets, but has a cost in that it makes
auditing harder.  I believe we can support both the old way (*.tar.gz
with pre-generated content, impossible to fully re-bootstrap reliably
without risks) and a new way (*-src.tar.gz with just source code) at the
same time.  This appears to me more reliable than to fix all kind of
re-bootstrapping problems with 'autoreconf -f -i'.

/Simon


signature.asc
Description: PGP signature

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Nick Bowler wrote:
> If I distribute a release package, what I have tested is exactly what is
> in that package.  If you start replacing different versions of m4 macros,
> or use some distribution-patched autoconf/automake/libtool or whatever,
> then this you have invalidated any and all release testing.

+1

Last month, I spent 2 days on prerelease testing of coreutils. If, after
downloading the carefully prepared tarball from ftp.gnu.org, the first
thing a distro does is to throw away the *.m4 files and regenerate the
configure script with their own one,
  * It shows no respect for the QA work that the upstream developers have
put in.
  * It increases the number of bug reports that reach upstream and yet
are caused by the distro.
  * In the long run, the upstream maintainers will be less willing to
handle bug reports from users of this distro.

Bruno

Re: autoreconf --force seemingly does not forcibly update everything

[Nick Bowler]

On 2024-04-01 16:43, Guillem Jover wrote:
> But if as a downstream distribution I explicitly request everything
> to be considered obsolete via --force, then I really do want to get
> whatever is in the system instead of in the upstream package.

If I distribute a release package, what I have tested is exactly what is
in that package.  If you start replacing different versions of m4 macros,
or use some distribution-patched autoconf/automake/libtool or whatever,
then this you have invalidated any and all release testing.

This is fine, modifying a package and distributing modified versions
are freedoms 1 and 3, but if it breaks you keep both pieces.

The aclocal --install feature should be seen as a feature to help update
dependencies as part of the process of preparing a modified version, not
something that should ever be routinely performed by system integrators.

GNU/Linux distributions have a long history of buggy backports to the
autotools.  For a recent example, Gentoo shipped a broken libtool 2.4.6
which included a patch to make Gentoo installs go faster but if you
prepared a package with this broken libtool version, the resulting
package would not build on HP-UX, oops.

Cheers,
  Nick

Re: autoreconf --force seemingly does not forcibly update everything

[Guillem Jover]

Hi!

[ See also my other reply to Eric. ]

On Mon, 2024-04-01 at 20:29:59 +0200, Bruno Haible wrote:
> Guillem Jover wrote in
> :
> > > While analyzing the recent xz backdoor hook into the build system [A],
> > > I noticed that one of the aspects why the hook worked was because it
> > > seems like «autoreconf -f -i» (that is run in Debian as part of
> > > dh-autoreconf via dh) still seems to take the serial into account,
> > > which was bumped in the tampered .m4 file. If either the gettext.m4
> > > had gotten downgraded (to the version currently in Debian, which would
> > > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > > gettext, the build-to-host.m4 would get downgraded to the upstream
> > > clean version, then the hook would have been disabled and the backdoor
> > > would be inert. (Of course at that point the malicious actor would
> > > have found another way to hook into the build system, but the less
> > > avenues there are the better.)
> > > 
> > > I've tried to search the list and checked for old bug reports on the
> > > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > > a very unexpected behavior, but it's not clear whether this is intentional
> > > or a bug. In any case regardless of either position, it would be good to
> > > improve this (either by fixing --force to force things even if
> > > downgrading, or otherwise perhaps to add a new option to really force
> > > everything).
> > > 
> > > [A] 
> > > Longish mail, search for "try to go in detail" for the analysis.
> 
> The 'serial' number in *.m4 files exists precisely so that tools like
> 'aclocal --install' or (as you say) 'autoreconf -f -i' can avoid
> overwriting newer .m4 files with older versions.

As mentioned in my other reply, I think taking the serial into account
in non-force mode makes total sense. But when using --force the
current behavior does not. The option description says:

-f, --force consider all generated and standard files obsolete

which to me would imply the serial should be considered missing or as
if set to 0 or equivalent.

> The situation is:
>   - Debian stable is still on gettext 0.21, which comes with
>   gettext.m4 serial 71

(This does not change much because in Debian stable and unstable
are both 0.21, but the packages go through unstable and that's where
this incident happened in Debian.)

>   - gettext-0.22 comes with
>   gettext.m4 serial 77
> which depends on build-to-host.m4.
> Assume furthermore that a package P relies on gettext (via AM_GNU_GETTEXT),
> and the user has installed gettext-0.22 or newer. And they run
>   $ autoreconf
> which invokes autopoint, which installs the gettext.m4.
> 
> If, in this situation, the newer .m4 file would not prevail, Debian's
> old gettext.m4 would take precedence. This would mean that new features
> from new gettext releases would not land in the users' hands; they
> would be blocked by Debian, for the time the user uses this distro+version
> (often 4 or 5 years).

If someone runes «autoreconf» or «autoreconf -i» I'd expect the serial
to be honored and .m4 files to not ever get downgraded (whether they
should always get upgraded I don't have an opinion at this point).

From an upstream PoV I sympathize with a desire to not lose features,
but then autotools are intended precisely to cope with old systems
with whatever might or might not be present and at different versions
and feature sets.

But if as a downstream distribution I explicitly request everything to
be considered obsolete via --force, then I really do want to get whatever
is in the system instead of in the upstream package. Because then I
can fix things centrally in a distribution dependency package, instead
of having to wait for upstreams to do so, for example, or because then I
have a higher chance of building from known system files instead of
local stuff.

> Worse, this would also hold for Red Hat distro releases, which typically
> are not 4-5 years, but 10-12 years behind the newest release. It would
> mean that *for 10 years*, packages P could not make use of new features
> (and bug fixes) that were added in GNU gettext upstream.

I'm not sure, but I think this is perhaps conflating the roles of the
distribution packages (which would have been frozen at the time they
got added to that release anyway), and of a user of such distribution
long after that release was done? For the latter I agree they should
have the choice, and to me that involves whether they use for example
«autoreconf -i» vs «autoreconf -f -i», or even no autoreconf at all!

> This is the problem that the 'serial' number annotation fixes.

I'm not denying the serial number solves problems! In case that was
not clear. :) I'm also using serial numbers in my own upstream .m4
files.

> It may be unexpected to you, but it is ver

Re: autoreconf --force seemingly does not forcibly update everything

[Simon Josefsson via Gnulib discussion list]

Eric Blake  writes:

> Widening the audience to include bug-gnulib, which is the upstream
> source of "# build-to-host.m4 serial 3" which was bypassed by the
> malicious "# build-to-host.m4 serial 30".
>
> On Sun, Mar 31, 2024 at 11:51:36PM +0200, Guillem Jover wrote:
>> Hi!
>> 
>> While analyzing the recent xz backdoor hook into the build system [A],
>> I noticed that one of the aspects why the hook worked was because it
>> seems like «autoreconf -f -i» (that is run in Debian as part of
>> dh-autoreconf via dh) still seems to take the serial into account,
>> which was bumped in the tampered .m4 file. If either the gettext.m4
>> had gotten downgraded (to the version currently in Debian, which would
>> not have pulled the tampered build-to-host.m4), or once Debian upgrades
>> gettext, the build-to-host.m4 would get downgraded to the upstream
>> clean version, then the hook would have been disabled and the backdoor
>> would be inert. (Of course at that point the malicious actor would
>> have found another way to hook into the build system, but the less
>> avenues there are the better.)
>> 
>> I've tried to search the list and checked for old bug reports on the
>> debbugs.gnu.org site, but didn't notice anything. To me this looks like
>> a very unexpected behavior, but it's not clear whether this is intentional
>> or a bug. In any case regardless of either position, it would be good to
>> improve this (either by fixing --force to force things even if
>> downgrading, or otherwise perhaps to add a new option to really force
>> everything).
>> 
>> [A] 
>> Longish mail, search for "try to go in detail" for the analysis.
>
> My understanding is that the use of serial numbers in .m4 snippets was
> intentional in gnulib (more or less where the practice originated),
> but only because gnulib prefers a linear history (everything is
> monotonically increasing, no forks for the serial number to diverge
> on).  In light of this weekend's mess, Bruno may have more ideas about
> how to prevent his files from being turned into backdoor delivery
> mechanisms in the future.

I think the root cause here is assuming 'autoreconf -fi' achieves
anything related to re-bootstrapping.  I think the entire concept of
re-bootstrapping from a source tarball with generated contents in it is
fundamentally flawed.  I have proposed that we should start to release
*-src.tar.gz tarballs that doesn't have any pre-generated in it, that
can be completely bootstrapped using external tools.  See writeup here:

https://blog.josefsson.org/2024/04/01/towards-reproducible-minimal-source-code-tarballs-please-welcome-src-tar-gz/

To me, moving things towards this approach allows incremental work that
eventually will be more reliable than anything that attempts to
re-boostrap from a tarball with some pre-generated artifacts in it
(because there will always be uncertainty if the artifact used was
actually built or came from the tarball).

I suggest that we extend 'make dist' to produce these *-src.tar.gz
tarballs, possibly only when some new automake AM_INIT_AUTOMAKE flag is
used.  There could be some functions to modify how the tarball is
generated, much like we have dist-hooks today that is often used to
generate ChangeLog for the tarballs.  Thoughts?

/Simon


signature.asc
Description: PGP signature

Re: autoreconf --force seemingly does not forcibly update everything

[Guillem Jover]

Hi!

On Mon, 2024-04-01 at 12:43:02 -0500, Eric Blake wrote:
> Widening the audience to include bug-gnulib, which is the upstream
> source of "# build-to-host.m4 serial 3" which was bypassed by the
> malicious "# build-to-host.m4 serial 30".
> 
> On Sun, Mar 31, 2024 at 11:51:36PM +0200, Guillem Jover wrote:
> > While analyzing the recent xz backdoor hook into the build system [A],
> > I noticed that one of the aspects why the hook worked was because it
> > seems like «autoreconf -f -i» (that is run in Debian as part of
> > dh-autoreconf via dh) still seems to take the serial into account,
> > which was bumped in the tampered .m4 file. If either the gettext.m4
> > had gotten downgraded (to the version currently in Debian, which would
> > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > gettext, the build-to-host.m4 would get downgraded to the upstream
> > clean version, then the hook would have been disabled and the backdoor
> > would be inert. (Of course at that point the malicious actor would
> > have found another way to hook into the build system, but the less
> > avenues there are the better.)
> > 
> > I've tried to search the list and checked for old bug reports on the
> > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > a very unexpected behavior, but it's not clear whether this is intentional
> > or a bug. In any case regardless of either position, it would be good to
> > improve this (either by fixing --force to force things even if
> > downgrading, or otherwise perhaps to add a new option to really force
> > everything).
> > 
> > [A] 
> > Longish mail, search for "try to go in detail" for the analysis.
> 
> My understanding is that the use of serial numbers in .m4 snippets was
> intentional in gnulib (more or less where the practice originated),
> but only because gnulib prefers a linear history (everything is
> monotonically increasing, no forks for the serial number to diverge
> on).  In light of this weekend's mess, Bruno may have more ideas about
> how to prevent his files from being turned into backdoor delivery
> mechanisms in the future.

I think the serial numbers and the non-downgrade behavior makes sense
in general in non-force mode, but I still expect and I find it very
surprising that --force does not overwrite everything, when that seems
like what's been asked.

Of course the problem then becomes, how much might might now break if
--force would truly force a refresh from system files, because I'm
assuming this has been the behavior all along?

At least in Debian I guess we could perform a mass rebuild with a
modified autoreconf that truly forces to see how much breaks. But even
if much breaks, I think that fallout is something we might want to fix
anyway, but perhaps in a controlled/staged way.

Thanks,
Guillem

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Jeffrey Walton wrote:
> Please forgive my ignorance... If you bump the authentic version of
> the m4 file to version 31, will the issue mostly clear itself?

If we bump gnulib's build-to-host.m4 to 'serial 31', this will override
the one from xz-5.6.x in *some* situations. In other situations, it will
not help. [1]

Therefore it is more reliable to curate the xz releases at their
download locations and in the distros.

Bruno

[1] https://www.gnu.org/software/automake/manual/html_node/Serials.html

Re: autoreconf --force seemingly does not forcibly update everything

[Jeffrey Walton]

On Mon, Apr 1, 2024 at 2:31 PM Bruno Haible  wrote:
>
> Thanks for the forward, Eric.
>
> Guillem Jover wrote in
> :
> > > Hi!
> > >
> > > While analyzing the recent xz backdoor hook into the build system [A],
> > > I noticed that one of the aspects why the hook worked was because it
> > > seems like «autoreconf -f -i» (that is run in Debian as part of
> > > dh-autoreconf via dh) still seems to take the serial into account,
> > > which was bumped in the tampered .m4 file. If either the gettext.m4
> > > had gotten downgraded (to the version currently in Debian, which would
> > > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > > gettext, the build-to-host.m4 would get downgraded to the upstream
> > > clean version, then the hook would have been disabled and the backdoor
> > > would be inert. (Of course at that point the malicious actor would
> > > have found another way to hook into the build system, but the less
> > > avenues there are the better.)
> > >
> > > I've tried to search the list and checked for old bug reports on the
> > > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > > a very unexpected behavior, but it's not clear whether this is intentional
> > > or a bug. In any case regardless of either position, it would be good to
> > > improve this (either by fixing --force to force things even if
> > > downgrading, or otherwise perhaps to add a new option to really force
> > > everything).
> > >
> > > [A] 
> > > Longish mail, search for "try to go in detail" for the analysis.
>
> The 'serial' number in *.m4 files exists precisely so that tools like
> 'aclocal --install' or (as you say) 'autoreconf -f -i' can avoid
> overwriting newer .m4 files with older versions.
>
> I don't know exactly about the situations in which this happens. But
> your investigations shed enough light on it. You write:
>   "If either the gettext.m4
>had gotten downgraded (to the version currently in Debian, which would
>not have pulled the tampered build-to-host.m4), or once Debian upgrades
>gettext, the build-to-host.m4 would get downgraded to the upstream
>clean version"
>
> The situation is:
>   - Debian stable is still on gettext 0.21, which comes with
>   gettext.m4 serial 71
>   - gettext-0.22 comes with
>   gettext.m4 serial 77
> which depends on build-to-host.m4.
> Assume furthermore that a package P relies on gettext (via AM_GNU_GETTEXT),
> and the user has installed gettext-0.22 or newer. And they run
>   $ autoreconf
> which invokes autopoint, which installs the gettext.m4.
>
> If, in this situation, the newer .m4 file would not prevail, Debian's
> old gettext.m4 would take precedence. This would mean that new features
> from new gettext releases would not land in the users' hands; they
> would be blocked by Debian, for the time the user uses this distro+version
> (often 4 or 5 years).
>
> Worse, this would also hold for Red Hat distro releases, which typically
> are not 4-5 years, but 10-12 years behind the newest release. It would
> mean that *for 10 years*, packages P could not make use of new features
> (and bug fixes) that were added in GNU gettext upstream.
>
> This is the problem that the 'serial' number annotation fixes.
>
> It may be unexpected to you, but it is very much intended.
>
> And if aclocal's preference was the other way around, always favouring
> the smaller serial number, then the attacker would not have picked
> 'serial 30' but 'serial 1', in order to override the distro's version.
>
> > My understanding is that the use of serial numbers in .m4 snippets was
> > intentional in gnulib (more or less where the practice originated),
> > but only because gnulib prefers a linear history (everything is
> > monotonically increasing, no forks for the serial number to diverge
> > on).
>
> The history is mostly linear, yes. There is a bit of nonlinearity
>   - because gettext.m4 gets copied from gettext to gnulib and/or
> from gnulib to gettext,
>   - because gettext has a release branch,
>   - because gnulib has stable branches,
> but all in all, it's linear.
>
> > In light of this weekend's mess, Bruno may have more ideas about
> > how to prevent his files from being turned into backdoor delivery
> > mechanisms in the future.
>
> I gave my opinion here:
>   https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00422.html
>   https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00421.html
>
> Basically I think that an attacker who is clever enough to bypass the
> RELRO protection by using IFUNC, knowing that IFUNC gets processed
> _before_ the GOT or PLT pages are made read-only, — such an attacker
> will also find ways to conceal their malware triggers, regardless of
> any simple measures at the source code level that we might invent.

Please forgive my ignorance... If you bump t

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Thanks for the forward, Eric.

Guillem Jover wrote in
:
> > Hi!
> > 
> > While analyzing the recent xz backdoor hook into the build system [A],
> > I noticed that one of the aspects why the hook worked was because it
> > seems like «autoreconf -f -i» (that is run in Debian as part of
> > dh-autoreconf via dh) still seems to take the serial into account,
> > which was bumped in the tampered .m4 file. If either the gettext.m4
> > had gotten downgraded (to the version currently in Debian, which would
> > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > gettext, the build-to-host.m4 would get downgraded to the upstream
> > clean version, then the hook would have been disabled and the backdoor
> > would be inert. (Of course at that point the malicious actor would
> > have found another way to hook into the build system, but the less
> > avenues there are the better.)
> > 
> > I've tried to search the list and checked for old bug reports on the
> > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > a very unexpected behavior, but it's not clear whether this is intentional
> > or a bug. In any case regardless of either position, it would be good to
> > improve this (either by fixing --force to force things even if
> > downgrading, or otherwise perhaps to add a new option to really force
> > everything).
> > 
> > [A] 
> > Longish mail, search for "try to go in detail" for the analysis.

The 'serial' number in *.m4 files exists precisely so that tools like
'aclocal --install' or (as you say) 'autoreconf -f -i' can avoid
overwriting newer .m4 files with older versions.

I don't know exactly about the situations in which this happens. But
your investigations shed enough light on it. You write:
  "If either the gettext.m4
   had gotten downgraded (to the version currently in Debian, which would
   not have pulled the tampered build-to-host.m4), or once Debian upgrades
   gettext, the build-to-host.m4 would get downgraded to the upstream
   clean version"

The situation is:
  - Debian stable is still on gettext 0.21, which comes with
  gettext.m4 serial 71
  - gettext-0.22 comes with
  gettext.m4 serial 77
which depends on build-to-host.m4.
Assume furthermore that a package P relies on gettext (via AM_GNU_GETTEXT),
and the user has installed gettext-0.22 or newer. And they run
  $ autoreconf
which invokes autopoint, which installs the gettext.m4.

If, in this situation, the newer .m4 file would not prevail, Debian's
old gettext.m4 would take precedence. This would mean that new features
from new gettext releases would not land in the users' hands; they
would be blocked by Debian, for the time the user uses this distro+version
(often 4 or 5 years).

Worse, this would also hold for Red Hat distro releases, which typically
are not 4-5 years, but 10-12 years behind the newest release. It would
mean that *for 10 years*, packages P could not make use of new features
(and bug fixes) that were added in GNU gettext upstream.

This is the problem that the 'serial' number annotation fixes.

It may be unexpected to you, but it is very much intended.

And if aclocal's preference was the other way around, always favouring
the smaller serial number, then the attacker would not have picked
'serial 30' but 'serial 1', in order to override the distro's version.

> My understanding is that the use of serial numbers in .m4 snippets was
> intentional in gnulib (more or less where the practice originated),
> but only because gnulib prefers a linear history (everything is
> monotonically increasing, no forks for the serial number to diverge
> on).

The history is mostly linear, yes. There is a bit of nonlinearity
  - because gettext.m4 gets copied from gettext to gnulib and/or
from gnulib to gettext,
  - because gettext has a release branch,
  - because gnulib has stable branches,
but all in all, it's linear.

> In light of this weekend's mess, Bruno may have more ideas about
> how to prevent his files from being turned into backdoor delivery
> mechanisms in the future.

I gave my opinion here:
  https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00422.html
  https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00421.html

Basically I think that an attacker who is clever enough to bypass the
RELRO protection by using IFUNC, knowing that IFUNC gets processed
_before_ the GOT or PLT pages are made read-only, — such an attacker
will also find ways to conceal their malware triggers, regardless of
any simple measures at the source code level that we might invent.

Bruno

Re: autoreconf --force seemingly does not forcibly update everything

[Eric Blake]

Widening the audience to include bug-gnulib, which is the upstream
source of "# build-to-host.m4 serial 3" which was bypassed by the
malicious "# build-to-host.m4 serial 30".

On Sun, Mar 31, 2024 at 11:51:36PM +0200, Guillem Jover wrote:
> Hi!
> 
> While analyzing the recent xz backdoor hook into the build system [A],
> I noticed that one of the aspects why the hook worked was because it
> seems like «autoreconf -f -i» (that is run in Debian as part of
> dh-autoreconf via dh) still seems to take the serial into account,
> which was bumped in the tampered .m4 file. If either the gettext.m4
> had gotten downgraded (to the version currently in Debian, which would
> not have pulled the tampered build-to-host.m4), or once Debian upgrades
> gettext, the build-to-host.m4 would get downgraded to the upstream
> clean version, then the hook would have been disabled and the backdoor
> would be inert. (Of course at that point the malicious actor would
> have found another way to hook into the build system, but the less
> avenues there are the better.)
> 
> I've tried to search the list and checked for old bug reports on the
> debbugs.gnu.org site, but didn't notice anything. To me this looks like
> a very unexpected behavior, but it's not clear whether this is intentional
> or a bug. In any case regardless of either position, it would be good to
> improve this (either by fixing --force to force things even if
> downgrading, or otherwise perhaps to add a new option to really force
> everything).
> 
> [A] 
> Longish mail, search for "try to go in detail" for the analysis.

My understanding is that the use of serial numbers in .m4 snippets was
intentional in gnulib (more or less where the practice originated),
but only because gnulib prefers a linear history (everything is
monotonically increasing, no forks for the serial number to diverge
on).  In light of this weekend's mess, Bruno may have more ideas about
how to prevent his files from being turned into backdoor delivery
mechanisms in the future.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org