Re: autoreconf --force seemingly does not forcibly update everything

[Simon Josefsson via Bug reports for autoconf]

Guillem Jover  writes:

> But if as a downstream distribution I explicitly request everything to
> be considered obsolete via --force, then I really do want to get whatever
> is in the system instead of in the upstream package. Because then I
> can fix things centrally in a distribution dependency package, instead
> of having to wait for upstreams to do so, for example, or because then I
> have a higher chance of building from known system files instead of
> local stuff.

I think that is the perception that leads to problems: 'autoreconf -f
-i' will not give you what you are looking for even if we make '-f' pull
in *.m4 files regardless of serial number.  There will be other files
that are not updated from the central system package.  It was never
intended as a re-bootstrapping tool.

Nick Bowler  writes:

> On 2024-04-01 16:43, Guillem Jover wrote:
>> But if as a downstream distribution I explicitly request everything
>> to be considered obsolete via --force, then I really do want to get
>> whatever is in the system instead of in the upstream package.
>
> If I distribute a release package, what I have tested is exactly what is
> in that package.  If you start replacing different versions of m4 macros,
> or use some distribution-patched autoconf/automake/libtool or whatever,
> then this you have invalidated any and all release testing.
>
> This is fine, modifying a package and distributing modified versions
> are freedoms 1 and 3, but if it breaks you keep both pieces.
>
> The aclocal --install feature should be seen as a feature to help update
> dependencies as part of the process of preparing a modified version, not
> something that should ever be routinely performed by system integrators.
>
> GNU/Linux distributions have a long history of buggy backports to the
> autotools.  For a recent example, Gentoo shipped a broken libtool 2.4.6
> which included a patch to make Gentoo installs go faster but if you
> prepared a package with this broken libtool version, the resulting
> package would not build on HP-UX, oops.

Indeed, I think distributions are using autoconf tools in unintended
ways, and when repeatedly being told so, the response has usually been
"we know what we are doing, and want it our way for portability".
Usually because distribution packagers find it easier to run 'autoreconf
-fi' than to patch config.guess etc to support new targets.

That said, co-operation between GNU and Debian has been historically
poor, and we should try to collaborate and keep the lowest common
denominator between the projects as healthy as possible.  So if there is
something that can be improved, let's do that, but let's base it on good
design rather than "I want it my way".  I think the fundamental feature
request -- to re-bootstrap all generated files -- from Guillem is a fair
request, and arguable the GNU project has not been helpful to provide
this historically.  Instead the approach has been "we want various files
to be included in the tarball for portability".  This approach is still
important for porting to new targets, but has a cost in that it makes
auditing harder.  I believe we can support both the old way (*.tar.gz
with pre-generated content, impossible to fully re-bootstrap reliably
without risks) and a new way (*-src.tar.gz with just source code) at the
same time.  This appears to me more reliable than to fix all kind of
re-bootstrapping problems with 'autoreconf -f -i'.

/Simon


signature.asc
Description: PGP signature

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Nick Bowler wrote:
> If I distribute a release package, what I have tested is exactly what is
> in that package.  If you start replacing different versions of m4 macros,
> or use some distribution-patched autoconf/automake/libtool or whatever,
> then this you have invalidated any and all release testing.

+1

Last month, I spent 2 days on prerelease testing of coreutils. If, after
downloading the carefully prepared tarball from ftp.gnu.org, the first
thing a distro does is to throw away the *.m4 files and regenerate the
configure script with their own one,
  * It shows no respect for the QA work that the upstream developers have
put in.
  * It increases the number of bug reports that reach upstream and yet
are caused by the distro.
  * In the long run, the upstream maintainers will be less willing to
handle bug reports from users of this distro.

Bruno

Re: [sr #111044] autoconf should assert existence of all subsidiary tools at startup

[Frank Ch. Eigler]

Hi -

On Mon, Apr 01, 2024 at 05:10:17PM -0400, Paul Eggert wrote:
> [...]
> Not sure I'd go that far. The
> [https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html GNU
> Coding Standards for utilities in makefiles] lists the following as usable
> without further ado:
> 
> awk cat cmp cp diff echo egrep expr false grep install-info ln ls
> mkdir mv printf pwd rm rmdir sed sleep sort tar test touch tr true
> [...]

On the other hand, we've had very real reports that e.g. diff was
missing on some real platforms, which led autoconf astray and ended up
in misconfigured build trees.  Perhaps autoconf can afford to do some
sanity checking on these reasonable-sounding but not-universal
expectations.


- FChE

Re: autoreconf --force seemingly does not forcibly update everything

[Nick Bowler]

On 2024-04-01 16:43, Guillem Jover wrote:
> But if as a downstream distribution I explicitly request everything
> to be considered obsolete via --force, then I really do want to get
> whatever is in the system instead of in the upstream package.

If I distribute a release package, what I have tested is exactly what is
in that package.  If you start replacing different versions of m4 macros,
or use some distribution-patched autoconf/automake/libtool or whatever,
then this you have invalidated any and all release testing.

This is fine, modifying a package and distributing modified versions
are freedoms 1 and 3, but if it breaks you keep both pieces.

The aclocal --install feature should be seen as a feature to help update
dependencies as part of the process of preparing a modified version, not
something that should ever be routinely performed by system integrators.

GNU/Linux distributions have a long history of buggy backports to the
autotools.  For a recent example, Gentoo shipped a broken libtool 2.4.6
which included a patch to make Gentoo installs go faster but if you
prepared a package with this broken libtool version, the resulting
package would not build on HP-UX, oops.

Cheers,
  Nick

[sr #111044] autoconf should assert existence of all subsidiary tools at startup

[Paul Eggert]

Follow-up Comment #3, sr #111044 (group autoconf):

[comment #2 comment #2:]
> neither `diff` nor `awk` (and arguably not even `sed`) should be an implicit
dependency.

Not sure I'd go that far. The
[https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html GNU
Coding Standards for utilities in makefiles] lists the following as usable
without further ado:

awk cat cmp cp diff echo egrep expr false grep install-info ln ls
mkdir mv printf pwd rm rmdir sed sleep sort tar test touch tr true


Although some of these are debatable (egrep is no longer required by POSIX and
people should use 'grep -E', install-info is needed only if using info) it's
not a bad list of utilities that should work everywhere. I'd hate for Autoconf
to have to worry about platforms lacking 'rm'


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

Re: autoreconf --force seemingly does not forcibly update everything

[Guillem Jover]

Hi!

[ See also my other reply to Eric. ]

On Mon, 2024-04-01 at 20:29:59 +0200, Bruno Haible wrote:
> Guillem Jover wrote in
> :
> > > While analyzing the recent xz backdoor hook into the build system [A],
> > > I noticed that one of the aspects why the hook worked was because it
> > > seems like «autoreconf -f -i» (that is run in Debian as part of
> > > dh-autoreconf via dh) still seems to take the serial into account,
> > > which was bumped in the tampered .m4 file. If either the gettext.m4
> > > had gotten downgraded (to the version currently in Debian, which would
> > > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > > gettext, the build-to-host.m4 would get downgraded to the upstream
> > > clean version, then the hook would have been disabled and the backdoor
> > > would be inert. (Of course at that point the malicious actor would
> > > have found another way to hook into the build system, but the less
> > > avenues there are the better.)
> > > 
> > > I've tried to search the list and checked for old bug reports on the
> > > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > > a very unexpected behavior, but it's not clear whether this is intentional
> > > or a bug. In any case regardless of either position, it would be good to
> > > improve this (either by fixing --force to force things even if
> > > downgrading, or otherwise perhaps to add a new option to really force
> > > everything).
> > > 
> > > [A] 
> > > Longish mail, search for "try to go in detail" for the analysis.
> 
> The 'serial' number in *.m4 files exists precisely so that tools like
> 'aclocal --install' or (as you say) 'autoreconf -f -i' can avoid
> overwriting newer .m4 files with older versions.

As mentioned in my other reply, I think taking the serial into account
in non-force mode makes total sense. But when using --force the
current behavior does not. The option description says:

-f, --force consider all generated and standard files obsolete

which to me would imply the serial should be considered missing or as
if set to 0 or equivalent.

> The situation is:
>   - Debian stable is still on gettext 0.21, which comes with
>   gettext.m4 serial 71

(This does not change much because in Debian stable and unstable
are both 0.21, but the packages go through unstable and that's where
this incident happened in Debian.)

>   - gettext-0.22 comes with
>   gettext.m4 serial 77
> which depends on build-to-host.m4.
> Assume furthermore that a package P relies on gettext (via AM_GNU_GETTEXT),
> and the user has installed gettext-0.22 or newer. And they run
>   $ autoreconf
> which invokes autopoint, which installs the gettext.m4.
> 
> If, in this situation, the newer .m4 file would not prevail, Debian's
> old gettext.m4 would take precedence. This would mean that new features
> from new gettext releases would not land in the users' hands; they
> would be blocked by Debian, for the time the user uses this distro+version
> (often 4 or 5 years).

If someone runes «autoreconf» or «autoreconf -i» I'd expect the serial
to be honored and .m4 files to not ever get downgraded (whether they
should always get upgraded I don't have an opinion at this point).

From an upstream PoV I sympathize with a desire to not lose features,
but then autotools are intended precisely to cope with old systems
with whatever might or might not be present and at different versions
and feature sets.

But if as a downstream distribution I explicitly request everything to
be considered obsolete via --force, then I really do want to get whatever
is in the system instead of in the upstream package. Because then I
can fix things centrally in a distribution dependency package, instead
of having to wait for upstreams to do so, for example, or because then I
have a higher chance of building from known system files instead of
local stuff.

> Worse, this would also hold for Red Hat distro releases, which typically
> are not 4-5 years, but 10-12 years behind the newest release. It would
> mean that *for 10 years*, packages P could not make use of new features
> (and bug fixes) that were added in GNU gettext upstream.

I'm not sure, but I think this is perhaps conflating the roles of the
distribution packages (which would have been frozen at the time they
got added to that release anyway), and of a user of such distribution
long after that release was done? For the latter I agree they should
have the choice, and to me that involves whether they use for example
«autoreconf -i» vs «autoreconf -f -i», or even no autoreconf at all!

> This is the problem that the 'serial' number annotation fixes.

I'm not denying the serial number solves problems! In case that was
not clear. :) I'm also using serial numbers in my own upstream .m4
files.

> It may be unexpected to you, but it is 

Re: autoreconf --force seemingly does not forcibly update everything

[Simon Josefsson via Bug reports for autoconf]

Eric Blake  writes:

> Widening the audience to include bug-gnulib, which is the upstream
> source of "# build-to-host.m4 serial 3" which was bypassed by the
> malicious "# build-to-host.m4 serial 30".
>
> On Sun, Mar 31, 2024 at 11:51:36PM +0200, Guillem Jover wrote:
>> Hi!
>> 
>> While analyzing the recent xz backdoor hook into the build system [A],
>> I noticed that one of the aspects why the hook worked was because it
>> seems like «autoreconf -f -i» (that is run in Debian as part of
>> dh-autoreconf via dh) still seems to take the serial into account,
>> which was bumped in the tampered .m4 file. If either the gettext.m4
>> had gotten downgraded (to the version currently in Debian, which would
>> not have pulled the tampered build-to-host.m4), or once Debian upgrades
>> gettext, the build-to-host.m4 would get downgraded to the upstream
>> clean version, then the hook would have been disabled and the backdoor
>> would be inert. (Of course at that point the malicious actor would
>> have found another way to hook into the build system, but the less
>> avenues there are the better.)
>> 
>> I've tried to search the list and checked for old bug reports on the
>> debbugs.gnu.org site, but didn't notice anything. To me this looks like
>> a very unexpected behavior, but it's not clear whether this is intentional
>> or a bug. In any case regardless of either position, it would be good to
>> improve this (either by fixing --force to force things even if
>> downgrading, or otherwise perhaps to add a new option to really force
>> everything).
>> 
>> [A] 
>> Longish mail, search for "try to go in detail" for the analysis.
>
> My understanding is that the use of serial numbers in .m4 snippets was
> intentional in gnulib (more or less where the practice originated),
> but only because gnulib prefers a linear history (everything is
> monotonically increasing, no forks for the serial number to diverge
> on).  In light of this weekend's mess, Bruno may have more ideas about
> how to prevent his files from being turned into backdoor delivery
> mechanisms in the future.

I think the root cause here is assuming 'autoreconf -fi' achieves
anything related to re-bootstrapping.  I think the entire concept of
re-bootstrapping from a source tarball with generated contents in it is
fundamentally flawed.  I have proposed that we should start to release
*-src.tar.gz tarballs that doesn't have any pre-generated in it, that
can be completely bootstrapped using external tools.  See writeup here:

https://blog.josefsson.org/2024/04/01/towards-reproducible-minimal-source-code-tarballs-please-welcome-src-tar-gz/

To me, moving things towards this approach allows incremental work that
eventually will be more reliable than anything that attempts to
re-boostrap from a tarball with some pre-generated artifacts in it
(because there will always be uncertainty if the artifact used was
actually built or came from the tarball).

I suggest that we extend 'make dist' to produce these *-src.tar.gz
tarballs, possibly only when some new automake AM_INIT_AUTOMAKE flag is
used.  There could be some functions to modify how the tarball is
generated, much like we have dist-hooks today that is often used to
generate ChangeLog for the tarballs.  Thoughts?

/Simon


signature.asc
Description: PGP signature

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Jeffrey Walton wrote:
> Please forgive my ignorance... If you bump the authentic version of
> the m4 file to version 31, will the issue mostly clear itself?

If we bump gnulib's build-to-host.m4 to 'serial 31', this will override
the one from xz-5.6.x in *some* situations. In other situations, it will
not help. [1]

Therefore it is more reliable to curate the xz releases at their
download locations and in the distros.

Bruno

[1] https://www.gnu.org/software/automake/manual/html_node/Serials.html

Re: autoreconf --force seemingly does not forcibly update everything

[Jeffrey Walton]

On Mon, Apr 1, 2024 at 2:31 PM Bruno Haible  wrote:
>
> Thanks for the forward, Eric.
>
> Guillem Jover wrote in
> :
> > > Hi!
> > >
> > > While analyzing the recent xz backdoor hook into the build system [A],
> > > I noticed that one of the aspects why the hook worked was because it
> > > seems like «autoreconf -f -i» (that is run in Debian as part of
> > > dh-autoreconf via dh) still seems to take the serial into account,
> > > which was bumped in the tampered .m4 file. If either the gettext.m4
> > > had gotten downgraded (to the version currently in Debian, which would
> > > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > > gettext, the build-to-host.m4 would get downgraded to the upstream
> > > clean version, then the hook would have been disabled and the backdoor
> > > would be inert. (Of course at that point the malicious actor would
> > > have found another way to hook into the build system, but the less
> > > avenues there are the better.)
> > >
> > > I've tried to search the list and checked for old bug reports on the
> > > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > > a very unexpected behavior, but it's not clear whether this is intentional
> > > or a bug. In any case regardless of either position, it would be good to
> > > improve this (either by fixing --force to force things even if
> > > downgrading, or otherwise perhaps to add a new option to really force
> > > everything).
> > >
> > > [A] 
> > > Longish mail, search for "try to go in detail" for the analysis.
>
> The 'serial' number in *.m4 files exists precisely so that tools like
> 'aclocal --install' or (as you say) 'autoreconf -f -i' can avoid
> overwriting newer .m4 files with older versions.
>
> I don't know exactly about the situations in which this happens. But
> your investigations shed enough light on it. You write:
>   "If either the gettext.m4
>had gotten downgraded (to the version currently in Debian, which would
>not have pulled the tampered build-to-host.m4), or once Debian upgrades
>gettext, the build-to-host.m4 would get downgraded to the upstream
>clean version"
>
> The situation is:
>   - Debian stable is still on gettext 0.21, which comes with
>   gettext.m4 serial 71
>   - gettext-0.22 comes with
>   gettext.m4 serial 77
> which depends on build-to-host.m4.
> Assume furthermore that a package P relies on gettext (via AM_GNU_GETTEXT),
> and the user has installed gettext-0.22 or newer. And they run
>   $ autoreconf
> which invokes autopoint, which installs the gettext.m4.
>
> If, in this situation, the newer .m4 file would not prevail, Debian's
> old gettext.m4 would take precedence. This would mean that new features
> from new gettext releases would not land in the users' hands; they
> would be blocked by Debian, for the time the user uses this distro+version
> (often 4 or 5 years).
>
> Worse, this would also hold for Red Hat distro releases, which typically
> are not 4-5 years, but 10-12 years behind the newest release. It would
> mean that *for 10 years*, packages P could not make use of new features
> (and bug fixes) that were added in GNU gettext upstream.
>
> This is the problem that the 'serial' number annotation fixes.
>
> It may be unexpected to you, but it is very much intended.
>
> And if aclocal's preference was the other way around, always favouring
> the smaller serial number, then the attacker would not have picked
> 'serial 30' but 'serial 1', in order to override the distro's version.
>
> > My understanding is that the use of serial numbers in .m4 snippets was
> > intentional in gnulib (more or less where the practice originated),
> > but only because gnulib prefers a linear history (everything is
> > monotonically increasing, no forks for the serial number to diverge
> > on).
>
> The history is mostly linear, yes. There is a bit of nonlinearity
>   - because gettext.m4 gets copied from gettext to gnulib and/or
> from gnulib to gettext,
>   - because gettext has a release branch,
>   - because gnulib has stable branches,
> but all in all, it's linear.
>
> > In light of this weekend's mess, Bruno may have more ideas about
> > how to prevent his files from being turned into backdoor delivery
> > mechanisms in the future.
>
> I gave my opinion here:
>   https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00422.html
>   https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00421.html
>
> Basically I think that an attacker who is clever enough to bypass the
> RELRO protection by using IFUNC, knowing that IFUNC gets processed
> _before_ the GOT or PLT pages are made read-only, — such an attacker
> will also find ways to conceal their malware triggers, regardless of
> any simple measures at the source code level that we might invent.

Please forgive my ignorance... If you bump 

Re: autoreconf --force seemingly does not forcibly update everything

[Bruno Haible]

Thanks for the forward, Eric.

Guillem Jover wrote in
:
> > Hi!
> > 
> > While analyzing the recent xz backdoor hook into the build system [A],
> > I noticed that one of the aspects why the hook worked was because it
> > seems like «autoreconf -f -i» (that is run in Debian as part of
> > dh-autoreconf via dh) still seems to take the serial into account,
> > which was bumped in the tampered .m4 file. If either the gettext.m4
> > had gotten downgraded (to the version currently in Debian, which would
> > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > gettext, the build-to-host.m4 would get downgraded to the upstream
> > clean version, then the hook would have been disabled and the backdoor
> > would be inert. (Of course at that point the malicious actor would
> > have found another way to hook into the build system, but the less
> > avenues there are the better.)
> > 
> > I've tried to search the list and checked for old bug reports on the
> > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > a very unexpected behavior, but it's not clear whether this is intentional
> > or a bug. In any case regardless of either position, it would be good to
> > improve this (either by fixing --force to force things even if
> > downgrading, or otherwise perhaps to add a new option to really force
> > everything).
> > 
> > [A] 
> > Longish mail, search for "try to go in detail" for the analysis.

The 'serial' number in *.m4 files exists precisely so that tools like
'aclocal --install' or (as you say) 'autoreconf -f -i' can avoid
overwriting newer .m4 files with older versions.

I don't know exactly about the situations in which this happens. But
your investigations shed enough light on it. You write:
  "If either the gettext.m4
   had gotten downgraded (to the version currently in Debian, which would
   not have pulled the tampered build-to-host.m4), or once Debian upgrades
   gettext, the build-to-host.m4 would get downgraded to the upstream
   clean version"

The situation is:
  - Debian stable is still on gettext 0.21, which comes with
  gettext.m4 serial 71
  - gettext-0.22 comes with
  gettext.m4 serial 77
which depends on build-to-host.m4.
Assume furthermore that a package P relies on gettext (via AM_GNU_GETTEXT),
and the user has installed gettext-0.22 or newer. And they run
  $ autoreconf
which invokes autopoint, which installs the gettext.m4.

If, in this situation, the newer .m4 file would not prevail, Debian's
old gettext.m4 would take precedence. This would mean that new features
from new gettext releases would not land in the users' hands; they
would be blocked by Debian, for the time the user uses this distro+version
(often 4 or 5 years).

Worse, this would also hold for Red Hat distro releases, which typically
are not 4-5 years, but 10-12 years behind the newest release. It would
mean that *for 10 years*, packages P could not make use of new features
(and bug fixes) that were added in GNU gettext upstream.

This is the problem that the 'serial' number annotation fixes.

It may be unexpected to you, but it is very much intended.

And if aclocal's preference was the other way around, always favouring
the smaller serial number, then the attacker would not have picked
'serial 30' but 'serial 1', in order to override the distro's version.

> My understanding is that the use of serial numbers in .m4 snippets was
> intentional in gnulib (more or less where the practice originated),
> but only because gnulib prefers a linear history (everything is
> monotonically increasing, no forks for the serial number to diverge
> on).

The history is mostly linear, yes. There is a bit of nonlinearity
  - because gettext.m4 gets copied from gettext to gnulib and/or
from gnulib to gettext,
  - because gettext has a release branch,
  - because gnulib has stable branches,
but all in all, it's linear.

> In light of this weekend's mess, Bruno may have more ideas about
> how to prevent his files from being turned into backdoor delivery
> mechanisms in the future.

I gave my opinion here:
  https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00422.html
  https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00421.html

Basically I think that an attacker who is clever enough to bypass the
RELRO protection by using IFUNC, knowing that IFUNC gets processed
_before_ the GOT or PLT pages are made read-only, — such an attacker
will also find ways to conceal their malware triggers, regardless of
any simple measures at the source code level that we might invent.

Bruno

Re: autoreconf --force seemingly does not forcibly update everything

[Guillem Jover]

Hi!

On Mon, 2024-04-01 at 12:43:02 -0500, Eric Blake wrote:
> Widening the audience to include bug-gnulib, which is the upstream
> source of "# build-to-host.m4 serial 3" which was bypassed by the
> malicious "# build-to-host.m4 serial 30".
> 
> On Sun, Mar 31, 2024 at 11:51:36PM +0200, Guillem Jover wrote:
> > While analyzing the recent xz backdoor hook into the build system [A],
> > I noticed that one of the aspects why the hook worked was because it
> > seems like «autoreconf -f -i» (that is run in Debian as part of
> > dh-autoreconf via dh) still seems to take the serial into account,
> > which was bumped in the tampered .m4 file. If either the gettext.m4
> > had gotten downgraded (to the version currently in Debian, which would
> > not have pulled the tampered build-to-host.m4), or once Debian upgrades
> > gettext, the build-to-host.m4 would get downgraded to the upstream
> > clean version, then the hook would have been disabled and the backdoor
> > would be inert. (Of course at that point the malicious actor would
> > have found another way to hook into the build system, but the less
> > avenues there are the better.)
> > 
> > I've tried to search the list and checked for old bug reports on the
> > debbugs.gnu.org site, but didn't notice anything. To me this looks like
> > a very unexpected behavior, but it's not clear whether this is intentional
> > or a bug. In any case regardless of either position, it would be good to
> > improve this (either by fixing --force to force things even if
> > downgrading, or otherwise perhaps to add a new option to really force
> > everything).
> > 
> > [A] 
> > Longish mail, search for "try to go in detail" for the analysis.
> 
> My understanding is that the use of serial numbers in .m4 snippets was
> intentional in gnulib (more or less where the practice originated),
> but only because gnulib prefers a linear history (everything is
> monotonically increasing, no forks for the serial number to diverge
> on).  In light of this weekend's mess, Bruno may have more ideas about
> how to prevent his files from being turned into backdoor delivery
> mechanisms in the future.

I think the serial numbers and the non-downgrade behavior makes sense
in general in non-force mode, but I still expect and I find it very
surprising that --force does not overwrite everything, when that seems
like what's been asked.

Of course the problem then becomes, how much might might now break if
--force would truly force a refresh from system files, because I'm
assuming this has been the behavior all along?

At least in Debian I guess we could perform a mass rebuild with a
modified autoreconf that truly forces to see how much breaks. But even
if much breaks, I think that fallout is something we might want to fix
anyway, but perhaps in a controlled/staged way.

Thanks,
Guillem

Re: autoreconf --force seemingly does not forcibly update everything

[Eric Blake]

Widening the audience to include bug-gnulib, which is the upstream
source of "# build-to-host.m4 serial 3" which was bypassed by the
malicious "# build-to-host.m4 serial 30".

On Sun, Mar 31, 2024 at 11:51:36PM +0200, Guillem Jover wrote:
> Hi!
> 
> While analyzing the recent xz backdoor hook into the build system [A],
> I noticed that one of the aspects why the hook worked was because it
> seems like «autoreconf -f -i» (that is run in Debian as part of
> dh-autoreconf via dh) still seems to take the serial into account,
> which was bumped in the tampered .m4 file. If either the gettext.m4
> had gotten downgraded (to the version currently in Debian, which would
> not have pulled the tampered build-to-host.m4), or once Debian upgrades
> gettext, the build-to-host.m4 would get downgraded to the upstream
> clean version, then the hook would have been disabled and the backdoor
> would be inert. (Of course at that point the malicious actor would
> have found another way to hook into the build system, but the less
> avenues there are the better.)
> 
> I've tried to search the list and checked for old bug reports on the
> debbugs.gnu.org site, but didn't notice anything. To me this looks like
> a very unexpected behavior, but it's not clear whether this is intentional
> or a bug. In any case regardless of either position, it would be good to
> improve this (either by fixing --force to force things even if
> downgrading, or otherwise perhaps to add a new option to really force
> everything).
> 
> [A] 
> Longish mail, search for "try to go in detail" for the analysis.

My understanding is that the use of serial numbers in .m4 snippets was
intentional in gnulib (more or less where the practice originated),
but only because gnulib prefers a linear history (everything is
monotonically increasing, no forks for the serial number to diverge
on).  In light of this weekend's mess, Bruno may have more ideas about
how to prevent his files from being turned into backdoor delivery
mechanisms in the future.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

[sr #111044] autoconf should assert existence of all subsidiary tools at startup

[Zack Weinberg]

Update of sr #111044 (group autoconf):

Priority:   5 - Unprioritized => 2 - Eventually 
Severity:  3 - Normal => 2 - Minor  
  Status:None => Need Info  

___

Follow-up Comment #2:

Thank you for (re-)reporting this bug.  I agree that autoconf should make
reasonable efforts to run in a minimal environment and, when the environment
is too minimal to tolerate, it should give clear error messages.  In
particular I agree that neither `diff` nor `awk` (and arguably not even `sed`)
should be an implicit dependency.

That said, this kind of bug is very difficult to work on without concrete test
cases and test environments.  If you could provide us with both of

* a minimized configure.ac that produces a configure script that malfunctions
(meaning, neither runs to completion correctly nor produces a useful error
message and halts) in the minimal environment you're working with

* details of the minimal environment you're working with, in particular the
complete list of available shell commands ("command names" as defined in
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01
-- most importantly, both shell built-ins and external executables) and any
cases where functionality guaranteed by POSIX.1-2001 is absent; f there is a
straightforward recipe for constructing a matching environment from widely
available free software, please describe it

that would be super helpful and would probably get the bug addressed orders of
magnitude faster.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #111044] autoconf should assert existence of all subsidiary tools at startup

[anonymous]

Follow-up Comment #1, sr #111044 (group autoconf):

To elaborate on that, people may use the output of such tools to enable or
disable certain compiler features. An example of such a case was found when
building OpenVPN [0].

This was also reported in 2008 [1].

[0] https://twitter.com/disconnect3d_pl/status/1774747022362325263 (probably
change twitter to nitter to get access without logging in)

[1] https://lists.gnu.org/archive/html/bug-autoconf/2008-03/msg00033.html


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #111044] autoconf should assert existence of all subsidiary tools at startup

[anonymous]

URL:
  

 Summary: autoconf should assert existence of all subsidiary
tools at startup
   Group: Autoconf
   Submitter: None
   Submitted: Mon 01 Apr 2024 03:10:23 PM UTC
Priority: 5 - Unprioritized
Severity: 3 - Normal
  Status: None
 Privacy: Public
 Assigned to: None
Originator Email: f...@redhat.com
 Open/Closed: Open
 Discussion Lock: Any
Operating System: None


___

Follow-up Comments:


---
Date: Mon 01 Apr 2024 03:10:23 PM UTC By: Anonymous
The script generated by autoconf may run in environments even sparser than the
gnu standards.  One recentish example is missing "diff", which results in
confusing diagnostics but not an outright failure.

It'd be helpful - and possibly improve security overall - if autoconf's
generated shell script were to enumerate and assert the existence of every
/usr/bin type basic utility its code relies on, and if absent, abort abort
abort.  Better that than partial or obscured failures leading to oddly
configured target programs.







___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/