Re: Sourceware mitigating and preventing the next xz-backdoor

[Trenton Davison via Gcc]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar via Gcc]

Hi Mark,

On Sun, Apr 21, 2024 at 10:40:14PM +0200, Alejandro Colomar wrote:

[...]

> Let's generate a v2 patch set, showing the range-diff against v1.  We
> need to check the commit IDs of the first set, which can be found in the
> mailing list archives, thanks to the trick we used.  The v1 range was
> 7ec952012^..892a12470.  So we just pass that range:
> 
>   $ git format-patch -o ./patches/ master..HEAD \
>   --range-diff=7ec952012^..892a12470 -v2 --cover-letter;
>   ./patches/v2--cover-letter.patch
>   
> ./patches/v2-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
>   ./patches/v2-0002-share-mk-build-pdf-book-Use-Unifont.patch
>   
> ./patches/v2-0003-share-mk-build-fonts-unifont-Specify-space-width-.patch
> 
> The v2 cover letter shows the changes introduced since v1:
> 
>   $ tail -n20 ./patches/v2--cover-letter.patch 
>create mode 100644 share/mk/build/fonts/unifont/dit.mk
>create mode 100644 share/mk/build/fonts/unifont/pfa.mk
>create mode 100644 
> share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk
> 
>   Range-diff against v1:
>   1:  7ec952012 = 1:  7ec952012 share/mk/: build-fonts-unifont: Build 
> UnifontR from unifont.otf
>   2:  d80376b08 = 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
>   3:  892a12470 ! 3:  bc7fa7d92 share/mk/: build-fonts-unifont: Specify 
> spacewidth in afmtodit(1)
>   @@ Metadata
>Author: Alejandro Colomar 
>
> ## Commit message ##
>   -share/mk/: build-fonts-unifont: Specify spacewidth in 
> afmtodit(1)
>   +share/mk/: build-fonts-unifont: Specify space width in 
> afmtodit(1)
>
>Link: 
> 
>Suggested-by: "G. Branden Robinson" 
>   -- 
>   2.43.0

I've added a recommendation in the Linux man-pages contributing
documentation that patches be sent with a range diff, and also that
patches be sent in PGP-signed mail (if the user has a PGP key).  It has
specific instructions like the above (but simplified).



Feel free to copy any of that documentation.

I also recommended specific mutt(1) settings:

set crypt_autosign = yes
set crypt_protected_headers_write = yes

And git-send-email(1) configuration for using with neomutt(1):

   [sendemail]
   sendmailcmd = neomutt -C -H - && true

For all the documentation for mail and patches, see these two files:



Have a lovely day!
Alex

-- 



signature.asc
Description: PGP signature

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andreas Schwab]

On Apr 24 2024, Boris Kolpackov wrote:

> https://github.com/build2/libbuild2-autoconf

Welcaome back, imake.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sourceware mitigating and preventing the next xz-backdoor

[Boris Kolpackov]

Martin Uecker  writes:

> Do we really still need complex build systems such as autoconf? Are
> there still so many different configurations with subtle differences
> that every single feature needs to be tested individually by running
> code at build time?

We have taken the alternative approach in build2. Specifically, instead
of dynamic compilation/linking tests (which can fail for all kinds of
reasons besides the absent feature), we use static expected values
based on the platform/compiler macro checks. For example, if we are
compiling with glibc and the version is 2.38 or later, then we know
the strl*() function family is available:

https://github.com/build2/libbuild2-autoconf

We currently have ~200 checks and have built quite a bit of software
using this approach (including Qt classic libraries and all their
dependencies) on the mainstream platforms (Linux, Windows, Mac OS,
FreeBSD).

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar via Gcc]

On Sun, Apr 21, 2024 at 10:40:14PM +0200, Alejandro Colomar wrote:
> On Sun, Apr 21, 2024 at 05:30:52PM +0200, Mark Wielaard wrote:
> > Hi Alejandro,
> 
> Hi Mark,
> 
> I've reordered your message, to organize my response.
> 
> > On Wed, Apr 10, 2024 at 06:30:42PM +0200, Alejandro Colomar wrote:
> > > It would also be interesting to require showing range-diffs between
> > > patch revisions.  They make it much more difficult to introduce a
> > > vulnerability after a reviewer has turned its mins into approving the
> > > patch.  Of course, the patch could go in if the submitter lies in the
> > > range-diff and the vuln is undetected, but then it can be verified a
> > > posteriory to prove that there was a lie.
> > 
> > Could you give an example of using git range-diff?
> 
> Sure!
> 
> > Normally when asked for changes to a
> > patch (series) I do an git rebase -i (on the local branch I used to
> > develop the feature/bug fix) and split/commit all requested changes
> > and then sent the new patches with git send-email again.
> 
> I do the same thing.
> 
> > But I guess
> > to use/combine that with git range-diffs I should start creating new
> > local branches for each patch (series) in development?
> 
> No; it's compatible with that workflow.
> 
> > How do you go from
> > v1 of a patch (series) to a v2?
> 
> I'll start first with how you go from nothing to v1, which will help
> explain how you go from v1 to v2.
> 
> Let's say I have a branch 'unifont' for adding the Unifont font to the
> Linux man-pages' PDF book.  I have it in a branch that applies on top of
> 'master'.
> 
>   $ git log --oneline --graph master..unifont;
>   * 892a12470 (HEAD -> unifont, alx/contrib, contrib) share/mk/: 
> build-fonts-unifont: Specify spacewidth in afmtodit(1)
>   * d80376b08 share/mk/: build-pdf-book: Use Unifont
>   * 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from 
> unifont.otf
> 
> I want to send that as a patch set (v1, of course, since it's the first
> send).  I would already use range-diff when generating the patches:
> 
>   $ git format-patch -o ./patches master..HEAD \
>   --range-diff=master -v1 --cover-letter;
>   ./patches/v1--cover-letter.patch
>   
> ./patches/v1-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
>   ./patches/v1-0002-share-mk-build-pdf-book-Use-Unifont.patch
>   
> ./patches/v1-0003-share-mk-build-fonts-unifont-Specify-spacewidth-i.patch
> 
> Why would you do that in v1?  It notes the commit IDs of the patches, so
> you can later check them when preparing v2; we'll come back to that.
> For now, see what this adds to the patch set:
> 
>   $ tail ./patches/v1--cover-letter.patch ;
>create mode 100644 share/mk/build/fonts/unifont/pfa.mk
>create mode 100644 
> share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk
> 
>   Range-diff against v0:
>   -:  - > 1:  7ec952012 share/mk/: build-fonts-unifont: Build 
> UnifontR from unifont.otf
>   -:  - > 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
>   -:  - > 3:  892a12470 share/mk/: build-fonts-unifont: Specify 
> spacewidth in afmtodit(1)
>   -- 
>   2.43.0
> 
> You can see the IDs of the commits that form this patch set, which match
> the git-log(1) that I showed above.  Now someone suggests a change.  For
> this example, I wasn't happy with a commit message, and added a space to
> the third commit message subject line.  The git-log(1) is now:
> 
>   $ git log --oneline --graph master..unifont
>   * bc7fa7d92 (HEAD -> unifont) share/mk/: build-fonts-unifont: Specify 
> space width in afmtodit(1)
>   * d80376b08 share/mk/: build-pdf-book: Use Unifont
>   * 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from 
> unifont.otf
> 
> Let's generate a v2 patch set, showing the range-diff against v1.  We
> need to check the commit IDs of the first set, which can be found in the
> mailing list archives, thanks to the trick we used.  The v1 range was
> 7ec952012^..892a12470.  So we just pass that range:
> 
>   $ git format-patch -o ./patches/ master..HEAD \
>   --range-diff=7ec952012^..892a12470 -v2 --cover-letter;
>   ./patches/v2--cover-letter.patch
>   
> ./patches/v2-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
>   ./patches/v2-0002-share-mk-build-pdf-book-Use-Unifont.patch
>   
> ./patches/v2-0003-share-mk-build-fonts-unifont-Specify-space-width-.patch
> 
> The v2 cover letter shows the changes introduced since v1:
> 
>   $ tail -n20 ./patches/v2--cover-letter.patch 
>create mode 100644 share/mk/build/fonts/unifont/dit.mk
>create mode 100644 share/mk/build/fonts/unifont/pfa.mk
>create mode 100644 
> share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk
> 
>   Range-diff against v1:
>   1:  7ec952012 = 1:  7ec952012 share/mk/: build-fonts-unifont: Build 
> UnifontR from unif

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar via Gcc]

On Sun, Apr 21, 2024 at 05:30:52PM +0200, Mark Wielaard wrote:
> Hi Alejandro,

Hi Mark,

I've reordered your message, to organize my response.

> On Wed, Apr 10, 2024 at 06:30:42PM +0200, Alejandro Colomar wrote:
> > It would also be interesting to require showing range-diffs between
> > patch revisions.  They make it much more difficult to introduce a
> > vulnerability after a reviewer has turned its mins into approving the
> > patch.  Of course, the patch could go in if the submitter lies in the
> > range-diff and the vuln is undetected, but then it can be verified a
> > posteriory to prove that there was a lie.
> 
> Could you give an example of using git range-diff?

Sure!

> Normally when asked for changes to a
> patch (series) I do an git rebase -i (on the local branch I used to
> develop the feature/bug fix) and split/commit all requested changes
> and then sent the new patches with git send-email again.

I do the same thing.

> But I guess
> to use/combine that with git range-diffs I should start creating new
> local branches for each patch (series) in development?

No; it's compatible with that workflow.

> How do you go from
> v1 of a patch (series) to a v2?

I'll start first with how you go from nothing to v1, which will help
explain how you go from v1 to v2.

Let's say I have a branch 'unifont' for adding the Unifont font to the
Linux man-pages' PDF book.  I have it in a branch that applies on top of
'master'.

$ git log --oneline --graph master..unifont;
* 892a12470 (HEAD -> unifont, alx/contrib, contrib) share/mk/: 
build-fonts-unifont: Specify spacewidth in afmtodit(1)
* d80376b08 share/mk/: build-pdf-book: Use Unifont
* 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from 
unifont.otf

I want to send that as a patch set (v1, of course, since it's the first
send).  I would already use range-diff when generating the patches:

$ git format-patch -o ./patches master..HEAD \
--range-diff=master -v1 --cover-letter;
./patches/v1--cover-letter.patch

./patches/v1-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
./patches/v1-0002-share-mk-build-pdf-book-Use-Unifont.patch

./patches/v1-0003-share-mk-build-fonts-unifont-Specify-spacewidth-i.patch

Why would you do that in v1?  It notes the commit IDs of the patches, so
you can later check them when preparing v2; we'll come back to that.
For now, see what this adds to the patch set:

$ tail ./patches/v1--cover-letter.patch ;
 create mode 100644 share/mk/build/fonts/unifont/pfa.mk
 create mode 100644 
share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk

Range-diff against v0:
-:  - > 1:  7ec952012 share/mk/: build-fonts-unifont: Build 
UnifontR from unifont.otf
-:  - > 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
-:  - > 3:  892a12470 share/mk/: build-fonts-unifont: Specify 
spacewidth in afmtodit(1)
-- 
2.43.0

You can see the IDs of the commits that form this patch set, which match
the git-log(1) that I showed above.  Now someone suggests a change.  For
this example, I wasn't happy with a commit message, and added a space to
the third commit message subject line.  The git-log(1) is now:

$ git log --oneline --graph master..unifont
* bc7fa7d92 (HEAD -> unifont) share/mk/: build-fonts-unifont: Specify 
space width in afmtodit(1)
* d80376b08 share/mk/: build-pdf-book: Use Unifont
* 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from 
unifont.otf

Let's generate a v2 patch set, showing the range-diff against v1.  We
need to check the commit IDs of the first set, which can be found in the
mailing list archives, thanks to the trick we used.  The v1 range was
7ec952012^..892a12470.  So we just pass that range:

$ git format-patch -o ./patches/ master..HEAD \
--range-diff=7ec952012^..892a12470 -v2 --cover-letter;
./patches/v2--cover-letter.patch

./patches/v2-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
./patches/v2-0002-share-mk-build-pdf-book-Use-Unifont.patch

./patches/v2-0003-share-mk-build-fonts-unifont-Specify-space-width-.patch

The v2 cover letter shows the changes introduced since v1:

$ tail -n20 ./patches/v2--cover-letter.patch 
 create mode 100644 share/mk/build/fonts/unifont/dit.mk
 create mode 100644 share/mk/build/fonts/unifont/pfa.mk
 create mode 100644 
share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk

Range-diff against v1:
1:  7ec952012 = 1:  7ec952012 share/mk/: build-fonts-unifont: Build 
UnifontR from unifont.otf
2:  d80376b08 = 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
3:  892a12470 ! 3:  bc7fa7d92 share/mk/: build-fonts-unifont: Specify 
spacewidth in afmtodit(1)
@@ Metadata
 Au

Re: Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

Hi Alejandro,

On Wed, Apr 10, 2024 at 06:30:42PM +0200, Alejandro Colomar wrote:
> It would also be interesting to require showing range-diffs between
> patch revisions.  They make it much more difficult to introduce a
> vulnerability after a reviewer has turned its mins into approving the
> patch.  Of course, the patch could go in if the submitter lies in the
> range-diff and the vuln is undetected, but then it can be verified a
> posteriory to prove that there was a lie.

Could you give an example of using git range-diff? How do you go from
v1 of a patch (series) to a v2? Normally when asked for changes to a
patch (series) I do an git rebase -i (on the local branch I used to
develop the feature/bug fix) and split/commit all requested changes
and then sent the new patches with git send-email again. But I guess
to use/combine that with git range-diffs I should start creating new
local branches for each patch (series) in development?

Thanks,

Mark

Re: Sourceware mitigating and preventing the next xz-backdoor

[James K. Lowden]

On Mon, 1 Apr 2024 17:06:17 +0200
Mark Wielaard  wrote:

> We should discuss what we have been doing and should do more to
> mitigate and prevent the next xz-backdoor. 

Since we're working on a compiler, "On Trusting Trust" comes to mind.
Russ Cox posted some thoughts last year that might be applicable.  

https://research.swtch.com/nih

On a different tack, ISTM it might also be possible to use quantitative
methods.  AIUI the xz attack was discovered while investigating
exorbitant power consumption.  Could the compiler's power consumption
be measured over some baseline, perhaps on a line-by-line basis?  If
so, each new commit could be power-measured, and deemed acceptable if
it's within some threshold, perhaps 10%.  That's a guess; over time
we'd learn how much variation to expect.  

As a public organization, any would-be attacker would obviously know
what we're doing, and would know to keep his attack under the
threshhold. That makes his job harder, which would have the effect of
encouraging him to look elsewhere. 

--jkl

Re: Sourceware mitigating and preventing the next xz-backdoor

[Frank Ch. Eigler via Gcc]

Hi -

>This is very true, however a few words of caution: IME this is a
>maintainability nightmare. Fixing patches that forgot to regenerate,
>regenerating on rebase, confirming everything is up-to-date before
>merge, etc etc. It can be handled, I have, but it was painful and
>time-consuming.The hardest part was ensuring everyone was actually
>running the "right" version of Auto* [...]

One way to make the nightmare into a light hassle is to let developers
commit auto* hand-written inputs with or without Complete Properly
refreshed generated bits, and let a maintainer or bot (but I repeat
myself) periodically regenerate the derived auto* content.

- FChE

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar via Gcc]

Hi Joel,

On Wed, Apr 03, 2024 at 08:53:21AM -0500, Joel Sherrill wrote:
> On Wed, Apr 3, 2024, 3:09 AM Florian Weimer via Gdb 
> wrote:
> 
> > * Guinevere Larsen via Overseers:
> >
> > > Beyond that, we (GDB) are already experimenting with approved-by, and
> > > I think glibc was doing the same.
> >
> > The glibc project uses Reviewed-by:, but it's completely unrelated to
> > this.  Everyone still pushes their own patches, and there are no
> > technical countermeasures in place to ensure that the pushed version is
> > the reviewed version.
> >
> 
> Or that there isn't "collusion" between a malicious author and reviewer.
> Just tagging it approved or reviewed by just gives you two people to blame.
> It is not a perfect solution either.

If those tags are given in a mailing list _and_ mails to the mailing
list are PGP-signed, then you can verify that the tags were valid, and
not just invented.

And with signed commits you have a guarantee that one doesn't overwrite
history attributing commits to other committers (it could happen with
authors, and indeed it seems to have happened in this case, but if
patches are sent via signed mail, then it's also verifyiable).

In the email side, there are a few things to improve:

For sending signed emails, there's patatt(5) (used by b4(5)), but it
might not be comfortable to use for everyone.  For those preferring
normal MUAs, neomutt(1) is an alternative:



And I have a few patches for improving the security of protected
messages:




And the corresponding security-vulnerability reports:







(I find it funny that I didn't know about this xz issue until yesterday,
 so not when I reported those issues, but they are interestingly
 related.)

It would also be interesting to require showing range-diffs between
patch revisions.  They make it much more difficult to introduce a
vulnerability after a reviewer has turned its mins into approving the
patch.  Of course, the patch could go in if the submitter lies in the
range-diff and the vuln is undetected, but then it can be verified a
posteriory to prove that there was a lie.

I recently started applying all of these (signing all email, including
patches, sign all commits and tags, and provide range-diffs), and it's
not too uncomfrotable; I'd say it's even more comfortable than not doing
it, as it allows me to easily roll back a patch to an older revision if
I find I introduced a mistake, and find where I introduced it.


Have a lovely day!
Alex

-- 



signature.asc
Description: PGP signature

Re: Sourceware mitigating and preventing the next xz-backdoor

[Frank Ch. Eigler via Gcc]

Hi -

> In Autotools, `make dist` produces a tarball that contains many
> files not present in the source respoitory, it includes build system
> core files and this fact was used for the xz attack. In contrast,
> for newer build systems the "release tarball" is purely a snapshot
> of the source repository: there is no `cmake dist`, and `meson dist`
> is essentially `git archive` [...]

For what it's worth, not every auto* using project uses "make dist" to
build their release tarballs.  If they can get over the matter of
including auto*-generated scripts being located in the source repo,
then indeed a "git archive" is sufficient.  Several of the projects I
work on do just this.  (As a bonus, that makes the git repos immediately
buildable by developers, without need to re-auto* anything.)

- FChE

Re: Sourceware mitigating and preventing the next xz-backdoor

[Claudio Bantaloukas via Gcc]

On 09/04/2024 18:57, Andreas Schwab wrote:
> On Apr 09 2024, anderson.jonath...@gmail.com wrote:
> 
>> - This xz backdoor injection unpacked attacker-controlled files and ran them 
>> during `configure`. Newer build systems implement a build abstraction (aka 
>> DSL) that acts similar to a sandbox and enforces rules (e.g. the only code 
>> run during `meson setup` is from `meson.build` files and CMake). Generally 
>> speaking the only way to disobey those rules is via an "escape" command 
>> (e.g. `run_command()`) of which there are few. This reduces the task of 
>> auditing the build scripts for sandbox-breaking malicious intent 
>> significantly, only the "escapes" need investigation and they which 
>> should(tm) be rare for well-behaved projects.
> 
> Just like you can put your backdoor in *.m4 files, you can put them in
> *.cmake files.
>

Hi Andreas,
Indeed you're right and seeing the hijacked CMakeLists.txt in the commit 
was eye opening.

There is a not so subtle difference though. The amount of nasty that the 
attacker thought could get away with was different between the 
build-to-host.m4 and the CMakeLists.txt attack vectors.

For the CMakeLists.txt file, the wanted change was very small, adding a 
dot to a piece of c code so that the test that runs it goes into one of 
the perfectly acceptable states (cannot compile the c file), thus 
disabling a security feature.
This change was "hidden" in a patch containing a bunch of pointless 
renames and veiled in plausible deniability (oops, that dot went in the 
by mistake, silly me, here's a patch to fix the file). The attacker was 
lucky because noone really checked.

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7;hp=eb8ad59e9bab32a8d655796afd39597ea6dcc64d

Compare that to what the m4 file did. Russ Cox has an interesting 
analysis https://research.swtch.com/xz-script

 From which I'll pick a choice quote: "makes a handful of inscrutable 
changes that don’t look terribly out of place".

I figured out the problem with the CMakeFile.txt quickly. I'm not 100% 
sure if the configure.ac is ok (after looking at it for 10 minutes, it 
looks ok, but I'm not sure!). I would not be able to recognise good code 
from bad in the m4 file.

Admittedly, I'm biased in favour of cmake's DSL, have more experience 
with it despite using ./configure since the mid 90s and have a 
preference. But it would be hard to argue against the fact that benign 
m4, as practiced in the wild today is hard to separate from malicious m4 
by a majority of developers, including experienced ones like Mr. Cox above.

Cheers,
Claudio Bantaloukas

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/9/24 15:22, Sam James wrote:

Paul Eggert  writes:


On 4/9/24 14:58, Sam James wrote:

Meson doesn't allow user-defined functions

Meson has ways to execute arbitrary user-defined code, so it's not
immune to this sort of exploit.

To be clear - not saying it's immune.


Sure, but someone who's not expert in Meson could easily misread "Meson 
doesn't allow user-defined functions" and think that this means Meson is 
immune to an xz-style attack, which it's not.



Just that it scopes the
user-defined code part to clearly defined sections.


As does Autoconf. To a determined attacker I daresay there's not much 
difference.



I think it makes sense to optimise for ease of review.


Ease of review definitely a good thing, all other things being equal.


It's just easy to go too far the other
way too and not change anything


I'm certainly not advocating that! All I'm saying is that we should use 
our limited development resources wisely.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James via Gcc]

Paul Eggert  writes:

> On 4/9/24 14:58, Sam James wrote:
>> Meson doesn't allow user-defined functions
>
> Meson has ways to execute arbitrary user-defined code, so it's not
> immune to this sort of exploit.

To be clear - not saying it's immune. Just that it scopes the
user-defined code part to clearly defined sections.

I think it makes sense to optimise for ease of review.

>
> It's of course better (all other things being equal) to use a build
> system with a smaller attack surface. However, any surface of nonzero
> size is attackable, so I'm not convinced that Meson is significantly
> safer against a determined insider. Although the xz exploit was tricky
> and is now famous (hey! the front page of the New York Times!)
> fundamentally it was sloppy and amateurish and it succeeded only
> because xz's project management was even sloppier.
>
> Yes, we need to defend against amateurish attacks. But we shouldn't
> waste valuable developer time on defenses that won't work against
> obvious future attacks and that will likely cost more than they'll
> benefit. That's just security theater.

Right, I'm not advocating that. It's just easy to go too far the other
way too and not change anything because it won't hold up against a state
actor.


signature.asc
Description: PGP signature

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/9/24 14:58, Sam James wrote:

Meson doesn't allow user-defined functions


Meson has ways to execute arbitrary user-defined code, so it's not 
immune to this sort of exploit.


It's of course better (all other things being equal) to use a build 
system with a smaller attack surface. However, any surface of nonzero 
size is attackable, so I'm not convinced that Meson is significantly 
safer against a determined insider. Although the xz exploit was tricky 
and is now famous (hey! the front page of the New York Times!) 
fundamentally it was sloppy and amateurish and it succeeded only because 
xz's project management was even sloppier.


Yes, we need to defend against amateurish attacks. But we shouldn't 
waste valuable developer time on defenses that won't work against 
obvious future attacks and that will likely cost more than they'll 
benefit. That's just security theater.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James via Gcc]

Jonathon Anderson  writes:

> On Tue, 2024-04-09 at 14:50 -0700, Paul Eggert wrote:
>
>  On 4/9/24 14:40, Jeffrey Walton wrote:
>
>  Code provenance and code integrity was not enforced. Part of the problem is 
> the Autotools design. It is from a
>  bygone era.
>
>  No, Andreas is right. This isn't an Autotools-vs-Meson thing.
>
>  Most of the Autotools-based projects I help maintain would have been
>  immune to this particular exploit, partly because they don't maintain
>  their own of Gnulib .m4 files. Conversely, any Meson-based project that
>  had the same sort of out-of-repository sloppiness and lack of review
>  that xz had, would be vulnerable to similar attacks.
>
> Xz doesn't either, the exploit was unique to the distributed make dist 
> tarballs. Which is an Autotools quirk present in
> all Autotools projects.
>
> I won't deny that a project could use Meson and be sloppy, a project could 
> use SSL/TLS/whatever and be completely
> insecure. But Autotools encourages and semi-requires this sloppy behavior, 
> and CMake and Meson strongly discourage this
> behavior.

Indeed. Talking about things in the context of "how can we make it
easier to spot" is a good thing. Obviously if we're trying to resist a
state actor, things are very hard. It doesn't mean don't bother.

>
> -Jonathon

thanks,
sam


signature.asc
Description: PGP signature

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson via Gcc]

On Tue, 2024-04-09 at 14:50 -0700, Paul Eggert wrote:
> On 4/9/24 14:40, Jeffrey Walton wrote:
> 
> > Code provenance and code integrity was not enforced. Part of the
> > problem is the Autotools design. It is from a bygone era.
> 
> 
> No, Andreas is right. This isn't an Autotools-vs-Meson thing.
> 
> Most of the Autotools-based projects I help maintain would have been  
> immune to this particular exploit, partly because they don't maintain  
> their own of Gnulib .m4 files. Conversely, any Meson-based project that  
> had the same sort of out-of-repository sloppiness and lack of review  
> that xz had, would be vulnerable to similar attacks.

Xz doesn't either, the exploit was unique to the distributed `make dist` 
tarballs. Which is an Autotools quirk present in all Autotools projects.

I won't deny that a project could use Meson and be sloppy, a project could use 
SSL/TLS/whatever and be completely insecure. But Autotools encourages and 
semi-requires this sloppy behavior, and CMake and Meson strongly discourage 
this behavior.

-Jonathon

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James via Gcc]

Jonathon Anderson via Gdb  writes:

> On Tue, 2024-04-09 at 16:11 -0400, Paul Koning wrote:
>>
>> On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc 
>> <[gcc@gcc.gnu.org](mailto:gcc@gcc.gnu.org)> wrote:
>>
>> > CMake has its own sandbox and rules and escapes (granted, much more of
>> > them). But regardless, the injection code would be committed to the
>> > repository (point 2) and would not hold up to a source directory mounted
>> > read-only (point 3).
>> 
>> Why would the injection code necessarily be committed to the
>> repository?  It wasn't in the xz attack -- one hole in the
>> procedures is that the kits didn't match the repository and no
>> checks caught this.  I don't see how a different build system would
>> cure that issue.  Instead, there needs to be some sort of audit that
>> verifies there aren't rogue or modified elements in the kit.
>
> In Autotools, `make dist` produces a tarball that contains many files
> not present in the source respoitory, it includes build system core
> files and this fact was used for the xz attack. In contrast, for newer
> build systems the "release tarball" is purely a snapshot of the source
> repository: there is no `cmake dist`, and `meson dist` is essentially
> `git archive`
> ([docs](https://mesonbuild.com/Creating-releases.html)). Thus for the
> injection code to be present in the release tarball, it needs to have
> first been checked into the repository.

(Of course, one could modify it after, but the point here is that it's by
design reproducible so any differences are suspicious, just to be
clear.)

>
> In fact, packagers don't *need* to use the tarballs, they can (and
> should) use the Git history from the source repository itself. In
> Debian this is one workflow implemented by the popular
> git-buildpackage
> ([docs](https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.upstream-git.html)).
>  The
> third-party package manager
> [Spack](https://spack.readthedocs.io/en/latest/packaging_guide.html#git)
> clones directly from the source repository. Others may have support
> for this as well, this isn't a novel idea.
>
> -Jonathon


signature.asc
Description: PGP signature

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James via Gcc]

Paul Eggert  writes:

> On 4/9/24 14:40, Jeffrey Walton wrote:
>> Code provenance and code integrity was not enforced. Part of the
>> problem is the Autotools design. It is from a bygone era.
>
> No, Andreas is right. This isn't an Autotools-vs-Meson thing.
>
> Most of the Autotools-based projects I help maintain would have been
> immune to this particular exploit, partly because they don't maintain
> their own of Gnulib .m4 files. Conversely, any Meson-based project
> that had the same sort of out-of-repository sloppiness and lack of
> review that xz had, would be vulnerable to similar attacks.
>

While it could indeed happen via a variety of methods, it's still worth
talking about how to reduce places for it to hide in, and make review
easier, I think.

>> No one should be able to override a named, GNU supplied m4 macro.
>
> That ship sailed long ago, for Autoconf and for Meson and for every
> other widely-available build tool I know of. Everyone can write and
> run their own code, whether it comes from GNU or not. That's a feature
> that developers want and need. Although this feature can be misused,
> it's not a bug per se.

Meson doesn't allow user-defined functions and macros to avoid
metaprogramming hell:
https://mesonbuild.com/FAQ.html#why-doesnt-meson-have-user-defined-functionsmacros.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson via Gcc]

On Tue, 2024-04-09 at 16:11 -0400, Paul Koning wrote:
>
> On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc 
> <[gcc@gcc.gnu.org](mailto:gcc@gcc.gnu.org)> wrote:
>
> > CMake has its own sandbox and rules and escapes (granted, much more of
> > them). But regardless, the injection code would be committed to the
> > repository (point 2) and would not hold up to a source directory mounted
> > read-only (point 3).
> 
> Why would the injection code necessarily be committed to the repository?  It 
> wasn't in the xz attack -- one hole in the procedures is that the kits didn't 
> match the repository and no checks caught this.  I don't see how a different 
> build system would cure that issue.  Instead, there needs to be some sort of 
> audit that verifies there aren't rogue or modified elements in the kit.

In Autotools, `make dist` produces a tarball that contains many files not 
present in the source respoitory, it includes build system core files and this 
fact was used for the xz attack. In contrast, for newer build systems the 
"release tarball" is purely a snapshot of the source repository: there is no 
`cmake dist`, and `meson dist` is essentially `git archive` 
([docs](https://mesonbuild.com/Creating-releases.html)). Thus for the injection 
code to be present in the release tarball, it needs to have first been checked 
into the repository.

In fact, packagers don't *need* to use the tarballs, they can (and should) use 
the Git history from the source repository itself. In Debian this is one 
workflow implemented by the popular git-buildpackage 
([docs](https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.upstream-git.html)).
 The third-party package manager 
[Spack](https://spack.readthedocs.io/en/latest/packaging_guide.html#git) clones 
directly from the source repository. Others may have support for this as well, 
this isn't a novel idea.

-Jonathon

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/9/24 14:40, Jeffrey Walton wrote:

Code provenance and code integrity was not enforced. Part of the
problem is the Autotools design. It is from a bygone era.


No, Andreas is right. This isn't an Autotools-vs-Meson thing.

Most of the Autotools-based projects I help maintain would have been 
immune to this particular exploit, partly because they don't maintain 
their own of Gnulib .m4 files. Conversely, any Meson-based project that 
had the same sort of out-of-repository sloppiness and lack of review 
that xz had, would be vulnerable to similar attacks.



No one should be able to override a named, GNU supplied m4 macro.


That ship sailed long ago, for Autoconf and for Meson and for every 
other widely-available build tool I know of. Everyone can write and run 
their own code, whether it comes from GNU or not. That's a feature that 
developers want and need. Although this feature can be misused, it's not 
a bug per se.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton via Gcc]

On Tue, Apr 9, 2024 at 4:11 PM Paul Koning via Gdb  wrote:
>
> > On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc  
> > wrote:
> >
> > On Tue, Apr 9, 2024, 10:57 Andreas Schwab  wrote:
> >
> >> On Apr 09 2024, anderson.jonath...@gmail.com wrote:
> >>
> >>> - This xz backdoor injection unpacked attacker-controlled files and ran
> >> them during `configure`. Newer build systems implement a build abstraction
> >> (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only
> >> code run during `meson setup` is from `meson.build` files and CMake).
> >> Generally speaking the only way to disobey those rules is via an "escape"
> >> command (e.g. `run_command()`) of which there are few. This reduces the
> >> task of auditing the build scripts for sandbox-breaking malicious intent
> >> significantly, only the "escapes" need investigation and they which
> >> should(tm) be rare for well-behaved projects.
> >>
> >> Just like you can put your backdoor in *.m4 files, you can put them in
> >> *.cmake files.
> >
> >
> > CMake has its own sandbox and rules and escapes (granted, much more of
> > them). But regardless, the injection code would be committed to the
> > repository (point 2) and would not hold up to a source directory mounted
> > read-only (point 3).
>
> Why would the injection code necessarily be committed to the repository?  It 
> wasn't in the xz attack -- one hole in the procedures is that the kits didn't 
> match the repository and no checks caught this.

In the Xz case, the victims should not have been able to execute a
well known but modified *.m4 file. The *.m4 files should have been on
the build machine already; or, the latest *.m4 files installed on the
build machine via an updated Autoconf; or the latest serial-numbered
*.m4 macro should have been fetched from a trusted source like GNU.
They provide the provenance needed to allow their execution.

In the browser world, content security policy (CSP) allows third party
code to execute, assuming a script-src and an integrity tag. The
provenance and integrity is enforced.

> I don't see how a different build system would cure that issue.  Instead, 
> there needs to be some sort of audit that verifies there aren't rogue or 
> modified elements in the kit.

Code provenance and code integrity was not enforced. Part of the
problem is the Autotools design. It is from a bygone era. No one
should be able to override a named, GNU supplied m4 macro. That's a
bug, not a feature.

Jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning via Gcc]

> On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc  wrote:
> 
> On Tue, Apr 9, 2024, 10:57 Andreas Schwab  wrote:
> 
>> On Apr 09 2024, anderson.jonath...@gmail.com wrote:
>> 
>>> - This xz backdoor injection unpacked attacker-controlled files and ran
>> them during `configure`. Newer build systems implement a build abstraction
>> (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only
>> code run during `meson setup` is from `meson.build` files and CMake).
>> Generally speaking the only way to disobey those rules is via an "escape"
>> command (e.g. `run_command()`) of which there are few. This reduces the
>> task of auditing the build scripts for sandbox-breaking malicious intent
>> significantly, only the "escapes" need investigation and they which
>> should(tm) be rare for well-behaved projects.
>> 
>> Just like you can put your backdoor in *.m4 files, you can put them in
>> *.cmake files.
> 
> 
> CMake has its own sandbox and rules and escapes (granted, much more of
> them). But regardless, the injection code would be committed to the
> repository (point 2) and would not hold up to a source directory mounted
> read-only (point 3).

Why would the injection code necessarily be committed to the repository?  It 
wasn't in the xz attack -- one hole in the procedures is that the kits didn't 
match the repository and no checks caught this.  I don't see how a different 
build system would cure that issue.  Instead, there needs to be some sort of 
audit that verifies there aren't rogue or modified elements in the kit.

paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson via Gcc]

On Tue, Apr 9, 2024, 10:57 Andreas Schwab  wrote:

> On Apr 09 2024, anderson.jonath...@gmail.com wrote:
>
> > - This xz backdoor injection unpacked attacker-controlled files and ran
> them during `configure`. Newer build systems implement a build abstraction
> (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only
> code run during `meson setup` is from `meson.build` files and CMake).
> Generally speaking the only way to disobey those rules is via an "escape"
> command (e.g. `run_command()`) of which there are few. This reduces the
> task of auditing the build scripts for sandbox-breaking malicious intent
> significantly, only the "escapes" need investigation and they which
> should(tm) be rare for well-behaved projects.
>
> Just like you can put your backdoor in *.m4 files, you can put them in
> *.cmake files.


CMake has its own sandbox and rules and escapes (granted, much more of
them). But regardless, the injection code would be committed to the
repository (point 2) and would not hold up to a source directory mounted
read-only (point 3).

If your build system is Meson, you can easily consider CMake code to be an
escape and give it a little more auditing attention. Or just avoid shipping
CMake scripts entirely, they are are rarely necessary.

-Jonathon

>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andreas Schwab]

On Apr 09 2024, anderson.jonath...@gmail.com wrote:

> - This xz backdoor injection unpacked attacker-controlled files and ran them 
> during `configure`. Newer build systems implement a build abstraction (aka 
> DSL) that acts similar to a sandbox and enforces rules (e.g. the only code 
> run during `meson setup` is from `meson.build` files and CMake). Generally 
> speaking the only way to disobey those rules is via an "escape" command (e.g. 
> `run_command()`) of which there are few. This reduces the task of auditing 
> the build scripts for sandbox-breaking malicious intent significantly, only 
> the "escapes" need investigation and they which should(tm) be rare for 
> well-behaved projects.

Just like you can put your backdoor in *.m4 files, you can put them in
*.cmake files.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sourceware mitigating and preventing the next xz-backdoor

[anderson.jonathonm--- via Gcc]

Hello,

On Thu, Apr 4, 2024, 09:00 Michael Matz  wrote:

> Hello,
>
> On Wed, 3 Apr 2024, Jonathon Anderson wrote:
>
> > Of course, this doesn't make the build system any less complex, but 
> > projects using newer build systems seem easier to secure and audit than 
> > those using overly flexible build systems like Autotools and maybe even 
> > CMake. IMHO using a late-model build system is a relatively low 
> > technical hurdle to overcome for the benefits noted above, switching 
> > should be considered and in a positive light.
>
> Note that we're talking not (only) about the build system itself, i.e. how 
> to declare dependencies within the sources, and how to declare how to 
> build them. [...]
>
> But Martin also specifically asked about alternatives for feature tests, 
> i.e. autoconfs purpose.  I simply don't see how any alternative to it 
> could be majorly "easier" or "less complex" at its core.  

My point was not that newer build systems are any less complex taken 
wholistically. The upthread has already discussed why configuration tests are 
necessary and aren't going away anytime soon. My horror stories would not add 
much to that conversation.

My point, or rather my opinion as a humble end-user, is that newer build 
systems (e.g. Meson) make identifying the xz backdoor and injections following 
the same pattern much easier than older build systems (e.g. 
autoconf+automake+libtool). Paraphrasing my original message to highlight the 
backdoor protections:

- This xz backdoor injection unpacked attacker-controlled files and ran them 
during `configure`. Newer build systems implement a build abstraction (aka DSL) 
that acts similar to a sandbox and enforces rules (e.g. the only code run 
during `meson setup` is from `meson.build` files and CMake). Generally speaking 
the only way to disobey those rules is via an "escape" command (e.g. 
`run_command()`) of which there are few. This reduces the task of auditing the 
build scripts for sandbox-breaking malicious intent significantly, only the 
"escapes" need investigation and they which should(tm) be rare for well-behaved 
projects.

- This xz backdoor injected worked due to altered versions of the build system 
implementation being shipped with the source. Newer build systems do not ship 
any part of the build system's implementation with the project's source, it 
somes as a separate package and the most the project does is specify a 
compatible version range. This removes the possibility of "hidden" code being 
mixed to the shipped source tarball, well-behaved projects will have source 
tarballs that can be byte-for-byte reproduced from the VCS (e.g. Git).

- This xz backdoor injected the payload code by altering the files in the 
source directory. Newer build systems do not allow altering the source 
directory (for good reason) in their build abstraction. For build workers/CI, 
this restriction can be enforced using low-level sandboxing (e.g. conainers or 
landlock) and well-behaved projects should(tm) be unaffected. This reduces the 
possibilities on how code can appear in the project, only code generation 
sequences (e.g. `custom_target()` or `generator()`) can produce arbitrary code.

Just by transitioning to a newer build system (and being "well-behaved") it 
seems like you get certain protections almost for free. And IMHO compared to 
other options to improve project security (e.g. automated fuzzing), 
transitioning build systems is a medium-effort, low-maintainence option with a 
comparatively high yield.

> make is just fine for that (as are many others).  (In a way 
> I think we meanwhile wouldn't really need automake and autogen, but 
> rewriting all that in pure GNUmake is a major undertaking).

Makefile (the language) is extremely complex, it is nigh impossible to ensure 
something won't expand to some malicious code injection when the right file 
arrangement is present. (IMHO Make also is not a great build system, it has 
none of the protections described above... and it's [just plain 
slow](https://david.rothlis.net/ninja-benchmark/).)

Newer build systems seem to be moving in a direction where the bulk of the 
build configuration/scripts are non-Turing-complete. This is a highly effective 
method for clarifying the high-level complexity of the DSL: simple things are 
short and sweet, complex things (and discouraged solutions) are long and 
complex. Less complex high-level build logic makes it easier to verify and 
audit for security (and other) purposes.

> Going with the 
> examples given upthread there is usually only one major solution: to check 
> if a given system supports FOOBAR you need to bite the bullet and compile 
> (and potentially run!) a small program using FOOBAR.  A configuration 
> system that can do that (and I don't see any real alternative to that), no 
> matter in which language it's written and how traditional or modern it is, 
> also gives you enough rope to hang yourself, if you so choose.

Both very true, 

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker via Gcc]

Am Samstag, dem 06.04.2024 um 15:00 +0200 schrieb Richard Biener:
> On Fri, Apr 5, 2024 at 11:18 PM Andrew Sutton via Gcc  wrote:
> > 
> > > 
> > > 
> > > 
> > > > I think the key difference here is that Autotools allows arbitrarily
> > > generated code to be executed at any time. More modern build systems
> > > require the use of specific commands/files to run arbitrary code, e.g.
> > > CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson
> > > ([`run_command()`][1]), Cargo ([`build.rs`][4]).\
> > > 
> > > To me it seems that Cargo is the absolute worst case with respect to
> > > supply chain attacks.
> > > 
> > > It pulls in dependencies recursively from a relatively uncurated
> > > list of projects, puts the source of all those dependencies into a
> > > hidden directory in home, and runs Build.rs automatically with
> > > user permissions.
> > > 
> > 
> > 100% this. Wait until you learn how proc macros work.
> 
> proc macro execution should be heavily sandboxed, otherwise it seems
> compiling something is enough to get arbitrary code executed with the
> permission of the compiling user.  I mean it's not rocket science - browsers
> do this for javascript.  Hmm, we need a webassembly target ;)

This would be useful anyhow. 

And locking down the compiler using landlock to only access specified
files / directories would also be nice in general.

Martin

Re: Sourceware mitigating and preventing the next xz-backdoor

[Richard Biener via Gcc]

On Fri, Apr 5, 2024 at 11:18 PM Andrew Sutton via Gcc  wrote:
>
> >
> >
> >
> > > I think the key difference here is that Autotools allows arbitrarily
> > generated code to be executed at any time. More modern build systems
> > require the use of specific commands/files to run arbitrary code, e.g.
> > CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson
> > ([`run_command()`][1]), Cargo ([`build.rs`][4]).\
> >
> > To me it seems that Cargo is the absolute worst case with respect to
> > supply chain attacks.
> >
> > It pulls in dependencies recursively from a relatively uncurated
> > list of projects, puts the source of all those dependencies into a
> > hidden directory in home, and runs Build.rs automatically with
> > user permissions.
> >
>
> 100% this. Wait until you learn how proc macros work.

proc macro execution should be heavily sandboxed, otherwise it seems
compiling something is enough to get arbitrary code executed with the
permission of the compiling user.  I mean it's not rocket science - browsers
do this for javascript.  Hmm, we need a webassembly target ;)

Richard.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andrew Sutton via Gcc]

>
>
>
> > I think the key difference here is that Autotools allows arbitrarily
> generated code to be executed at any time. More modern build systems
> require the use of specific commands/files to run arbitrary code, e.g.
> CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson
> ([`run_command()`][1]), Cargo ([`build.rs`][4]).\
>
> To me it seems that Cargo is the absolute worst case with respect to
> supply chain attacks.
>
> It pulls in dependencies recursively from a relatively uncurated
> list of projects, puts the source of all those dependencies into a
> hidden directory in home, and runs Build.rs automatically with
> user permissions.
>

100% this. Wait until you learn how proc macros work.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Michael Matz via Gcc]

Hello,

On Wed, 3 Apr 2024, Jonathon Anderson wrote:

> Of course, this doesn't make the build system any less complex, but 
> projects using newer build systems seem easier to secure and audit than 
> those using overly flexible build systems like Autotools and maybe even 
> CMake. IMHO using a late-model build system is a relatively low 
> technical hurdle to overcome for the benefits noted above, switching 
> should be considered and in a positive light.

Note that we're talking not (only) about the build system itself, i.e. how 
to declare dependencies within the sources, and how to declare how to 
build them.  make it just fine for that (as are many others).  (In a way 
I think we meanwhile wouldn't really need automake and autogen, but 
rewriting all that in pure GNUmake is a major undertaking).

But Martin also specifically asked about alternatives for feature tests, 
i.e. autoconfs purpose.  I simply don't see how any alternative to it 
could be majorly "easier" or "less complex" at its core.  Going with the 
examples given upthread there is usually only one major solution: to check 
if a given system supports FOOBAR you need to bite the bullet and compile 
(and potentially run!) a small program using FOOBAR.  A configuration 
system that can do that (and I don't see any real alternative to that), no 
matter in which language it's written and how traditional or modern it is, 
also gives you enough rope to hang yourself, if you so choose.

If you get away without many configuration tests in your project then this 
is because what (e.g.) the compiler gives you, in the form of libstdc++ 
for example, abstracts away many of the peculiarities of a system.  But 
in order to be able to do that something (namely the config system of 
libstdc++) needs to determine what is or isn't supported by the system in 
order to correctly implement these abstractions.  I.e. things you depend 
on did the major lifting of hiding system divergence.

(Well, that, or you are very limited in the number of systems you support, 
which can be the right thing as well!)


Ciao,
Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

Hi,

On Wed, 2024-04-03 at 08:42 -0600, Jeff Law wrote:
> On 4/3/24 8:04 AM, Tom Tromey wrote:
> > > > > > > "Florian" == Florian Weimer  writes:
> > 
> > Florian> Everyone still pushes their own patches, and there are no
> > Florian> technical countermeasures in place to ensure that the pushed 
> > version is
> > Florian> the reviewed version.
> > 
> > This is a problem for gdb as well.
> > 
> > Probably we should switch to some kind of pull-request model, where
> > patches can only be landed via the UI, after sufficient review; and
> > where all generated files are regenerated by the robot before checkin.
> > (Or alternatively some CI runs and rejects patches where they don't
> > match.)
> I've very much prefer to move to a pull-request model.

Do you need any infrastructure updates to help (experiment) with that?
Now would be a great time to request some updates to patchwork or get
us to resurrect the gerrit server if that would be helpful.

We just published the Sourceware 2024 infrastructure plan:
https://inbox.sourceware.org/20240325095827.gi5...@gnu.wildebeest.org/
Setting priorities for the infrastructure for 2024 (and beyond). We are
just now scheduling and budgeting that work. So please get your
requests in.

Cheers,

Mark

Re: Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

Hi,

On Wed, 2024-04-03 at 08:53 -0500, Joel Sherrill wrote:
> On Wed, Apr 3, 2024, 3:09 AM Florian Weimer via Gdb  
> wrote:
> > * Guinevere Larsen via Overseers:
> > 
> > > Beyond that, we (GDB) are already experimenting with approved-by, and
> > > I think glibc was doing the same.
> > 
> > The glibc project uses Reviewed-by:, but it's completely unrelated to
> > this.  Everyone still pushes their own patches, and there are no
> > technical countermeasures in place to ensure that the pushed version is
> > the reviewed version.
> 
> Or that there isn't "collusion" between a malicious author and reviewer.
> Just tagging it approved or reviewed by just gives you two people to blame.
> It is not a perfect solution either.
> 
> But double checking and checklists are good practices.
> They are not foolproof if some bad actor is determined enough.

Agreed. If you just focus on completely fool proof technically
checkable measures then you end up doing nothing. But making things
like always getting a Reviewed-by or Tested-by tag in your commit
message does strengthen the social norms. And once they are common
practice you could even add some technical checks.

I am sure a really determined bad actor can always find some social or
technical engineering trick to "defeat" our project policies. But that
doesn't mean we shouldn't do things which are good practices anyway.

Cheers,

Mark

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker via Gcc]

Am Mittwoch, dem 03.04.2024 um 13:46 -0500 schrieb Jonathon Anderson via Gcc:
> Hello all,
> 
> On Wed, 2024-04-03 at 16:00 +0200, Michael Matz wrote:
> > > My take a way is that software needs to become less complex. Do 
> > > we really still need complex build systems such as autoconf?
> > 
> > (And, FWIW, testing for features isn't "complex".  And have you looked at 
> > other build systems?  I have, and none of them are less complex, just 
> > opaque in different ways from make+autotools).
> 
> Some brief opinions from a humble end-user:
> 
> I think the key difference here is that Autotools allows arbitrarily 
> generated code to be executed at any time. More modern build systems require 
> the use of specific commands/files to run arbitrary code, e.g. CMake (IIRC 
> [`execute_process()`][2] and [`ExternalProject`][3]), Meson 
> ([`run_command()`][1]), Cargo ([`build.rs`][4]).\

To me it seems that Cargo is the absolute worst case with respect to
supply chain attacks.

It pulls in dependencies recursively from a relatively uncurated
list of projects, puts the source of all those dependencies into a 
hidden directory in home, and runs Build.rs automatically with
user permissions.

Martin





> IMHO there are similarities here to the memory "safety" of Rust: Rust code 
> can have memory errors, but it can only come from Rust code declared as 
> `unsafe` (or bugs in the compiler itself). The scope is limited and those 
> scopes can be audited with more powerful microscopes... and removed if/when 
> the build system gains first-class support upstream.
> 
> There are other features in the newest build systems listed here (Meson and 
> Cargo) that make this particular attack vector harder. These build systems 
> don't have release tarballs with auxiliary files (e.g. [Meson's is very close 
> to `git archive`][5]), nor do their DSLs allow writing files to the source 
> tree. One could imagine a build/CI worker where the source tree is a 
> read-only bind-mount of a `git archive` extract, that might help defend 
> against attacks of this specific design.
> 
> It's also worth noting that Meson and Cargo use non-Turing-complete 
> declarative DSLs for their build configuration. This implies there is an 
> upper bound on the [cyclomatic complexity][6]-per-line of the build script 
> DSL itself. That doesn't mean you can't write complex Meson code (I have), 
> but it ends up being suspiciously long and thus clear something complex and 
> out of the ordinary is being done.
> 
> Of course, this doesn't make the build system any less complex, but projects 
> using newer build systems seem easier to secure and audit than those using 
> overly flexible build systems like Autotools and maybe even CMake. IMHO using 
> a late-model build system is a relatively low technical hurdle to overcome 
> for the benefits noted above, switching should be considered and in a 
> positive light.
> 
> (For context: my team recently switched our main C/C++ project from Autotools 
> to Meson. The one-time refactor itself was an effort, but after that we got 
> back up to speed quickly and we haven't looked back. Other projects may have 
> an easier time using an unofficial port in the [Meson WrapDB][7] as a 
> starting point.)
> 
> -Jonathon
> 
> [1]: https://mesonbuild.com/External-commands.html
> [2]: 
> https://cmake.org/cmake/help/latest/command/execute_process.html#execute-process
> [3]: https://cmake.org/cmake/help/latest/module/ExternalProject.html
> [4]: https://doc.rust-lang.org/cargo/reference/build-scripts.html
> [5]: https://mesonbuild.com/Creating-releases.html
> [6]: https://en.wikipedia.org/wiki/Cyclomatic_complexity
> [7]: https://mesonbuild.com/Wrapdb-projects.html

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathan Wakely via Gcc]

On Wed, 3 Apr 2024 at 19:36, Toon Moene  wrote:
>
> On 4/3/24 20:25, Ian Lance Taylor wrote:
>
> > Note that the attack really didn't have anything to do with
> > compressing data.  The library used an IFUNC to change the PLT of a
> > different function, so it effectively took control of the code that
> > verified the cryptographic key.  The only part of the attack that
> > involved compression was the fact that it happened to live in a
> > compression library.  And it wouldn't matter whether the code that
> > verified the cryptographic key was run as root either; the effect of
> > the attack was to say that the key was OK, and that sshd should
> > execute the command, and of course that execution must be done on
> > behalf of the requesting user, which (as I understand it) could be
> > root.
>
> Ah, OK - that's what I missed.
>
> Does your explanation mean that - if, as I do in my sshd config file -
> you *forbid* root access via sshd in *any* way, you wouldn't be vulnerable ?


No, sshd is still running as root.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andreas Schwab]

On Apr 03 2024, Paul Floyd via Gcc wrote:

> On 03-04-24 14:32, Martin Uecker via Gcc wrote:
>
>> The backdoor was hidden in a complicated autoconf script...
>
> How many uncomplicated autoconf scripts exist in the real world?

Probably the same amount as in any other build system.  Building
(portable) software requires a certain level of complexity.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson via Gcc]

Hello all,

On Wed, 2024-04-03 at 16:00 +0200, Michael Matz wrote:
> > My take a way is that software needs to become less complex. Do 
> > we really still need complex build systems such as autoconf?
>
> (And, FWIW, testing for features isn't "complex".  And have you looked at 
> other build systems?  I have, and none of them are less complex, just 
> opaque in different ways from make+autotools).

Some brief opinions from a humble end-user:

I think the key difference here is that Autotools allows arbitrarily generated 
code to be executed at any time. More modern build systems require the use of 
specific commands/files to run arbitrary code, e.g. CMake (IIRC 
[`execute_process()`][2] and [`ExternalProject`][3]), Meson 
([`run_command()`][1]), Cargo ([`build.rs`][4]).\
IMHO there are similarities here to the memory "safety" of Rust: Rust code can 
have memory errors, but it can only come from Rust code declared as `unsafe` 
(or bugs in the compiler itself). The scope is limited and those scopes can be 
audited with more powerful microscopes... and removed if/when the build system 
gains first-class support upstream.

There are other features in the newest build systems listed here (Meson and 
Cargo) that make this particular attack vector harder. These build systems 
don't have release tarballs with auxiliary files (e.g. [Meson's is very close 
to `git archive`][5]), nor do their DSLs allow writing files to the source 
tree. One could imagine a build/CI worker where the source tree is a read-only 
bind-mount of a `git archive` extract, that might help defend against attacks 
of this specific design.

It's also worth noting that Meson and Cargo use non-Turing-complete declarative 
DSLs for their build configuration. This implies there is an upper bound on the 
[cyclomatic complexity][6]-per-line of the build script DSL itself. That 
doesn't mean you can't write complex Meson code (I have), but it ends up being 
suspiciously long and thus clear something complex and out of the ordinary is 
being done.

Of course, this doesn't make the build system any less complex, but projects 
using newer build systems seem easier to secure and audit than those using 
overly flexible build systems like Autotools and maybe even CMake. IMHO using a 
late-model build system is a relatively low technical hurdle to overcome for 
the benefits noted above, switching should be considered and in a positive 
light.

(For context: my team recently switched our main C/C++ project from Autotools 
to Meson. The one-time refactor itself was an effort, but after that we got 
back up to speed quickly and we haven't looked back. Other projects may have an 
easier time using an unofficial port in the [Meson WrapDB][7] as a starting 
point.)

-Jonathon

[1]: https://mesonbuild.com/External-commands.html
[2]: 
https://cmake.org/cmake/help/latest/command/execute_process.html#execute-process
[3]: https://cmake.org/cmake/help/latest/module/ExternalProject.html
[4]: https://doc.rust-lang.org/cargo/reference/build-scripts.html
[5]: https://mesonbuild.com/Creating-releases.html
[6]: https://en.wikipedia.org/wiki/Cyclomatic_complexity
[7]: https://mesonbuild.com/Wrapdb-projects.html

Re: Sourceware mitigating and preventing the next xz-backdoor

[Toon Moene]

On 4/3/24 20:25, Ian Lance Taylor wrote:


Note that the attack really didn't have anything to do with
compressing data.  The library used an IFUNC to change the PLT of a
different function, so it effectively took control of the code that
verified the cryptographic key.  The only part of the attack that
involved compression was the fact that it happened to live in a
compression library.  And it wouldn't matter whether the code that
verified the cryptographic key was run as root either; the effect of
the attack was to say that the key was OK, and that sshd should
execute the command, and of course that execution must be done on
behalf of the requesting user, which (as I understand it) could be
root.


Ah, OK - that's what I missed.

Does your explanation mean that - if, as I do in my sshd config file - 
you *forbid* root access via sshd in *any* way, you wouldn't be vulnerable ?


Thanks,

--
Toon Moene - e-mail: t...@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands

Re: Sourceware mitigating and preventing the next xz-backdoor

[Ian Lance Taylor via Gcc]

On Wed, Apr 3, 2024 at 11:05 AM Toon Moene  wrote:
>
> Two questions arise (as far as I am concerned):
>
> 1. Do daemons like sshd *have* to be linked with shared libraries ?
> Or could it be left to the security minded of the downstream
> (binary) distributions to link it statically with known & proven
> correct libraries ?

I like static linking personally, but it seems like glibc has made a
decision that shared linking is much preferred over static.  That said
my guess is that this kind of attack would have been effective on any
executable built as PIE.  It relied on using an IFUNC hook to adjust
the PLT entry for a different function.  And, of course, most
executables are built as PIE these days, because that is more secure
against different kinds of attacks.

> 2. Is it a limitation of the Unix / Linux daemon concept that, once
> such a process needs root access, it has to have root access
> *always* - even when performing trivial tasks like compressing
> data ?
>
> I recall quite well (vis-a-vis question 2) that the VMS equivalent would
> drop all privileges at the start of the code, and request only those
> relevant when actually needed (e.g., to open a file for reading that was
> owned by [the equivalent on VMS] of root - or perform other functions
> that only root could do), and then drop them immediately afterwards again.

Note that the attack really didn't have anything to do with
compressing data.  The library used an IFUNC to change the PLT of a
different function, so it effectively took control of the code that
verified the cryptographic key.  The only part of the attack that
involved compression was the fact that it happened to live in a
compression library.  And it wouldn't matter whether the code that
verified the cryptographic key was run as root either; the effect of
the attack was to say that the key was OK, and that sshd should
execute the command, and of course that execution must be done on
behalf of the requesting user, which (as I understand it) could be
root.

Ian

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning via Gcc]

> On Apr 3, 2024, at 2:04 PM, Toon Moene  wrote:
> 
> On 4/1/24 17:06, Mark Wielaard wrote:
> 
>> A big thanks to everybody working this long Easter weekend who helped
>> analyze the xz-backdoor and making sure the impact on Sourceware and
>> the hosted projects was minimal.
> 
> Thanks for those efforts !
> 
> Now, I have seen two more days of thinking about this vulnerability ... but 
> no one seem to address the following issues:
> 
> A hack was made in liblzma, which, when the code was executed by a daemon 
> that by virtue of its function, *has* to be run as root, was effective.
> 
> Two questions arise (as far as I am concerned):
> 
> 1. Do daemons like sshd *have* to be linked with shared libraries ?
>   Or could it be left to the security minded of the downstream
>   (binary) distributions to link it statically with known & proven
>   correct libraries ?

I would add: should IFUNC be deleted?  Or alternatively, should it be strictly 
limited only to non-security-sensitive applications when not running as root?

> 2. Is it a limitation of the Unix / Linux daemon concept that, once
>   such a process needs root access, it has to have root access
>   *always* - even when performing trivial tasks like compressing
>   data ?

Clearly not, given the existence of the "seteuid" syscall.

> I recall quite well (vis-a-vis question 2) that the VMS equivalent would drop 
> all privileges at the start of the code, and request only those relevant when 
> actually needed (e.g., to open a file for reading that was owned by [the 
> equivalent on VMS] of root - or perform other functions that only root could 
> do), and then drop them immediately afterwards again.

Yes, and with additional effort all "root" type applications could be written 
that way.

paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Toon Moene]

On 4/1/24 17:06, Mark Wielaard wrote:


A big thanks to everybody working this long Easter weekend who helped
analyze the xz-backdoor and making sure the impact on Sourceware and
the hosted projects was minimal.


Thanks for those efforts !

Now, I have seen two more days of thinking about this vulnerability ... 
but no one seem to address the following issues:


A hack was made in liblzma, which, when the code was executed by a 
daemon that by virtue of its function, *has* to be run as root, was 
effective.


Two questions arise (as far as I am concerned):

1. Do daemons like sshd *have* to be linked with shared libraries ?
   Or could it be left to the security minded of the downstream
   (binary) distributions to link it statically with known & proven
   correct libraries ?

2. Is it a limitation of the Unix / Linux daemon concept that, once
   such a process needs root access, it has to have root access
   *always* - even when performing trivial tasks like compressing
   data ?

I recall quite well (vis-a-vis question 2) that the VMS equivalent would 
drop all privileges at the start of the code, and request only those 
relevant when actually needed (e.g., to open a file for reading that was 
owned by [the equivalent on VMS] of root - or perform other functions 
that only root could do), and then drop them immediately afterwards again.


Kind regards,

--
Toon Moene - e-mail: t...@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Floyd via Gcc]

On 03-04-24 14:32, Martin Uecker via Gcc wrote:



The backdoor was hidden in a complicated autoconf script...


How many uncomplicated autoconf scripts exist in the real world?

A+
Paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathan Wakely via Gcc]

On Wed, 3 Apr 2024 at 15:34, Martin Uecker via Gcc  wrote:
> I ask a very specific question: To what extend is testing
> for features instead of semantic versions and/or supported
> standards still necessary?  This seems like a problematic approach
> that  may have been necessary decades ago, but it seems it may be
> time to move on.


What standard or version should I check for to detect nl_langinfo_l
support? It's in POSIX 2017 but macOS supports it in 
despite not being POSIX 2017 compliant.

What about _get_osfhandle? Or fwrite_unlocked?

What about whether the platform ABI allows aligning global objects to
the cacheline size?

Those are just a few of the most recent autoconf checks I've written
for libstdc++ in the past few months, not decades ago.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andreas Schwab]

On Apr 03 2024, Martin Uecker wrote:

> I ask a very specific question: To what extend is testing 
> for features instead of semantic versions and/or supported
> standards still necessary?  This seems like a problematic approach
> that  may have been necessary decades ago, but it seems it may be
> time to move on.

CMake is doing mostly the same feature tests under the hood.  Because
they are proven convenient and effective.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker via Gcc]

Am Mittwoch, dem 03.04.2024 um 18:02 +0200 schrieb Michael Matz:
> Hello,
> 
> On Wed, 3 Apr 2024, Martin Uecker wrote:
> 
> > The backdoor was hidden in a complicated autoconf script...
> 
> Which itself had multiple layers and could just as well have been a 
> complicated cmake function.

Don't me wrong, cmake is no way better. Another problem was 
actually hidden in a cmake test in upstream git in plain
sight:

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00;hp=af071ef7702debef4f1d324616a0137a5001c14c

> 
> > > (And, FWIW, testing for features isn't "complex".  And have you looked at 
> > > other build systems?  I have, and none of them are less complex, just 
> > > opaque in different ways from make+autotools).
> > 
> > I ask a very specific question: To what extend is testing 
> > for features instead of semantic versions and/or supported
> > standards still necessary?
> 
> I can't answer this with absolute certainty, but points to consider: the 
> semantic versions need to be maintained just as well, in some magic way.  

It would certainly need to be maintained just like now autoconf
configuration needs to be maintained.

> Because ultimately software depend on features of dependencies not on 
> arbitrary numbers given to them.  The numbers encode these features, in 
> the best case, when there are no errors.  So, no, version numbers are not 
> a replacement for feature tests, they are a proxy.  One that is manually 
> maintained, and hence prone to errors.

Tests are also prone to errors and - as the example above shows -
also very fragile and susceptible to manipulation.

> 
> Now, supported standards: which one? ;-)  Or more in earnest: while on 
> this mailing list here we could chose a certain set, POSIX, some 
> languages, Windows, MacOS (versions so-and-so).  What about other 
> software relying on other 3rdparty feature providers (libraries or system 
> services)?  Without standards?
> 
> So, without absolute certainty, but with a little bit of it: yes, feature 
> tests are required in general.  That doesn't mean that we could not 
> do away with quite some of them for (e.g.) GCC, those that hold true on 
> any platform we support.  But we can't get rid of the infrastructure for 
> that, and can't get rid of certain classes of tests.
> 
> > This seems like a problematic approach that may have been necessary 
> > decades ago, but it seems it may be time to move on.
> 
> I don't see that.  Many aspects of systems remain non-standardized.

This is just part of the problem.

Martin

> 
> 
> Ciao,
> Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Joel Sherrill]

On Wed, Apr 3, 2024 at 11:03 AM Michael Matz via Gdb 
wrote:

> Hello,
>
> On Wed, 3 Apr 2024, Martin Uecker wrote:
>
> > The backdoor was hidden in a complicated autoconf script...
>
> Which itself had multiple layers and could just as well have been a
> complicated cmake function.
>
> > > (And, FWIW, testing for features isn't "complex".  And have you looked
> at
> > > other build systems?  I have, and none of them are less complex, just
> > > opaque in different ways from make+autotools).
> >
> > I ask a very specific question: To what extend is testing
> > for features instead of semantic versions and/or supported
> > standards still necessary?
>
> I can't answer this with absolute certainty, but points to consider: the
> semantic versions need to be maintained just as well, in some magic way.
> Because ultimately software depend on features of dependencies not on
> arbitrary numbers given to them.  The numbers encode these features, in
> the best case, when there are no errors.  So, no, version numbers are not
> a replacement for feature tests, they are a proxy.  One that is manually
> maintained, and hence prone to errors.
>
> Now, supported standards: which one? ;-)  Or more in earnest: while on
> this mailing list here we could chose a certain set, POSIX, some
> languages, Windows, MacOS (versions so-and-so).  What about other
> software relying on other 3rdparty feature providers (libraries or system
> services)?  Without standards?
>
> So, without absolute certainty, but with a little bit of it: yes, feature
> tests are required in general.  That doesn't mean that we could not
> do away with quite some of them for (e.g.) GCC, those that hold true on
> any platform we support.  But we can't get rid of the infrastructure for
> that, and can't get rid of certain classes of tests.
>

Even assuming a set of standards still leaves issues like supporting a
wide range of OS versions which may evolve to support different versions
of the same standard. And then there is the case of standards having
deliberately implementation defined behavior. Last week, I had to double
check what malloc(0) and free(NULL) are supposed to do per POSIX.
Guess what -- implementation defined behavior options.

And worse, POSIX doesn't define the default attributes for pthreads and
the associated synchronization objects. These vary widely across OSes.

Then there are the broken things autoconf checks for. Somehow the
"broken sed from Solaris" check sticks in my head. Or needing to use
GNU sed instead of FreeBSD sed to build a mips embedded target.

And this ignores issues like word size issues. 32 vs 64 bits for one,
Or weird CPUs where nothing is byte addressable (TI C3x/4x is one
example).


> > This seems like a problematic approach that may have been necessary
> > decades ago, but it seems it may be time to move on.
>
> I don't see that.  Many aspects of systems remain non-standardized.
>

Again from pthreads, manipulating affinity on multi-core systems and
having names for pthreads are non-standard but commonly available
with varying APIs.

And the standards have plenty of wiggle room in them. Undefined areas
or deliberately implementation defined.

--joel

>
>
> Ciao,
> Michael.
>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Michael Matz via Gcc]

Hello,

On Wed, 3 Apr 2024, Martin Uecker wrote:

> The backdoor was hidden in a complicated autoconf script...

Which itself had multiple layers and could just as well have been a 
complicated cmake function.

> > (And, FWIW, testing for features isn't "complex".  And have you looked at 
> > other build systems?  I have, and none of them are less complex, just 
> > opaque in different ways from make+autotools).
> 
> I ask a very specific question: To what extend is testing 
> for features instead of semantic versions and/or supported
> standards still necessary?

I can't answer this with absolute certainty, but points to consider: the 
semantic versions need to be maintained just as well, in some magic way.  
Because ultimately software depend on features of dependencies not on 
arbitrary numbers given to them.  The numbers encode these features, in 
the best case, when there are no errors.  So, no, version numbers are not 
a replacement for feature tests, they are a proxy.  One that is manually 
maintained, and hence prone to errors.

Now, supported standards: which one? ;-)  Or more in earnest: while on 
this mailing list here we could chose a certain set, POSIX, some 
languages, Windows, MacOS (versions so-and-so).  What about other 
software relying on other 3rdparty feature providers (libraries or system 
services)?  Without standards?

So, without absolute certainty, but with a little bit of it: yes, feature 
tests are required in general.  That doesn't mean that we could not 
do away with quite some of them for (e.g.) GCC, those that hold true on 
any platform we support.  But we can't get rid of the infrastructure for 
that, and can't get rid of certain classes of tests.

> This seems like a problematic approach that may have been necessary 
> decades ago, but it seems it may be time to move on.

I don't see that.  Many aspects of systems remain non-standardized.


Ciao,
Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton via Gcc]

On Wed, Apr 3, 2024 at 10:33 AM Martin Uecker via Gdb
 wrote:
>
> Am Mittwoch, dem 03.04.2024 um 16:00 +0200 schrieb Michael Matz:
> > [...]
> > (And, FWIW, testing for features isn't "complex".  And have you looked at
> > other build systems?  I have, and none of them are less complex, just
> > opaque in different ways from make+autotools).
>
> I ask a very specific question: To what extend is testing
> for features instead of semantic versions and/or supported
> standards still necessary?  This seems like a problematic approach
> that  may have been necessary decades ago, but it seems it may be
> time to move on.

I think it is still needed. As a first example, Musl does not define
preprocessor macros to identify itself. The project feels features
should be probed at build time with a tool like Autoconf. As a second
example, activating code paths, like AVX2 and AVX512, requires
ensuring the compiler actually supports the ISA. Building a program on
an older distribution with an older compiler could run afoul.

Jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeff Law via Gcc]

On 4/3/24 8:04 AM, Tom Tromey wrote:

"Florian" == Florian Weimer  writes:


Florian> Everyone still pushes their own patches, and there are no
Florian> technical countermeasures in place to ensure that the pushed version is
Florian> the reviewed version.

This is a problem for gdb as well.

Probably we should switch to some kind of pull-request model, where
patches can only be landed via the UI, after sufficient review; and
where all generated files are regenerated by the robot before checkin.
(Or alternatively some CI runs and rejects patches where they don't
match.)

I've very much prefer to move to a pull-request model.

jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker via Gcc]

Am Mittwoch, dem 03.04.2024 um 16:00 +0200 schrieb Michael Matz:
> Hello,
> 
> On Wed, 3 Apr 2024, Martin Uecker via Gcc wrote:
> 
> > > > Seems reasonable, but note that it wouldn't make any difference to
> > > > this attack.  The liblzma library was modified to corrupt the sshd
> > > > binary, when sshd was linked against liblzma.  The actual attack
> > > > occurred via a connection to a corrupt sshd.  If sshd was running as
> > > > root, as is normal, the attacker had root access to the machine.  None
> > > > of the attacking steps had anything to do with having root access
> > > > while building or installing the program.
> > 
> > There does not seem a single good solution against something like this.
> > 
> > My take a way is that software needs to become less complex. Do 
> > we really still need complex build systems such as autoconf?
> 
> Do we really need complex languages like C++ to write our software in?  
> SCNR :)  

Probably not.

> Complexity lies in the eye of the beholder, but to be honest in 
> the software that we're dealing with here, the build system or autoconf 
> does _not_ come to mind first when thinking about complexity.

The backdoor was hidden in a complicated autoconf script...

> 
> (And, FWIW, testing for features isn't "complex".  And have you looked at 
> other build systems?  I have, and none of them are less complex, just 
> opaque in different ways from make+autotools).

I ask a very specific question: To what extend is testing 
for features instead of semantic versions and/or supported
standards still necessary?  This seems like a problematic approach
that  may have been necessary decades ago, but it seems it may be
time to move on.


Martin

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning via Gcc]

> On Apr 3, 2024, at 10:00 AM, Michael Matz  wrote:
> 
> Hello,
> 
> On Wed, 3 Apr 2024, Martin Uecker via Gcc wrote:
> 
 Seems reasonable, but note that it wouldn't make any difference to
 this attack.  The liblzma library was modified to corrupt the sshd
 binary, when sshd was linked against liblzma.  The actual attack
 occurred via a connection to a corrupt sshd.  If sshd was running as
 root, as is normal, the attacker had root access to the machine.  None
 of the attacking steps had anything to do with having root access
 while building or installing the program.
>> 
>> There does not seem a single good solution against something like this.
>> 
>> My take a way is that software needs to become less complex. Do 
>> we really still need complex build systems such as autoconf?
> 
> Do we really need complex languages like C++ to write our software in?  
> SCNR :)  Complexity lies in the eye of the beholder, but to be honest in 
> the software that we're dealing with here, the build system or autoconf 
> does _not_ come to mind first when thinking about complexity.
> 
> (And, FWIW, testing for features isn't "complex".  And have you looked at 
> other build systems?  I have, and none of them are less complex, just 
> opaque in different ways from make+autotools).
> 
> Ciao,
> Michael.

I would tend to agree with that even given my limited exposure to alternatives.

One aspect of the present attack that needs to be cured is that -- as I 
understand it -- the open source repository was fine but the kit as distributed 
had been subverted.  In other words, the standard assumption that the 
repository actually corresponds to the released code was not valid.  And 
furthermore, that it wasn't unusual for the kit to contain different or 
additional elements, just that it wasn't supposed to differ in malicious ways.

One possible answer is for all elements of kits to be made explicitly visible, 
though generated files probably don't want to be held in a normal source 
control system.  Another possible answer is for consumers of kits to treat kits 
as suspect, and have them unpacked and examined -- including any elements not 
source controlled -- before acceptance.  I think the first option is better 
because it exposes these additional elements to ongoing scrutiny from the 
entire community, rather than only one-time inspection by release managers who 
are probably quite pressed for time.

Either way, the reasons for these extra files to exist and the manner in which 
they are supposed to be generated would need to be both well documented and 
readily reproducible by outside parties.

paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Tom Tromey]

> "Florian" == Florian Weimer  writes:

Florian> Everyone still pushes their own patches, and there are no
Florian> technical countermeasures in place to ensure that the pushed version is
Florian> the reviewed version.

This is a problem for gdb as well.

Probably we should switch to some kind of pull-request model, where
patches can only be landed via the UI, after sufficient review; and
where all generated files are regenerated by the robot before checkin.
(Or alternatively some CI runs and rejects patches where they don't
match.)

Tom

Re: Sourceware mitigating and preventing the next xz-backdoor

[Michael Matz via Gcc]

Hello,

On Wed, 3 Apr 2024, Martin Uecker via Gcc wrote:

> > > Seems reasonable, but note that it wouldn't make any difference to
> > > this attack.  The liblzma library was modified to corrupt the sshd
> > > binary, when sshd was linked against liblzma.  The actual attack
> > > occurred via a connection to a corrupt sshd.  If sshd was running as
> > > root, as is normal, the attacker had root access to the machine.  None
> > > of the attacking steps had anything to do with having root access
> > > while building or installing the program.
> 
> There does not seem a single good solution against something like this.
> 
> My take a way is that software needs to become less complex. Do 
> we really still need complex build systems such as autoconf?

Do we really need complex languages like C++ to write our software in?  
SCNR :)  Complexity lies in the eye of the beholder, but to be honest in 
the software that we're dealing with here, the build system or autoconf 
does _not_ come to mind first when thinking about complexity.

(And, FWIW, testing for features isn't "complex".  And have you looked at 
other build systems?  I have, and none of them are less complex, just 
opaque in different ways from make+autotools).


Ciao,
Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Joel Sherrill]

On Wed, Apr 3, 2024, 3:09 AM Florian Weimer via Gdb 
wrote:

> * Guinevere Larsen via Overseers:
>
> > Beyond that, we (GDB) are already experimenting with approved-by, and
> > I think glibc was doing the same.
>
> The glibc project uses Reviewed-by:, but it's completely unrelated to
> this.  Everyone still pushes their own patches, and there are no
> technical countermeasures in place to ensure that the pushed version is
> the reviewed version.
>

Or that there isn't "collusion" between a malicious author and reviewer.
Just tagging it approved or reviewed by just gives you two people to blame.
It is not a perfect solution either.

But double checking and checklists are good practices. They are not
foolproof if some bad actor is determined enough.

--joel



> Thanks,
> Florian
>
>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Florian Weimer via Gcc]

* Guinevere Larsen via Overseers:

> Beyond that, we (GDB) are already experimenting with approved-by, and
> I think glibc was doing the same.

The glibc project uses Reviewed-by:, but it's completely unrelated to
this.  Everyone still pushes their own patches, and there are no
technical countermeasures in place to ensure that the pushed version is
the reviewed version.

Thanks,
Florian

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker via Gcc]

Am Dienstag, dem 02.04.2024 um 13:28 -0700 schrieb Ian Lance Taylor via Gcc:
> > On Tue, Apr 2, 2024 at 1:21 PM Paul Koning via Gcc  wrote:
> > > > 
> > > > Would it help to require (rather than just recommend) "don't use root 
> > > > except for the actual 'install' step" ?
> > 
> > Seems reasonable, but note that it wouldn't make any difference to
> > this attack.  The liblzma library was modified to corrupt the sshd
> > binary, when sshd was linked against liblzma.  The actual attack
> > occurred via a connection to a corrupt sshd.  If sshd was running as
> > root, as is normal, the attacker had root access to the machine.  None
> > of the attacking steps had anything to do with having root access
> > while building or installing the program.

There does not seem a single good solution against something like this.

My take a way is that software needs to become less complex. Do 
we really still need complex build systems such as autoconf?
Are there still so many different configurations with subtle differences 
that every single feature needs to be tested individually by
running code at build time?

Martin

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton via Gcc]

On Tue, Apr 2, 2024 at 7:35 PM Paul Koning via Gdb  wrote:
> [...]
>
> I agree that GDB, and for that matter other projects with significant numbers 
> of contributors, are not nearly as likely to be vulnerable to this sort of 
> attack.  But I worry that xz may not be the only project that's small enough 
> to be vulnerable, and be security-relevant in not so obvious ways.

This cuts a lot deeper than folks think. Here are two other examples
off the top of my head...

Other vulnerable projects include ncurses and libnettle. Ncurses is
run by Thomas Dickey (https://invisible-island.net/). libnettle is run
by Niels Möller (https://www.lysator.liu.se/~nisse/nettle/). Both are
one-man shows with no continuity plans. Dickey does not even run a
public version control system. You have to download his release
tarballs, and there's no history to review or make pull requests
against. If DIckey or Möller got hit by a bus crossing the street,
there would be problems for years.

Jeff



> One question that comes to mind is whether there has been an effort across 
> the open source community to identify possible other targets of such attacks. 
>  Contributions elsewhere by the suspect in this case are an obvious concern, 
> but similar scenarios with different names could also be.  That probably 
> should be an ongoing activity: whenever some external component is used, it 
> would be worth knowing how it is maintained, and how many eyeballs are 
> involved.  Even if this isn't done by everyone, it seems like a proper 
> precaution for security sensitive projects.
>
> Another question that comes to mind: I would guess that relevant law 
> enforcement agencies are already looking into this, but it would seem 
> appropriate for those closest to the attacked software to reach out 
> explicitly and assist in any criminal investigations.
>
> paul
>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning via Gcc]

> On Apr 2, 2024, at 6:08 PM, Guinevere Larsen  wrote:
> 
> On 4/2/24 16:54, Sandra Loosemore wrote:
>> On 4/1/24 09:06, Mark Wielaard wrote:
>>> A big thanks to everybody working this long Easter weekend who helped
>>> analyze the xz-backdoor and making sure the impact on Sourceware and
>>> the hosted projects was minimal.
>>> 
>>> This email isn't about the xz-backdoor itself. Do see Sam James FAQ
>>> https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
>>> (Sorry for the github link, but this one does seem viewable without
>>> proprietary javascript)
>>> 
>>> We should discuss what we have been doing and should do more to
>>> mitigate and prevent the next xz-backdoor. There are a couple of
>>> Sourceware services that can help with that.
>>> 
>>> TLDR;
>>> - Replicatable isolated container/VMs are nice, we want more.
>>> - autoregen buildbots, it should be transparent (and automated) how to
>>>regenerate build/source files.
>>> - Automate (snapshot) releases tarballs.
>>> - Reproducible releases (from git).
>>> 
>>> [snip]
>> 
>> While I appreciate the effort to harden the Sourceware infrastructure 
>> against malicious attacks and want to join in on thanking everyone who 
>> helped analyze this issue, to me it seems like the much bigger problem is 
>> that XZ had a maintainer who appears to have acted in bad faith.  Are the 
>> development processes used by the GNU toolchain components robust enough to 
>> cope with deliberate sabotage of the code base?  Do we have enough eyes 
>> available to ensure that every commit, even those by designated maintainers, 
>> is vetted by someone else?  Do we to harden our process, too, to require all 
>> patches to be signed off by someone else before committing?
>> 
>> -Sandra
>> 
>> 
> What likely happened for the maintainer who acted in bad faith was that they 
> entered the project with bad faith intent from the start - seeing as they 
> were only involved with the project for 2 years, and there was much social 
> pressure from fake email accounts for the single maintainer of XZ to accept 
> help.
> 
> While we would obviously like to have more area maintainers and possibly 
> global maintainers to help spread the load, I don't think any of the projects 
> listed here are all that susceptible to the same type of social engineering. 
> For one, getting the same type of blanket approval would be a much more 
> involved process because we already have a reasonable amount of people with 
> those privileges, no one is dealing with burnout and sassy customers saying 
> we aren't doing enough.
> 
> Beyond that, we (GDB) are already experimenting with approved-by, and I think 
> glibc was doing the same. That guarantees at least a second set of eyes that 
> analyzed and agreed with the patch, I don't think signed-off would add more 
> than that tag (even if security was not the reason why we implemented them).
> 
> -- 
> Cheers,
> Guinevere Larsen
> She/Her/Hers

I agree that GDB, and for that matter other projects with significant numbers 
of contributors, are not nearly as likely to be vulnerable to this sort of 
attack.  But I worry that xz may not be the only project that's small enough to 
be vulnerable, and be security-relevant in not so obvious ways.

One question that comes to mind is whether there has been an effort across the 
open source community to identify possible other targets of such attacks.  
Contributions elsewhere by the suspect in this case are an obvious concern, but 
similar scenarios with different names could also be.  That probably should be 
an ongoing activity: whenever some external component is used, it would be 
worth knowing how it is maintained, and how many eyeballs are involved.  Even 
if this isn't done by everyone, it seems like a proper precaution for security 
sensitive projects.

Another question that comes to mind: I would guess that relevant law 
enforcement agencies are already looking into this, but it would seem 
appropriate for those closest to the attacked software to reach out explicitly 
and assist in any criminal investigations.

paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton via Gcc]

On Tue, Apr 2, 2024 at 6:09 PM Guinevere Larsen via Gdb
 wrote:
> [...]
> What likely happened for the maintainer who acted in bad faith was that
> they entered the project with bad faith intent from the start - seeing
> as they were only involved with the project for 2 years, and there was
> much social pressure from fake email accounts for the single maintainer
> of XZ to accept help.

The infiltration appears to have started offline, earlier than June
2022. See .

> While we would obviously like to have more area maintainers and possibly
> global maintainers to help spread the load, I don't think any of the
> projects listed here are all that susceptible to the same type of social
> engineering. For one, getting the same type of blanket approval would be
> a much more involved process because we already have a reasonable amount
> of people with those privileges, no one is dealing with burnout and
> sassy customers saying we aren't doing enough.
>
> Beyond that, we (GDB) are already experimenting with approved-by, and I
> think glibc was doing the same. That guarantees at least a second set of
> eyes that analyzed and agreed with the patch, I don't think signed-off
> would add more than that tag (even if security was not the reason why we
> implemented them).

Jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Guinevere Larsen via Gcc]

On 4/2/24 16:54, Sandra Loosemore wrote:

On 4/1/24 09:06, Mark Wielaard wrote:

A big thanks to everybody working this long Easter weekend who helped
analyze the xz-backdoor and making sure the impact on Sourceware and
the hosted projects was minimal.

This email isn't about the xz-backdoor itself. Do see Sam James FAQ
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
(Sorry for the github link, but this one does seem viewable without
proprietary javascript)

We should discuss what we have been doing and should do more to
mitigate and prevent the next xz-backdoor. There are a couple of
Sourceware services that can help with that.

TLDR;
- Replicatable isolated container/VMs are nice, we want more.
- autoregen buildbots, it should be transparent (and automated) how to
   regenerate build/source files.
- Automate (snapshot) releases tarballs.
- Reproducible releases (from git).

[snip]


While I appreciate the effort to harden the Sourceware infrastructure 
against malicious attacks and want to join in on thanking everyone who 
helped analyze this issue, to me it seems like the much bigger problem 
is that XZ had a maintainer who appears to have acted in bad faith.  
Are the development processes used by the GNU toolchain components 
robust enough to cope with deliberate sabotage of the code base?  Do 
we have enough eyes available to ensure that every commit, even those 
by designated maintainers, is vetted by someone else?  Do we to harden 
our process, too, to require all patches to be signed off by someone 
else before committing?


-Sandra


What likely happened for the maintainer who acted in bad faith was that 
they entered the project with bad faith intent from the start - seeing 
as they were only involved with the project for 2 years, and there was 
much social pressure from fake email accounts for the single maintainer 
of XZ to accept help.


While we would obviously like to have more area maintainers and possibly 
global maintainers to help spread the load, I don't think any of the 
projects listed here are all that susceptible to the same type of social 
engineering. For one, getting the same type of blanket approval would be 
a much more involved process because we already have a reasonable amount 
of people with those privileges, no one is dealing with burnout and 
sassy customers saying we aren't doing enough.


Beyond that, we (GDB) are already experimenting with approved-by, and I 
think glibc was doing the same. That guarantees at least a second set of 
eyes that analyzed and agreed with the patch, I don't think signed-off 
would add more than that tag (even if security was not the reason why we 
implemented them).


--
Cheers,
Guinevere Larsen
She/Her/Hers

Re: Sourceware mitigating and preventing the next xz-backdoor

[Ian Lance Taylor via Gcc]

On Tue, Apr 2, 2024 at 1:21 PM Paul Koning via Gcc  wrote:
>
> Would it help to require (rather than just recommend) "don't use root except 
> for the actual 'install' step" ?

Seems reasonable, but note that it wouldn't make any difference to
this attack.  The liblzma library was modified to corrupt the sshd
binary, when sshd was linked against liblzma.  The actual attack
occurred via a connection to a corrupt sshd.  If sshd was running as
root, as is normal, the attacker had root access to the machine.  None
of the attacking steps had anything to do with having root access
while building or installing the program.

Ian

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning via Gcc]

> On Apr 2, 2024, at 4:03 PM, Paul Eggert  wrote:
> 
> On 4/2/24 12:54, Sandra Loosemore wrote:
>> Do we to harden our process, too, to require all patches to be signed off by 
>> someone else before committing?
> 
> It's easy for an attacker to arrange to have "someone else" in cahoots.
> 
> Although signoffs can indeed help catch inadvertent mistakes, they're 
> relatively useless against determined attacks of this form, and we must 
> assume that nation-state attackers will be determined.

Another consideration is the size of the project.  "Many eyeballs" helps if 
there are plenty of people watching.  For smaller tools that have only a small 
body of contributors, it's easier for one or two malicious ones to subvert 
things.

Would it help to require (rather than just recommend) "don't use root except 
for the actual 'install' step" ?

paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/2/24 12:54, Sandra Loosemore wrote:
Do we to harden our process, too, to require all patches to be signed 
off by someone else before committing?


It's easy for an attacker to arrange to have "someone else" in cahoots.

Although signoffs can indeed help catch inadvertent mistakes, they're 
relatively useless against determined attacks of this form, and we must 
assume that nation-state attackers will be determined.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sandra Loosemore]

On 4/1/24 09:06, Mark Wielaard wrote:

A big thanks to everybody working this long Easter weekend who helped
analyze the xz-backdoor and making sure the impact on Sourceware and
the hosted projects was minimal.

This email isn't about the xz-backdoor itself. Do see Sam James FAQ
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
(Sorry for the github link, but this one does seem viewable without
proprietary javascript)

We should discuss what we have been doing and should do more to
mitigate and prevent the next xz-backdoor. There are a couple of
Sourceware services that can help with that.

TLDR;
- Replicatable isolated container/VMs are nice, we want more.
- autoregen buildbots, it should be transparent (and automated) how to
   regenerate build/source files.
- Automate (snapshot) releases tarballs.
- Reproducible releases (from git).

[snip]


While I appreciate the effort to harden the Sourceware infrastructure 
against malicious attacks and want to join in on thanking everyone who 
helped analyze this issue, to me it seems like the much bigger problem 
is that XZ had a maintainer who appears to have acted in bad faith.  Are 
the development processes used by the GNU toolchain components robust 
enough to cope with deliberate sabotage of the code base?  Do we have 
enough eyes available to ensure that every commit, even those by 
designated maintainers, is vetted by someone else?  Do we to harden our 
process, too, to require all patches to be signed off by someone else 
before committing?


-Sandra

Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

A big thanks to everybody working this long Easter weekend who helped
analyze the xz-backdoor and making sure the impact on Sourceware and
the hosted projects was minimal.

This email isn't about the xz-backdoor itself. Do see Sam James FAQ
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
(Sorry for the github link, but this one does seem viewable without
proprietary javascript)

We should discuss what we have been doing and should do more to
mitigate and prevent the next xz-backdoor. There are a couple of
Sourceware services that can help with that.

TLDR;
- Replicatable isolated container/VMs are nice, we want more.
- autoregen buildbots, it should be transparent (and automated) how to
  regenerate build/source files.
- Automate (snapshot) releases tarballs.
- Reproducible releases (from git).

The impact on Sourceware was minimal because we were able to just zap
the affected systems. That is because we started putting everything
into containers and isolated VMs that are easy to reinstall.

In this case we were prepared, but it was also luck that the systems
that were potentially affected were already setup this way. We would
like to be able to do this for any system we deploy. That is why the
Sourceware Project Leadership Committee together with the Software
Freedom Conservancy staff put together "Sourceware 2024 - The Plan"
with as main theme isolation.

https://inbox.sourceware.org/20240325095827.gi5...@gnu.wildebeest.org/

We should be proactive and make sure we are prepared. This will not
just help security of the systems. But by moving services into their
own container or VM will help scaling by making it easy to move
between separate physical machines. And it makes us provide services
that anybody can replicate and setup themselves.

A large part of the xz-backdoor was possible because of social
engineering, which is difficult to solve with technical measures. But
another part was because generation of some of the build/source files
and the release tarball was not transparent and not reproducible.

Having source files or release tarballs being hard to (re)generate
lowers our own trust and the trust that our users have that the
sources in git matches what the released software is build from. It
also makes reviews and automated (pre-commit) CI harder. We have been
working on these issues in two ways, creating automated release
snapshots and the autoregen buildbots.

For projects like gcc, gdb and binutils we have introduced autoregen
buildbots which warn when some generated files are not properly
regenerated from source.
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
This has caught various issues, but isn't complete yet. Christophe
Lyon and Simon Marchi have been working on more generic make
regenerate targets.
https://inbox.sourceware.org/20240313080237.1143034-1-christophe.l...@linaro.org

See also Eric Gallager's discussion on automake make distcheck:
https://lists.gnu.org/archive/html/automake/2024-03/msg7.html

With the help of OSUOSL we have also introduced automated release
snapshots. Projects, like glibc, valgrind, libabigail have fully
automated release tarball (and documentation) generation against
current git. https://snapshots.sourceware.org/

This has caught some issues early which would otherwise only have been
detected at release time and has made sure the generation of other
release artifacts e.g. documentation keeps working. All this is done
in known containers, so also checks that releases can be made with
stable tools.

There are some possible future steps that we can do to make it easier
to check that the release tarballs are created from the actual git
sources. We can ask release managers to check release tarballs they
create against the automated snapshots releases. And/or ask them to
reuse the containers from snapshots or autoregen builders to create
releases. https://sourceware.org/cgit/builder/tree/README_containers

And we should try to work with the reproducible builds project to make
sure that release artifacts can be reproducibly created from the git
sources. https://reproducible-builds.org/ Reproducible builds is also
a Software Freedom Conservancy member project.