A big thanks to everybody working this long Easter weekend who helped analyze the xz-backdoor and making sure the impact on Sourceware and the hosted projects was minimal.
This email isn't about the xz-backdoor itself. Do see Sam James FAQ https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 (Sorry for the github link, but this one does seem viewable without proprietary javascript) We should discuss what we have been doing and should do more to mitigate and prevent the next xz-backdoor. There are a couple of Sourceware services that can help with that. TLDR; - Replicatable isolated container/VMs are nice, we want more. - autoregen buildbots, it should be transparent (and automated) how to regenerate build/source files. - Automate (snapshot) releases tarballs. - Reproducible releases (from git). The impact on Sourceware was minimal because we were able to just zap the affected systems. That is because we started putting everything into containers and isolated VMs that are easy to reinstall. In this case we were prepared, but it was also luck that the systems that were potentially affected were already setup this way. We would like to be able to do this for any system we deploy. That is why the Sourceware Project Leadership Committee together with the Software Freedom Conservancy staff put together "Sourceware 2024 - The Plan" with as main theme isolation. https://inbox.sourceware.org/20240325095827.gi5...@gnu.wildebeest.org/ We should be proactive and make sure we are prepared. This will not just help security of the systems. But by moving services into their own container or VM will help scaling by making it easy to move between separate physical machines. And it makes us provide services that anybody can replicate and setup themselves. A large part of the xz-backdoor was possible because of social engineering, which is difficult to solve with technical measures. But another part was because generation of some of the build/source files and the release tarball was not transparent and not reproducible. Having source files or release tarballs being hard to (re)generate lowers our own trust and the trust that our users have that the sources in git matches what the released software is build from. It also makes reviews and automated (pre-commit) CI harder. We have been working on these issues in two ways, creating automated release snapshots and the autoregen buildbots. For projects like gcc, gdb and binutils we have introduced autoregen buildbots which warn when some generated files are not properly regenerated from source. https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen This has caught various issues, but isn't complete yet. Christophe Lyon and Simon Marchi have been working on more generic make regenerate targets. https://inbox.sourceware.org/20240313080237.1143034-1-christophe.l...@linaro.org See also Eric Gallager's discussion on automake make distcheck: https://lists.gnu.org/archive/html/automake/2024-03/msg00007.html With the help of OSUOSL we have also introduced automated release snapshots. Projects, like glibc, valgrind, libabigail have fully automated release tarball (and documentation) generation against current git. https://snapshots.sourceware.org/ This has caught some issues early which would otherwise only have been detected at release time and has made sure the generation of other release artifacts e.g. documentation keeps working. All this is done in known containers, so also checks that releases can be made with stable tools. There are some possible future steps that we can do to make it easier to check that the release tarballs are created from the actual git sources. We can ask release managers to check release tarballs they create against the automated snapshots releases. And/or ask them to reuse the containers from snapshots or autoregen builders to create releases. https://sourceware.org/cgit/builder/tree/README_containers And we should try to work with the reproducible builds project to make sure that release artifacts can be reproducibly created from the git sources. https://reproducible-builds.org/ Reproducible builds is also a Software Freedom Conservancy member project.
