This is a complex subject unfortunately. There are several aspects, but I think it is really comes down to trust—whether we trust that the GH actions runner is not going to be compromised.
SLSA attestations are only as good as the SLSA attestations of **everything** you use to build your artifacts (build environment, dependencies, build tools, hardware). Essentially, they place trust in whatever other attestation you have, and you cannot achieve a higher attestation level than any of your "transient" attestations (which requires you to trust whoever provides them). But yes they are better than not having them, they are not a (direct) replacement for reproducibility checks. I think the current requirement for "reproducibility" is a pragmatic choice made in the past. We believe the highest risk in the build process is compromising the build system on which the build runs; this approach essentially ignores the risk that any of our dependencies (images, build tools, OS dependencies) have been compromised. So it addressed a very "narrow" aspect of the build - but one that is most within our control—because we need to trust other parties for pretty much everything else anyway (unless we follow "turtles all the way down" and build everything from sources - including firmware on the devices, and have a way to verify that the hardware has no backdoors built in by the hardware producer). But .. let's just discard those things, as they are not the risks either "local" or "GitHub Actions" builds. Whether we build locally or in GH, those risks remain very similar and, most importantly, are totally outside of our control. Security is rarely a zero-or-one thing; it's almost always about acceptable risk and the scenarios we want to protect against. I think attestations provided with a remote, inaccessible (by the release manager) build system inherently differ from reproducible builds and protect against a bit different scenarios. One thing they do indeed - they do prevent the risk from a rogue release manager. But they also shift trust to GH making it more difficult to trace the sources of compromise when external attackers breach them - or even to know that a breach occurred. The latter point is crucial: if your GH runner is compromised, you will only find out after the release. If the machine used by the release manager is compromised and you have reproducible builds, you will know before the release. I think for me the last point is really important, and it's the reason why dropping reproducible builds and switching to GH makes it a bit more dangerous. GH actions do not provide the same level of security or address exactly the same risk as reproducibility, mostly because compromising the single system that provides attestations invalidates the attestation for that build. Reproducibility addresses this by requiring you to compromise few such systems - potentially largely independent ones—which makes it exponentially harder to pull of - unless you are able to hack-in the tools (IDEs, AI CLIS etc.) that multiple people are likely to use (and remote build system doesn't need). One thing it provides that GH does not (regardless of attestations) is the ability to repeat the process in the future, on a new machine with different hardware, comparing the results and **verifying** if the original build system has been compromised. Build attestations cannot really provide this—even if you keep build logs and store them securely in a ledger, the assurance is only as good as the security of that single build system. If this system is compromised, the attestation produced is essentially worthless. And again - you will only find out after some users discover (the hard way) that there is no way to check if that single GH runner environment was compromised before release. Also, trading reproducibility for GH builds loses the "future verification" option. If we find malware in the artifact built yesterday on an ephemeral GH machine, we have virtually no way to tell if the build system was compromised during the build. We have no way to narrow down the potential source of malware and are at a loss there. Reproducibility gives you the ability to run the process on clean hardware or by a different person and verify that it indeed produces different, unexplainable results—with the same dependencies. This allows you to narrow the issue down to a compromise in the build system where it was originally created. For example, this was the case with XZ: an attacker who controlled the hardware used an untraced injection to add test artifacts (present in the sources) to the production artifact during the build. This was demonstrably verifiable because preparing the artifact on someone else's machine revealed a difference. If XZ required a reproducibility check by a different person (like we do in Airflow for example), this particular attack would not be possible. It was verifiable at the artifact level, not just the source level. With GitHub Actions, this single release manager also could not do it (the GH build was done well) but if someone who hacks the GitHub runner could, and it would be far more difficult to track. Of course, if you implement it properly in GH Actions, a single person (the release manager) might not be able to inject anything into the build process that is not in the repo. While you achieve a similar goal, you essentially introduce an SPOF (Single Point of Failure)—you still have to trust that GH sufficiently protects the build system against both external and internal attacks. Reproducibility achieves this without such an implicit single point of failure and provides better traceability. So, we essentially shift the risk to GH, introduce a single point of failure (SPOF), and make it more difficult or even impossible to trace the cause. We also accept the risk that if a breach occurs, we will not discover it before release—all in exchange for trusting GH to secure the runners more than several of our PMC members. That might be a reasonable trade-off if we choose to do so. But it's not either-or; you do not have to have bit-to-bit reproducibility. People often mistake those two. I don't even think this is what INFRA expects. As long as whatever you produce has "explainable differences," this is still reproducibility and serves the same purpose as bit-to-bit. Bit-for-bit verification is easy, but if you can demonstrably show and verify that the difference between two artifacts you produce comes only from metadata (timestamps, file permissions, and so on) and not the code - it's still reproducibility. This is actually the best of both worlds: when you can provide GH "controlled environment" and some kind of reproducibility, you have all the benefits of both. So maybe that is worth exploring? J. On Thu, Mar 19, 2026 at 4:45 PM Neil C Smith <[email protected]> wrote: > Hi Piotr, > > Thanks for your feedback - few things inline ... > > On Wed, 18 Mar 2026 at 21:53, Piotr P. Karwasz > <[email protected]> wrote: > > However, I > > think that with the current state of the art, we have reached something > > of a paradox. > > > > At present, we tend to prefer releases built on committer-owned machines > > over those produced by GitHub Actions, even though: > > > > - GitHub Actions builds run in clean, ephemeral environments (Docker > > images), > > > > - while committer-owned machines may contain residual artifacts from > > previous builds and dirty cached dependencies. > > > > From this perspective, CI-based builds can in practice offer a more > > controlled and reproducible environment than local builds. > > This! And for example the bit on the SLSA site about build isolation > strength. There may be as much of an argument that only reproducible > builds should be allowed on committer-owned machines. As someone who > has code-signed numerous releases locally, the only guarantee that > they are what they say they are is my signature as release manager, > and the hope that my machines are correctly configured and I've not > made any mistakes. :-) At least the CI offers another more detailed > trail that the PMC can validate against. > > > In the case of NetBeans, I believe one way to address the trust concerns > > would be to generate SLSA build attestations as part of the CI process. > > The Release Manager could then verify these attestations prior to > > initiating a release vote. The SLSA specification already defines a > > verification procedure for this purpose: > > I was vaguely aware of SLSA, but hadn't really considered for this > purpose (ie. provenance for voting purposes). Generating these might > offer a more useful format for this rather than relying on the whole > log. Although I don't know if it would include anything not already > output? The log contains other build information that would be useful > to persist anyway. > > Do we have ASF examples / a process for signing SLSA provenance > file(s) during the run, independently of the release manager doing so > afterwards? > > I would expect the link to the particular build run to be included in > the vote email - everyone voting should then be able to verify any > attached attestations, logs, etc. completely independently of the RM. > > This actually also raises another thought. We used to vote on > installers as part of our main release vote. My current workflow > builds from the released binary zip after the release vote. For a > useful provenance here, we might need to have a two-step voting > process, or we might need a permalink for input binaries that persists > through voting, release and archiving. > > Thanks and best wishes, > > Neil > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: > [email protected] > >
