> That is an unfair characterization of what the concern is here. And > what is a priority for individual PMCs should be left to them to > decide.
I would say it's one-sided, maybe, but not necessarily unfair. That's just my point of view :). I am not the decision-maker here—certainly not alone. And yes - I think some level of decision should be left to PMCs. However, the release process is generally one of the best described and least flexible processes in the ASF for a reason: it is essentially a legal act of the Foundation and carries certain risks for the Foundation. This is basically why INFRA has the upper hand there - delegated by either the board or the president (I can't recall precisely)—to strictly regulate that process. So my PMC - for example - do pretty much what I've been told we should do there - automating what can be automated, but other than that it has to be 100% compliant with https://www.apache.org/legal/release-policy.html that delegates to a inumer of other documents and has MUST in quite a few places. > At the moment we are in a catch 22 - the very things we need access to on shared infrastructure (eg. Apple's signing and packaging tools) are the very things that stop the process from being reproducible. For the last two releases, the only installers for NetBeans have been produced, signed and distributed by me at my own expense. We might get another provider soon. But we've also had questions within ASF, including from the board, on why we've dropped provision of installers directly, and the security and other ramifications of relying on "third-parties" for this. If we are ever to bring back installers directly at ASF, we need to find the best answer here, not the perfect one. Sure—so now I better understand: what you are **really** looking for is not automating the release process on Infra hardware, but strictly automating just the signing process. This means that, according to policy, such a signed package must be "Verified" in local hardware owned by the release manager. This might - or might not - mean that the whole release process is done in infra - it might for example mean that there is a service (very well contrilled by Infra) where you upload binary (which otherwise you prepared locally and your release process is done entirely by release manager on their machine. So, perhaps you need a remote signing service managed by Infra, not a release process on Infra machines. I did similar things years ago. One time I designed and implemented the entire process for building and signing the whole Android OS from sources for a company that produced Android-based payment terminals. One step required the person signing to enter a Faraday cage, where 3 people had to assemble 3 keys so that person could enter and sign the OS with the company's secret key. And the process included verification signatures of all commits that customized the OS - but it was also a "small" part of the whole release process (but yeah - definitely most painful and we had to be **really** sure that what will be signed was working. So I guess we are discussing a similar thing (but likely with **slightly** :) less security constraints - maybe without the Faraday Cage ;) ). In that case, I will stop talking, because I was discussing the entire release process, the reasoning for reproducibility and voting, while you seem focused only on packaging and signing. We often conflate some of our processes, so I think it's good to clarify. Since I know next to nothing about macOS and Windows signing and I have no insight into what Infra can do about it (nor do I have any power there), I will stop talking. I hope the discussion helps others who have the authority and knowledge regarding what can or cannot be done. J. On Sun, Mar 22, 2026 at 1:09 PM Neil C Smith <[email protected]> wrote: > Hi, > > On Sat, 21 Mar 2026 at 18:52, Jarek Potiuk <[email protected]> wrote: > > if you > > can write a script (purely sources you can read) and have it released > > together in the sources, and if yoy can run the script on two files and > > that script shows and explains all the differences when you run the same > > package locally and in the cloud, then this meets **my** definition of > > "reproducible-enough". > > If you mean a pure sources script that replicates the process in GHA, > then that's easy - that's what the repo is (linked in my first post). > The workflow sets up the certificates and environment, then runs a > single Java source file that does the verification and packaging. > This is designed to be runnable locally. > > If you mean a script to verify where the differences are, then while > doable, I think this is better verified in other ways. For a start > it's all OS-specific, with at least one OS probably requiring running > the installation, and the differences will differ depending on code > signing access. We would have 3 things to check against - original > binary zip payload, local package without code signing, and GHA > package with code signing. > > > Also the question is where the input files are > > downloaded from. I asume from SVN of ours, and there is similar > > reproducible check before to produce those. :). > > The packager is released and downloaded from Maven Central. As far as > I'm aware, it is reproducible. > > NetBeans itself is not reproducible. It probably never will be. It's > a massive project with 30 years of technical debt, and a few > questionable build decisions. My current external workflow (and this > might change if we replicate internally) downloads from nightlies.a.o > for weekly release candidates, and from dlcdn.a.o for the actual > release. Signed release candidates are actually an essential part of > testing. > > NetBeans releases its sources (obviously!), along with a binary zip. > Derived from the binaries in that zip are hundreds of Maven artefacts > and any installer packages. Provenance back to the binaries in that > zip is important, and is all we have for verification purposes. > > But as far as I'm concerned, the payload isn't relevant here. We've > been code signing installers that bundle the released zip for years. > Moving the signing on to GHA, while verifying to the maximum extent > feasible that it is actually bundling that zip, does not change > anything fundamental about what we're doing. If distributing > non-reproducible binaries in itself becomes the issue, then you might > as well tell NetBeans to shut up shop. > > > In short for me, the goal of building things in GH is not to **reduce the > > load**—in fact, I think proper security for a release introduces some > > healthy friction and requires effort. It's deliberately designed to make > > people think, pause and actually verify that what they are doing is good > > and secure. Rather than striving to spend as little effort as possible > ... > > That is an unfair characterization of what the concern is here. And > what is a priority for individual PMCs should be left to them to > decide. > > What we require most is to reduce release overhead by not relying too > much on individuals, and finding ways to share access to these > resources. We need to be able to share access to code signing, > including with members of the PMC who may not be in a position to run > things locally. > > I have been delivering installers for NetBeans+JDK externally for five > years now. These are the only installers available right now. I was > the only person on our PMC capable of building ASF macOS installers > for years until we dropped them last year, because we had no way of > sharing access to the foundation's Apple certificate. Leaving aside > the issue with finding the next person willing to be lumbered with the > time consuming job even if we did - I don't even use macOS! :-) > > At the moment we are in a catch 22 - the very things we need access to > on shared infrastructure (eg. Apple's signing and packaging tools) are > the very things that stop the process from being reproducible. For > the last two releases, the only installers for NetBeans have been > produced, signed and distributed by me at my own expense. We might > get another provider soon. But we've also had questions within ASF, > including from the board, on why we've dropped provision of installers > directly, and the security and other ramifications of relying on > "third-parties" for this. If we are ever to bring back installers > directly at ASF, we need to find the best answer here, not the perfect > one. > > Thanks, > > Neil > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: > [email protected] > >
