I can certainly empathize with the difficulty of maintaining a release verification script and also the difficulty of remembering which combination of environment variables are needed on my system to make it work. The general sentiment of "anybody should be able to check that this release actually works" is a good one, and (at least for R) we have encountered many errors by running tests locally that were never surfaced (and might never have been surfaced) on CI.
I wonder if the difficulty of maintaining the release script is at least partially tied to the fact that we are trying to release so many components simultaneously? Is it time to start releasing components independently? On Tue, Aug 22, 2023 at 10:11 PM Sutou Kouhei <k...@clear-code.com> wrote: > > Hi, > > We can ask the ASF about this. memb...@apache.org? > > > Thanks, > -- > kou > > In <367f6bb7-d8a5-ea49-f5d8-e4fa1afae...@python.org> > "Re: [Discuss] Do we need a release verification script?" on Tue, 22 Aug > 2023 17:18:02 +0200, > Antoine Pitrou <anto...@python.org> wrote: > > > > > And of course this is a bit pedantic, and only important if we want to > > comply with *the letter* of the ASF policies. My own personal opinion > > is that complying in spirit is enough (but I'm also not sure I > > understand the ASF's spirit :-)). > > > > Regards > > > > Antoine. > > > > > > Le 22/08/2023 à 17:10, Antoine Pitrou a écrit : > >> Hmm... perhaps Flatbuffers compilation is usually more deterministic > >> than compiling C++ code into machine code, but that's mainly (AFAIK) > >> because the transformation step is much simpler in the former case, > >> than > >> in the latter. The Flatbuffers compiler also has a range of options > >> that > >> influence code generation, certainly with less variation than a C++ > >> compiler, but still. > >> In other words, I don't think being deterministic is a good criterion > >> to > >> know what "compiled code" means. There is a growing movement towards > >> making generation of machine code artifacts deterministic: > >> https://reproducible-builds.org/ > >> Regards > >> Antoine. > >> Le 22/08/2023 à 16:47, Adam Lippai a écrit : > >>> Compiled code usually means binaries you can’t derive in a > >>> deterministic, > >>> verifiable way from the source code *shipped next to it*. So in this > >>> case > >>> any developer should be able to reproduce the flatbuffers output from > >>> the > >>> release package only. > >>> > >>> “Caches”, multi stage compilation etc should be ok. > >>> > >>> Best regards, > >>> Adam Lippai > >>> > >>> On Tue, Aug 22, 2023 at 10:40 Antoine Pitrou <anto...@python.org> > >>> wrote: > >>> > >>>> > >>>> If the main impetus for the verification script is to comply with ASF > >>>> requirements, probably the script can be made much simpler, such as > >>>> just > >>>> verify the GPG signatures are valid? Or perhaps this can be achieved > >>>> without a script at all. > >>>> > >>>> The irony is that, however complex, our verification script doesn't > >>>> seem > >>>> to check the actual ASF requirements on artifacts. > >>>> > >>>> For example, we don't check that """a source release SHOULD not > >>>> contain > >>>> compiled code""" (also, what does "compiled code" mean? does generated > >>>> code, e.g. by the Flatbuffers compiler, apply?) > >>>> > >>>> Checking that the release """MUST be sufficient for a user to build > >>>> and > >>>> test the release provided they have access to the appropriate platform > >>>> and tools""" is ill-defined and potentially tautologic, because the > >>>> "appropriate platform and tools" is too imprecise and contextual (can > >>>> the "appropriate platform and tools" contain a bunch of proprietary > >>>> software that gets linked with the binaries? Well, it can, otherwise > >>>> you > >>>> can't build on Windows). > >>>> > >>>> Regards > >>>> > >>>> Antoine. > >>>> > >>>> > >>>> > >>>> Le 22/08/2023 à 12:31, Raúl Cumplido a écrit : > >>>>> Hi, > >>>>> > >>>>> I do agree that currently verifying the release locally provides > >>>>> little benefit for the effort we have to put in but I thought this was > >>>>> required as per Apache policy: > >>>>> https://www.apache.org/legal/release-policy.html#release-approval > >>>>> > >>>>> Copying the important bit: > >>>>> """ > >>>>> Before casting +1 binding votes, individuals are REQUIRED to download > >>>>> all signed source code packages onto their own hardware, verify that > >>>>> they meet all requirements of ASF policy on releases as described > >>>>> below, validate all cryptographic signatures, compile as provided, and > >>>>> test the result on their own platform. > >>>>> """ > >>>>> > >>>>> I also think we should try and challenge those. > >>>>> > >>>>> In the past we have identified some minor issues on the local > >>>>> verification but I don't recall any of them being blockers for the > >>>>> release. > >>>>> > >>>>> Thanks, > >>>>> Raúl > >>>>> > >>>>> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>) > >>>> escribió: > >>>>>> > >>>>>> The Rust arrow implementation (arrow-rs) and DataFusion also use > >>>>>> release > >>>>>> verification scripts, mostly inherited from when they were split from > >>>> the > >>>>>> mono repo. They have found issues from time to time, for us, but those > >>>>>> issues are often not platform related and have not been release > >>>> blockers. > >>>>>> > >>>>>> Thankfully for Rust, the verification scripts don't need much > >>>> maintenance > >>>>>> so we just continue the ceremony. However, I certainly don't think we > >>>> would > >>>>>> lose much/any test coverage if we stopped their use. > >>>>>> > >>>>>> Andrew > >>>>>> > >>>>>> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <anto...@python.org> > >>>> wrote: > >>>>>> > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> Abiding by the Apache Software Foundation's guidelines, every Arrow > >>>>>>> release is voted on and requires at least 3 "binding" votes to be > >>>> approved. > >>>>>>> > >>>>>>> Also, every Arrow release vote is accompanied by a little ceremonial > >>>>>>> where contributors and core developers run a release verification > >>>> script > >>>>>>> on their machine, wait for long minutes (sometimes an hour) and report > >>>>>>> the results. > >>>>>>> > >>>>>>> This ceremonial has gone on for years, and it has not really been > >>>>>>> questioned. Yet, it's not obvious to me what it is achieving exactly. > >>>>>>> I've been here since 2018, but I don't really understand what the > >>>>>>> verification script is testing for, or, more importantly, *why* it is > >>>>>>> testing for what it is testing. I'm probably not the only one? > >>>>>>> > >>>>>>> I would like to bring the following points: > >>>>>>> > >>>>>>> * platform compatibility is (supposed to be) exercised on Continuous > >>>>>>> Integration; there is no understandable reason why it should be > >>>>>>> ceremoniously tested on each developer's machine before the release > >>>>>>> > >>>>>>> * just before a release is probably the wrong time to be testing > >>>>>>> platform compatibility, and fixing compatibility bugs (though, of > >>>>>>> course, it might still be better than not noticing?) > >>>>>>> > >>>>>>> * home environments are unstable, and not all developers run the > >>>>>>> verification script for each release, so each release is actually > >>>>>>> verified on different, uncontrolled, platforms > >>>>>>> > >>>>>>> * as for sanity checks on binary packages, GPG signatures, etc., there > >>>>>>> shouldn't be any need to run them on multiple different machines, as > >>>>>>> they are (should be?) entirely deterministic and platform-agnostic > >>>>>>> > >>>>>>> * maintaining the verification scripts is a thankless task, in part > >>>>>>> due > >>>>>>> to their nature (they need to track and mirror changes made in each > >>>>>>> implementation's build chain), in part due to implementation choices > >>>>>>> > >>>>>>> * due to the existence of the verification scripts, the release vote > >>>>>>> is > >>>>>>> focussed on getting the script to run successfully (a very contextual > >>>>>>> and non-reproducible result), rather than the actual *contents* of the > >>>>>>> release > >>>>>>> > >>>>>>> The most positive thing I can personally say about the verification > >>>>>>> scripts is that they *may* help us trust the release is not broken? > >>>>>>> But > >>>>>>> that's a very unqualified statement, and is very close to > >>>> cargo-culting. > >>>>>>> > >>>>>>> Regards > >>>>>>> > >>>>>>> Antoine. > >>>>>>> > >>>> > >>>
