Looks like this thread is touching a few different issues: - Process documentation: I was trying to learn the details behind the automation, release signatures, etc in the Spark release management official documentation (http://spark.apache.org/release-process.html) , and it looks like not much is described there.
- Sharing release keys: As described in the Apache Release Creation Process (http://www.apache.org/dev/release-publishing.html#signed) it is recommended that, "If you plan to serve as a release manager, you should generate a key and publish it well in advance of creating a release." which clearly recommends an individual key per RM. If the keys are going to be shared (which I don't recommend) it should at least be an "Apache Spark" key, instead of an individual person key. IMHO sharing a key, particularly when this key is available in a non-apache managed infrastructure, makes it much more susceptible to become compromised, particularly because the PMC does not control who has access to the environment. - Inability to customize the release automation by the RM: Looks like the Jenkins jobs that are responsible for automating the spark releases are created/updated by private scripts that are not available to all Spark PMC and/or RMs, this also makes the ability to improve the release process a lot more complicated. Would the Spark PMC please look into addressing these issues asap. Thanks On Sun, Sep 17, 2017 at 10:12 PM, Patrick Wendell <patr...@databricks.com> wrote: > Sparks release pipeline is automated and part of that automation includes > securely injecting this key for the purpose of signing. I asked the ASF to > provide a service account key several years ago but they suggested that we > use a key attributed to an individual even if the process is automated. > > I believe other projects that release with high frequency also have > automated the signing process. > > This key is injected during the build process. A really ambitious release > manager could reverse engineer this in a way that reveals the private key, > however if someone is a release manager then they themselves can do quite a > bit of nefarious things anyways. > > It is true that we trust all previous release managers instead of only > one. We could probably rotate the jenkins credentials periodically in order > to compensate for this, if we think this is a nontrivial risk. > > - Patrick > > On Sun, Sep 17, 2017 at 7:04 PM, Holden Karau <hol...@pigscanfly.ca> > wrote: > >> Would any of Patrick/Josh/Shane (or other PMC folks with >> understanding/opinions on this setup) care to comment? If this is a >> blocking issue I can cancel the current release vote thread while we >> discuss this some more. >> >> On Fri, Sep 15, 2017 at 5:18 PM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> Oh yes and to keep people more informed I've been updating a PR for the >>> release documentation as I go to write down some of this unwritten >>> knowledge -- https://github.com/apache/spark-website/pull/66 >>> >>> >>> On Fri, Sep 15, 2017 at 5:12 PM Holden Karau <hol...@pigscanfly.ca> >>> wrote: >>> >>>> Also continuing the discussion from the vote threads, Shane probably >>>> has the best idea on the ACLs for Jenkins so I've CC'd him as well. >>>> >>>> >>>> On Fri, Sep 15, 2017 at 5:09 PM Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> Changing the release jobs, beyond the available parameters, right now >>>>> depends on Josh arisen as there are some scripts which generate the jobs >>>>> which aren't public. I've done temporary fixes in the past with the Python >>>>> packaging but my understanding is that in the medium term it requires >>>>> access to the scripts. >>>>> >>>>> So +CC Josh. >>>>> >>>>> On Fri, Sep 15, 2017 at 4:38 PM Ryan Blue <rb...@netflix.com> wrote: >>>>> >>>>>> I think this needs to be fixed. It's true that there are barriers to >>>>>> publication, but the signature is what we use to authenticate Apache >>>>>> releases. >>>>>> >>>>>> If Patrick's key is available on Jenkins for any Spark committer to >>>>>> use, then the chance of a compromise are much higher than for a normal RM >>>>>> key. >>>>>> >>>>>> rb >>>>>> >>>>>> On Fri, Sep 15, 2017 at 12:34 PM, Sean Owen <so...@cloudera.com> >>>>>> wrote: >>>>>> >>>>>>> Yeah I had meant to ask about that in the past. While I presume >>>>>>> Patrick consents to this and all that, it does mean that anyone with >>>>>>> access >>>>>>> to said Jenkins scripts can create a signed Spark release, regardless of >>>>>>> who they are. >>>>>>> >>>>>>> I haven't thought through whether that's a theoretical issue we can >>>>>>> ignore or something we need to fix up. For example you can't get a >>>>>>> release >>>>>>> on the ASF mirrors without more authentication. >>>>>>> >>>>>>> How hard would it be to make the script take in a key? it sort of >>>>>>> looks like the script already takes GPG_KEY, but don't know how to >>>>>>> modify >>>>>>> the jobs. I suppose it would be ideal, in any event, for the actual >>>>>>> release >>>>>>> manager to sign. >>>>>>> >>>>>>> On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <hol...@pigscanfly.ca> >>>>>>> wrote: >>>>>>> >>>>>>>> That's a good question, I built the release candidate however the >>>>>>>> Jenkins scripts don't take a parameter for configuring who signs them >>>>>>>> rather it always signs them with Patrick's key. You can see this from >>>>>>>> previous releases which were managed by other folks but still signed by >>>>>>>> Patrick. >>>>>>>> >>>>>>>> On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <rb...@netflix.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> The signature is valid, but why was the release signed with >>>>>>>>> Patrick Wendell's private key? Did Patrick build the release >>>>>>>>> candidate? >>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Software Engineer >>>>>> Netflix >>>>>> >>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> >>>> -- >>>> Twitter: https://twitter.com/holdenkarau >>>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> >> -- >> Twitter: https://twitter.com/holdenkarau >> > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
