I don't understand why it is necessary to share a release key. If this is something that can be automated in a Jenkins job, then can it be a script with a reasonable set of build requirements for Mac and Ubuntu? That's the approach I've seen the most in other projects.
I'm also not just concerned about release managers. Having a key stored persistently on outside infrastructure adds the most risk, as Luciano noted as well. We should also start publishing checksums in the Spark VOTE thread, which are currently missing. The risk I'm concerned about is that if the key were compromised, it would be possible to replace binaries with perfectly valid ones, at least on some mirrors. If the Apache copy were replaced, then we wouldn't even be able to catch that it had happened. Given the high profile of Spark and the number of companies that run it, I think we need to take extra care to make sure that can't happen, even if it is an annoyance for the release managers. On Sun, Sep 17, 2017 at 10:12 PM, Patrick Wendell <patr...@databricks.com> wrote: > Sparks release pipeline is automated and part of that automation includes > securely injecting this key for the purpose of signing. I asked the ASF to > provide a service account key several years ago but they suggested that we > use a key attributed to an individual even if the process is automated. > > I believe other projects that release with high frequency also have > automated the signing process. > > This key is injected during the build process. A really ambitious release > manager could reverse engineer this in a way that reveals the private key, > however if someone is a release manager then they themselves can do quite a > bit of nefarious things anyways. > > It is true that we trust all previous release managers instead of only > one. We could probably rotate the jenkins credentials periodically in order > to compensate for this, if we think this is a nontrivial risk. > > - Patrick > > On Sun, Sep 17, 2017 at 7:04 PM, Holden Karau <hol...@pigscanfly.ca> > wrote: > >> Would any of Patrick/Josh/Shane (or other PMC folks with >> understanding/opinions on this setup) care to comment? If this is a >> blocking issue I can cancel the current release vote thread while we >> discuss this some more. >> >> On Fri, Sep 15, 2017 at 5:18 PM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> Oh yes and to keep people more informed I've been updating a PR for the >>> release documentation as I go to write down some of this unwritten >>> knowledge -- https://github.com/apache/spark-website/pull/66 >>> >>> >>> On Fri, Sep 15, 2017 at 5:12 PM Holden Karau <hol...@pigscanfly.ca> >>> wrote: >>> >>>> Also continuing the discussion from the vote threads, Shane probably >>>> has the best idea on the ACLs for Jenkins so I've CC'd him as well. >>>> >>>> >>>> On Fri, Sep 15, 2017 at 5:09 PM Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> Changing the release jobs, beyond the available parameters, right now >>>>> depends on Josh arisen as there are some scripts which generate the jobs >>>>> which aren't public. I've done temporary fixes in the past with the Python >>>>> packaging but my understanding is that in the medium term it requires >>>>> access to the scripts. >>>>> >>>>> So +CC Josh. >>>>> >>>>> On Fri, Sep 15, 2017 at 4:38 PM Ryan Blue <rb...@netflix.com> wrote: >>>>> >>>>>> I think this needs to be fixed. It's true that there are barriers to >>>>>> publication, but the signature is what we use to authenticate Apache >>>>>> releases. >>>>>> >>>>>> If Patrick's key is available on Jenkins for any Spark committer to >>>>>> use, then the chance of a compromise are much higher than for a normal RM >>>>>> key. >>>>>> >>>>>> rb >>>>>> >>>>>> On Fri, Sep 15, 2017 at 12:34 PM, Sean Owen <so...@cloudera.com> >>>>>> wrote: >>>>>> >>>>>>> Yeah I had meant to ask about that in the past. While I presume >>>>>>> Patrick consents to this and all that, it does mean that anyone with >>>>>>> access >>>>>>> to said Jenkins scripts can create a signed Spark release, regardless of >>>>>>> who they are. >>>>>>> >>>>>>> I haven't thought through whether that's a theoretical issue we can >>>>>>> ignore or something we need to fix up. For example you can't get a >>>>>>> release >>>>>>> on the ASF mirrors without more authentication. >>>>>>> >>>>>>> How hard would it be to make the script take in a key? it sort of >>>>>>> looks like the script already takes GPG_KEY, but don't know how to >>>>>>> modify >>>>>>> the jobs. I suppose it would be ideal, in any event, for the actual >>>>>>> release >>>>>>> manager to sign. >>>>>>> >>>>>>> On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <hol...@pigscanfly.ca> >>>>>>> wrote: >>>>>>> >>>>>>>> That's a good question, I built the release candidate however the >>>>>>>> Jenkins scripts don't take a parameter for configuring who signs them >>>>>>>> rather it always signs them with Patrick's key. You can see this from >>>>>>>> previous releases which were managed by other folks but still signed by >>>>>>>> Patrick. >>>>>>>> >>>>>>>> On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <rb...@netflix.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> The signature is valid, but why was the release signed with >>>>>>>>> Patrick Wendell's private key? Did Patrick build the release >>>>>>>>> candidate? >>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Software Engineer >>>>>> Netflix >>>>>> >>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> >>>> -- >>>> Twitter: https://twitter.com/holdenkarau >>>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> >> -- >> Twitter: https://twitter.com/holdenkarau >> > > -- Ryan Blue Software Engineer Netflix
