Another option is I can just run the build locally, this might be better
approach since it will help make sure we have the dependencies documented
for the eventual transition to dockerized builds?
On Tue, Sep 19, 2017 at 9:53 AM, Holden Karau wrote:
> Thanks for the reminder :)
>
> On Tue, Sep 1
Thanks for the reminder :)
On Tue, Sep 19, 2017 at 9:02 AM Luciano Resende
wrote:
> Manually signing seems a good compromise for now, but note that there are
> two places that this needs to happen, the artifacts that goes to dist.a.o
> as well as the ones that are published to maven.
>
> On Tue,
Manually signing seems a good compromise for now, but note that there are
two places that this needs to happen, the artifacts that goes to dist.a.o
as well as the ones that are published to maven.
On Tue, Sep 19, 2017 at 8:53 AM, Ryan Blue
wrote:
> +1. Thanks for coming up with a solution, every
+1. Thanks for coming up with a solution, everyone! I think the manually
signed RC as a work around will work well, and it will be an improvement
for the rest to be updated.
On Mon, Sep 18, 2017 at 8:25 PM, Patrick Wendell
wrote:
> Sounds good - thanks Holden!
>
> On Mon, Sep 18, 2017 at 8:21 PM
Sounds good - thanks Holden!
On Mon, Sep 18, 2017 at 8:21 PM, Holden Karau wrote:
> That sounds like a pretty good temporary work around if folks agree I'll
> cancel release vote for 2.1.2 and work on getting an RC2 out later this
> week manually signed. I've filed JIRA SPARK-22055 & SPARK-22054
That sounds like a pretty good temporary work around if folks agree I'll
cancel release vote for 2.1.2 and work on getting an RC2 out later this
week manually signed. I've filed JIRA SPARK-22055 & SPARK-22054 to port the
release scripts and allow injecting of the RM's key.
On Mon, Sep 18, 2017 at
For the current release - maybe Holden could just sign the artifacts with
her own key manually, if this is a concern. I don't think that would
require modifying the release pipeline, except to just remove/ignore the
existing signatures.
- Patrick
On Mon, Sep 18, 2017 at 7:56 PM, Reynold Xin wrot
Does anybody know whether this is a hard blocker? If it is not, we should
probably push 2.1.2 forward quickly and do the infrastructure improvement
in parallel.
On Mon, Sep 18, 2017 at 7:49 PM, Holden Karau wrote:
> I'm more than willing to help migrate the scripts as part of either this
> relea
I'm more than willing to help migrate the scripts as part of either this
release or the next.
It sounds like there is a consensus developing around changing the process
-- should we hold off on the 2.1.2 release or roll this into the next one?
On Mon, Sep 18, 2017 at 7:37 PM, Marcelo Vanzin wrot
+1 to this. There should be a script in the Spark repo that has all
the logic needed for a release. That script should take the RM's key
as a parameter.
if there's a desire to keep the current Jenkins job to create the
release, it should be based on that script. But from what I'm seeing
there are
Hey I talked more with Josh Rosen about this who has helped with automation
since I became less involved in release management.
I can think of a few different things that would improve our RM based on
these suggestions:
(1) We could remove signing step from the rest of the automation and as the
R
i will detail how we control access to the jenkins infra tomorrow.
we're pretty well locked down, but there is absolutely room for
improvement.
this thread is also a good reminder that we (RMs + pwendell + ?)
should audit who still has, but does not need direct (or special)
access to jenkins.
reg
One thing we could do is modify the release tooling to allow the key to be
injected each time, thus allowing any RM to insert their own key at build
time.
Patrick
On Mon, Sep 18, 2017 at 4:56 PM Ryan Blue wrote:
> I don't understand why it is necessary to share a release key. If this is
> somet
I don't understand why it is necessary to share a release key. If this is
something that can be automated in a Jenkins job, then can it be a script
with a reasonable set of build requirements for Mac and Ubuntu? That's the
approach I've seen the most in other projects.
I'm also not just concerned
Looks like this thread is touching a few different issues:
- Process documentation: I was trying to learn the details behind the
automation, release signatures, etc in the Spark release management
official documentation (http://spark.apache.org/release-process.html) , and
it looks like not much is
Sparks release pipeline is automated and part of that automation includes
securely injecting this key for the purpose of signing. I asked the ASF to
provide a service account key several years ago but they suggested that we
use a key attributed to an individual even if the process is automated.
I
Would any of Patrick/Josh/Shane (or other PMC folks with
understanding/opinions on this setup) care to comment? If this is a
blocking issue I can cancel the current release vote thread while we
discuss this some more.
On Fri, Sep 15, 2017 at 5:18 PM Holden Karau wrote:
> Oh yes and to keep peopl
Oh yes and to keep people more informed I've been updating a PR for the
release documentation as I go to write down some of this unwritten
knowledge -- https://github.com/apache/spark-website/pull/66
On Fri, Sep 15, 2017 at 5:12 PM Holden Karau wrote:
> Also continuing the discussion from the v
Also continuing the discussion from the vote threads, Shane probably has
the best idea on the ACLs for Jenkins so I've CC'd him as well.
On Fri, Sep 15, 2017 at 5:09 PM Holden Karau wrote:
> Changing the release jobs, beyond the available parameters, right now
> depends on Josh arisen as there
Changing the release jobs, beyond the available parameters, right now
depends on Josh arisen as there are some scripts which generate the jobs
which aren't public. I've done temporary fixes in the past with the Python
packaging but my understanding is that in the medium term it requires
access to t
I think this needs to be fixed. It's true that there are barriers to
publication, but the signature is what we use to authenticate Apache
releases.
If Patrick's key is available on Jenkins for any Spark committer to use,
then the chance of a compromise are much higher than for a normal RM key.
rb
Yeah I had meant to ask about that in the past. While I presume Patrick
consents to this and all that, it does mean that anyone with access to said
Jenkins scripts can create a signed Spark release, regardless of who they
are.
I haven't thought through whether that's a theoretical issue we can ign
22 matches
Mail list logo