When to cut RCs

[Sean Owen]

Moving this to dev --

That's good, that'll help. Technically there's still a Blocker bug:
https://issues.apache.org/jira/browse/SPARK-12000

The 'race' doesn't matter that much, but release planning remains the
real bug-bear here. There are still, for instance, 52 issues targeted
at 1.6.0, 42 of which were raised and targeted by committers. For
example, I count 6 'umbrella' JIRAs in ML alone that are still open.
The release is theoretically several weeks behind plan on what's
intended to be a fixed release cycle too. This is why I'm not sure why
today it's suddenly potentially ready for release.

I'm just curious, am I the only one that thinks this isn't roughly
normal, or do other people manage releases this way? I know the real
world is messy and this is better than in the past, but I still get
surprised by how each 1.x release actually comes about.

On Wed, Dec 2, 2015 at 8:12 PM, marmbrus  wrote:
> Github user marmbrus commented on the pull request:
>
> https://github.com/apache/spark/pull/10108#issuecomment-161420041
>
> Sorry if this seemed arbitrary, I cut the release as soon as all the 
> blocker issues were resolve.  In the future I'll announce on the dev list 
> first.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
> -
> To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
> For additional commands, e-mail: reviews-h...@spark.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: When to cut RCs

[Michael Armbrust]

>
> Sorry for a second email so soon. I meant to also ask, what keeps the cost
> of making an RC high? Can we bring it down with better tooling?
>

There is a lot of tooling:
https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/

Still you have check JIRA, sync with people who have been working on known
issues, run the jenkins jobs (which take 1+ hours) and then write that
email which has a bunch of links in it.  Short of automating the creation
of the email (PRs welcome!) I'm not sure what else you would automate.
That said, this is all I have done since I came into work today.

Re: When to cut RCs

[Sean Owen]

On Wed, Dec 2, 2015 at 9:06 PM, Michael Armbrust  wrote:
> This can be debated, but I explicitly ignored test and documentation issues.
> Since the docs are published separately and easy to update, I don't think
> its worth further disturbing the release cadence for these JIRAs.

It makes sense to not hold up an RC since they don't affect testing of
functionality. Prior releases have ultimately gone out with doc issues
still outstanding (and bugs) though. This doesn't seem to be on
anyone's release checklist, and maybe part of it is because they're
let slide for RCs.  Your suggestion to check-point release status
below sounds spot on; I sort of tried to do that earlier.


> Up until today various committers have told me that there were known issues
> with branch-1.6 that would cause them to -1 the release.  Whenever this
> happened, I asked them to ensure there was a properly targeted blocker JIRA
> open so people could publicly track the status of the release.  As long as
> such issues were open, I only published a preview since making an RC is
> pretty high cost.

Makes sense if these are all getting translated into Blockers and
resolved before an RC. It's the simplest mechanism to communicate and
track this in a distributed way.

"No blockers" is a minimal criterion for release. It still seems funny
to release with so many issues targeted for 1.6.0, including issues
that aren't critical or bugs. Sure, that's just hygiene. But without
it, do people take "Target Version" seriously? if they don't, is there
any force guiding people to prioritize or decide what to (not) work
on? I'm sure the communication happens, just doesn't seem like it's
fully on JIRA, which is ultimately suboptimal.


> I actually did spent quite a bit of time asking people to close various
> umbrella issues, and I was pretty strict about watching JIRA throughout the
> process.  Perhaps as an additional step, future preview releases or branch
> cuts can include a link to an authoritative dashboard that we will use to
> decide when we are ready to make an RC.  I'm also open to other suggestions.

Yes, that's great. It takes the same effort from everyone. Having a
green light on a dashboard at release time is only the symptom of
decent planning. The effect I think it really needs to have occurs
now: what's really probably on the menu for 1.7? and periodically
track against that goal. Then the release process is with any luck
just a formality with no surprises.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: When to cut RCs

[Sean Busbey]

On Wed, Dec 2, 2015 at 3:19 PM, Sean Busbey  wrote:

>
>
> On Wed, Dec 2, 2015 at 3:06 PM, Michael Armbrust 
> wrote:
>
>>
>>>
>> The release is theoretically several weeks behind plan on what's
>>> intended to be a fixed release cycle too. This is why I'm not sure why
>>> today it's suddenly potentially ready for release.
>>>
>>
>> Up until today various committers have told me that there were known
>> issues with branch-1.6 that would cause them to -1 the release.  Whenever
>> this happened, I asked them to ensure there was a properly targeted blocker
>> JIRA open so people could publicly track the status of the release.  As
>> long as such issues were open, I only published a preview since making an
>> RC is pretty high cost.
>>
>> I'm sorry that it felt sudden to you, but as of last night all such known
>> issues were resolved and thus I cut a release as soon as this was the case.
>>
>>
>
> It would help me, and I'm sure other less-active folks, if this kind of
> feedback cycle were visible on dev@spark. It would also have the benefit
> of getting feedback from non-committers who might e.g. have some issue that
> they see in their own production deployments.
>
>
>
Sorry for a second email so soon. I meant to also ask, what keeps the cost
of making an RC high? Can we bring it down with better tooling?


-- 
Sean

Re: When to cut RCs

[Sean Busbey]

On Wed, Dec 2, 2015 at 3:06 PM, Michael Armbrust 
wrote:

>
>>
> The release is theoretically several weeks behind plan on what's
>> intended to be a fixed release cycle too. This is why I'm not sure why
>> today it's suddenly potentially ready for release.
>>
>
> Up until today various committers have told me that there were known
> issues with branch-1.6 that would cause them to -1 the release.  Whenever
> this happened, I asked them to ensure there was a properly targeted blocker
> JIRA open so people could publicly track the status of the release.  As
> long as such issues were open, I only published a preview since making an
> RC is pretty high cost.
>
> I'm sorry that it felt sudden to you, but as of last night all such known
> issues were resolved and thus I cut a release as soon as this was the case.
>
>

It would help me, and I'm sure other less-active folks, if this kind of
feedback cycle were visible on dev@spark. It would also have the benefit of
getting feedback from non-committers who might e.g. have some issue that
they see in their own production deployments.



-- 
Sean

Re: When to cut RCs

[Patrick Wendell]

In terms of advertising to people the status of the release and whether an
RC is likely to go out, the best mechanism I can think of is our current
mechanism of using JIRA and respecting the semantics of a blocker JIRA. We
could do a better job though creating a JIRA dashboard for each release and
linking to it publicly so it's very clear to people what is going on. I
have always used one privately when managing previous releases, but no
reason we can't put one up on the website or wiki.

IMO a mailing list is not a great mechanism for the fine-grained work of
release management because of the sheer complexity and volume of finalizing
a spark release. Being a release manager means tracking over a course of
several weeks typically dozens of distinct issues and trying to prioritize
them, get more clarity from the report of those issues, possibly reaching
out to people on the phone or in person to get more details, etc. You want
a mutable dashboard where you can convey the current status clearly.

What might be good in the early stages is a weekly e-mail to the dev@ list
just refreshing what is on the JIRA and letting people know how things are
looking. So someone just passing by has some idea of how things are going
and can chime in, etc.

Once an RC is cut then we do mostly rely on the mailing list for
discussion. At that point the number of known issues is small enough I
think to discuss in an all-to-all fashion.

- Patrick

On Wed, Dec 2, 2015 at 1:25 PM, Sean Owen  wrote:

> On Wed, Dec 2, 2015 at 9:06 PM, Michael Armbrust 
> wrote:
> > This can be debated, but I explicitly ignored test and documentation
> issues.
> > Since the docs are published separately and easy to update, I don't think
> > its worth further disturbing the release cadence for these JIRAs.
>
> It makes sense to not hold up an RC since they don't affect testing of
> functionality. Prior releases have ultimately gone out with doc issues
> still outstanding (and bugs) though. This doesn't seem to be on
> anyone's release checklist, and maybe part of it is because they're
> let slide for RCs.  Your suggestion to check-point release status
> below sounds spot on; I sort of tried to do that earlier.
>
>
> > Up until today various committers have told me that there were known
> issues
> > with branch-1.6 that would cause them to -1 the release.  Whenever this
> > happened, I asked them to ensure there was a properly targeted blocker
> JIRA
> > open so people could publicly track the status of the release.  As long
> as
> > such issues were open, I only published a preview since making an RC is
> > pretty high cost.
>
> Makes sense if these are all getting translated into Blockers and
> resolved before an RC. It's the simplest mechanism to communicate and
> track this in a distributed way.
>
> "No blockers" is a minimal criterion for release. It still seems funny
> to release with so many issues targeted for 1.6.0, including issues
> that aren't critical or bugs. Sure, that's just hygiene. But without
> it, do people take "Target Version" seriously? if they don't, is there
> any force guiding people to prioritize or decide what to (not) work
> on? I'm sure the communication happens, just doesn't seem like it's
> fully on JIRA, which is ultimately suboptimal.
>
>
> > I actually did spent quite a bit of time asking people to close various
> > umbrella issues, and I was pretty strict about watching JIRA throughout
> the
> > process.  Perhaps as an additional step, future preview releases or
> branch
> > cuts can include a link to an authoritative dashboard that we will use to
> > decide when we are ready to make an RC.  I'm also open to other
> suggestions.
>
> Yes, that's great. It takes the same effort from everyone. Having a
> green light on a dashboard at release time is only the symptom of
> decent planning. The effect I think it really needs to have occurs
> now: what's really probably on the menu for 1.7? and periodically
> track against that goal. Then the release process is with any luck
> just a formality with no surprises.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: When to cut RCs

[Michael Armbrust]

Thanks for bringing this up Sean. I think we are all happy to adopt
concrete suggestions to make the release process more transparent,
including pinging the list before kicking off the release build.

Technically there's still a Blocker bug:
> https://issues.apache.org/jira/browse/SPARK-12000


Sorry, I misprioritized this particular issue when I thought that it was
going to block the release by causing the doc build to fail. When I
realized the failure was non-deterministic and isolated to OSX (i.e. the
official release build on jenkins is not affected) I failed to update the
issue.  It doesn't show up on the dashboard that I've been using to track
the release

since
its labeled documentation.


> The 'race' doesn't matter that much, but release planning remains the
> real bug-bear here. There are still, for instance, 52 issues targeted
> at 1.6.0, 42 of which were raised and targeted by committers. For
> example, I count 6 'umbrella' JIRAs in ML alone that are still open.
>

This can be debated, but I explicitly ignored test and documentation
issues.  Since the docs are published separately and easy to update, I
don't think its worth further disturbing the release cadence for these
JIRAs.


> The release is theoretically several weeks behind plan on what's
> intended to be a fixed release cycle too. This is why I'm not sure why
> today it's suddenly potentially ready for release.
>

Up until today various committers have told me that there were known issues
with branch-1.6 that would cause them to -1 the release.  Whenever this
happened, I asked them to ensure there was a properly targeted blocker JIRA
open so people could publicly track the status of the release.  As long as
such issues were open, I only published a preview since making an RC is
pretty high cost.

I'm sorry that it felt sudden to you, but as of last night all such known
issues were resolved and thus I cut a release as soon as this was the case.

I'm just curious, am I the only one that thinks this isn't roughly
> normal, or do other people manage releases this way? I know the real
> world is messy and this is better than in the past, but I still get
> surprised by how each 1.x release actually comes about.
>

I actually did spent quite a bit of time asking people to close various
umbrella issues, and I was pretty strict about watching JIRA throughout the
process.  Perhaps as an additional step, future preview releases or branch
cuts can include a link to an authoritative dashboard that we will use to
decide when we are ready to make an RC.  I'm also open to other suggestions.

Michael