Re: [Openstack-operators] Upstream LTS Releases

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2017-11-11 12:19:56 -0500:
> Excerpts from John Dickinson's message of 2017-11-10 14:51:08 -0800:
> > On 7 Nov 2017, at 15:28, Erik McCormick wrote:
> > 
> > > Hello Ops folks,
> > >
> > > This morning at the Sydney Summit we had a very well attended and very
> > > productive session about how to go about keeping a selection of past
> > > releases available and maintained for a longer period of time (LTS).
> > >
> > > There was agreement in the room that this could be accomplished by
> > > moving the responsibility for those releases from the Stable Branch
> > > team down to those who are already creating and testing patches for
> > > old releases: The distros, deployers, and operators.
> > >
> > > The concept, in general, is to create a new set of cores from these
> > > groups, and use 3rd party CI to validate patches. There are lots of
> > > details to be worked out yet, but our amazing UC (User Committee) will
> > > be begin working out the details.
> > >
> > > Please take a look at the Etherpad from the session if you'd like to
> > > see the details. More importantly, if you would like to contribute to
> > > this effort, please add your name to the list starting on line 133.
> > >
> > > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
> > >
> > > Thanks to everyone who participated!
> > >
> > > Cheers,
> > > Erik
> > >
> > > ___
> > > OpenStack-operators mailing list
> > > OpenStack-operators@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> > 
> > I'm not a fan of the current proposal. I feel like the discussion jumped 
> > into a policy/procedure solution without getting much more feedback from 
> > operators. The room heard "ops want LTS" and we now have a new governance 
> > model to work out.
> > 
> > What I heard from ops in the room is that they want (to start) one release 
> > a year who's branch isn't deleted after a year. What if that's exactly what 
> > we did? I propose that OpenStack only do one release a year instead of two. 
> > We still keep N-2 stable releases around. We still do backports to all open 
> > stable branches. We still do all the things we're doing now, we just do it 
> > once a year instead of twice.
> 
> We have so far only been able to find people to maintain stable
> branches for 12-18 months. Keeping N-2 branches for annual releases
> open would mean extending that support period to 2+ years. So, if
> we're going to do that, we need to address the fact that we haven't
> been able to retain anyone's attention that long up to this point.
> Do you think keeping the branches open longer will be sufficient
> to attract contributors to actually work on them?
> 

I don't think that, no. However, I do think if you also relieved the
current stable release pressure by cutting stable releases less often,
you'd be tasking the same backporters and maintainers with the same
amount of work. The difference would be that the work done early in a
cycle would mean more, as it would last longer, and thus might get
a bump in priority and overall community efficiency.

It's not all free though. There's a balancing act to play here. If you
force users to wait longer for things that aren't being backported,
they're likely to diverge more downstream

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Upstream LTS Releases

[Doug Hellmann]

Excerpts from John Dickinson's message of 2017-11-10 14:51:08 -0800:
> On 7 Nov 2017, at 15:28, Erik McCormick wrote:
> 
> > Hello Ops folks,
> >
> > This morning at the Sydney Summit we had a very well attended and very
> > productive session about how to go about keeping a selection of past
> > releases available and maintained for a longer period of time (LTS).
> >
> > There was agreement in the room that this could be accomplished by
> > moving the responsibility for those releases from the Stable Branch
> > team down to those who are already creating and testing patches for
> > old releases: The distros, deployers, and operators.
> >
> > The concept, in general, is to create a new set of cores from these
> > groups, and use 3rd party CI to validate patches. There are lots of
> > details to be worked out yet, but our amazing UC (User Committee) will
> > be begin working out the details.
> >
> > Please take a look at the Etherpad from the session if you'd like to
> > see the details. More importantly, if you would like to contribute to
> > this effort, please add your name to the list starting on line 133.
> >
> > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
> >
> > Thanks to everyone who participated!
> >
> > Cheers,
> > Erik
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> I'm not a fan of the current proposal. I feel like the discussion jumped into 
> a policy/procedure solution without getting much more feedback from 
> operators. The room heard "ops want LTS" and we now have a new governance 
> model to work out.
> 
> What I heard from ops in the room is that they want (to start) one release a 
> year who's branch isn't deleted after a year. What if that's exactly what we 
> did? I propose that OpenStack only do one release a year instead of two. We 
> still keep N-2 stable releases around. We still do backports to all open 
> stable branches. We still do all the things we're doing now, we just do it 
> once a year instead of twice.

We have so far only been able to find people to maintain stable
branches for 12-18 months. Keeping N-2 branches for annual releases
open would mean extending that support period to 2+ years. So, if
we're going to do that, we need to address the fact that we haven't
been able to retain anyone's attention that long up to this point.
Do you think keeping the branches open longer will be sufficient
to attract contributors to actually work on them?

> 
> Looking at current deliverables in the openstack releases repo, most (by 
> nearly a factor of 2x) are using "cycle-with-intermediary".
> 
> john@europa:~/Documents/openstack_releases/deliverables/pike(master)$ 
> grep release-model * | cut -d ':' -f 2- | sort | uniq -c
>   44 release-model: cycle-trailing
>  147 release-model: cycle-with-intermediary
>   37 release-model: cycle-with-milestones
>2 release-model: untagged
> 
> Any deliverable that using this model is already successfully dealing with 
> skip-level upgrades. Skip-level upgrades are already identified as needed and 
> prioritized functionality in projects that don't yet support them. Let's keep 
> working on getting that functionality supported across all OpenStack 
> deliverables. Let's move to one LTS release a year. And let's get all project 
> deliverables to start using cycle-with-intermediary releases.

Most of those deliverables are libraries. I see 16 services released
under pike as cycle-with-intermediary (there are 20 using
cycle-with-milestones).

$ cd openstack/releases
$ tox -e venv -- list-deliverables -v --series pike --type service --model 
cycle-with-intermediary
aodh   Telemetry5.0.0
ceilometer Telemetry9.0.1
cloudkitty cloudkitty   6.0.0
ironic ironic   9.1.2
karbor karbor   0.5.1
magnum magnum   5.0.1
monasca-apimonasca  2.2.0
monasca-log-apimonasca  2.3.0
panko  Telemetry3.0.0
solum  solum5.4.0
swift  swift2.15.1
tacker tacker   0.8.0
tricircle  tricircle4.0.0
vitragevitrage  1.8.2
watcherwatcher  1.4.1
zunzun  0.2.1

Of the cycle-with-intermediary deliverables, only 3 (ceilometer,
ironic, and swift) even claim the assert:supports-upgrade tag [1]
(our simplest upgrade tag, which indicates support for "cold" offline
upgrades).  I would be interested to hear from the other 

Re: [Openstack-operators] Upstream LTS Releases

[Saverio Proto]

The 1 year release cycle makes a lot of sense to me too. +1

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Upstream LTS Releases

[Samuel Cassiba]

On Fri, Nov 10, 2017 at 2:51 PM, John Dickinson  wrote:
> What I heard from ops in the room is that they want (to start) one release a
> year who's branch isn't deleted after a year. What if that's exactly what we
> did? I propose that OpenStack only do one release a year instead of two. We
> still keep N-2 stable releases around. We still do backports to all open
> stable branches. We still do all the things we're doing now, we just do it
> once a year instead of twice.
>

This seems like a much more reasonable proposal with less of a musical
chairs feeling. The spun up software developer in my basement nods in
violent agreement with the idea, and the tortured QA engineer I keep
locked up out back would love nothing more than some extra time to
test.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Upstream LTS Releases

[Jonathan Mills]

"I propose that OpenStack only do one release a year instead of two.”

I am all too happy to chime in and second (or third, or fourth) this notion.  
It is extremely challenging for many kinds of organizations (government, 
industry) to keep pace with the two releases per year model.  I think it 
actively harms OpenStack adoption, by creating a sense of instability, or 
un-maintainability.  Big organizations have many dependencies, bureaucracies, 
privilege separations (often across departments, e.g. systems vs networking vs 
storage).  By the time some orgs get OpenStack running, they’ve already been 
left in the dust by the pace of the release schedule.  Then they run into the 
security nightmare of maybe not being able to fully patch systems without 
breaking OpenStack (as Mitaka operators just hit with CentOS 7.4), which 
scenario forces the constant-upgrade loop pressure (very stressful).  An awful 
lot of negative feelings will be avoided just by having more supportable 
releases/schedules.  Doubtful any of this is a surprise to you all, but it is 
cathartic for me to say so nonetheless.  At any rate, I was not at this Summit 
and thus missed the session...

> On Nov 10, 2017, at 5:51 PM, John Dickinson  wrote:
> 
> On 7 Nov 2017, at 15:28, Erik McCormick wrote:
> 
> Hello Ops folks,
> 
> This morning at the Sydney Summit we had a very well attended and very
> productive session about how to go about keeping a selection of past
> releases available and maintained for a longer period of time (LTS).
> 
> There was agreement in the room that this could be accomplished by
> moving the responsibility for those releases from the Stable Branch
> team down to those who are already creating and testing patches for
> old releases: The distros, deployers, and operators.
> 
> The concept, in general, is to create a new set of cores from these
> groups, and use 3rd party CI to validate patches. There are lots of
> details to be worked out yet, but our amazing UC (User Committee) will
> be begin working out the details.
> 
> Please take a look at the Etherpad from the session if you'd like to
> see the details. More importantly, if you would like to contribute to
> this effort, please add your name to the list starting on line 133.
> 
> https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases 
> 
> 
> Thanks to everyone who participated!
> 
> Cheers,
> Erik
> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators 
> 
> I'm not a fan of the current proposal. I feel like the discussion jumped into 
> a policy/procedure solution without getting much more feedback from 
> operators. The room heard "ops want LTS" and we now have a new governance 
> model to work out.
> 
> What I heard from ops in the room is that they want (to start) one release a 
> year who's branch isn't deleted after a year. What if that's exactly what we 
> did? I propose that OpenStack only do one release a year instead of two. We 
> still keep N-2 stable releases around. We still do backports to all open 
> stable branches. We still do all the things we're doing now, we just do it 
> once a year instead of twice.
> 
> Looking at current deliverables in the openstack releases repo, most (by 
> nearly a factor of 2x) are using "cycle-with-intermediary".
> 
> john@europa:~/Documents/openstack_releases/deliverables/pike(master)$ grep 
> release-model * | cut -d ':' -f 2- | sort | uniq -c
>   44 release-model: cycle-trailing
>  147 release-model: cycle-with-intermediary
>   37 release-model: cycle-with-milestones
>2 release-model: untagged
> Any deliverable that using this model is already successfully dealing with 
> skip-level upgrades. Skip-level upgrades are already identified as needed and 
> prioritized functionality in projects that don't yet support them. Let's keep 
> working on getting that functionality supported across all OpenStack 
> deliverables. Let's move to one LTS release a year. And let's get all project 
> deliverables to start using cycle-with-intermediary releases.
> 
> --John
> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Upstream LTS Releases

[John Dickinson]

On 7 Nov 2017, at 15:28, Erik McCormick wrote:

> Hello Ops folks,
>
> This morning at the Sydney Summit we had a very well attended and very
> productive session about how to go about keeping a selection of past
> releases available and maintained for a longer period of time (LTS).
>
> There was agreement in the room that this could be accomplished by
> moving the responsibility for those releases from the Stable Branch
> team down to those who are already creating and testing patches for
> old releases: The distros, deployers, and operators.
>
> The concept, in general, is to create a new set of cores from these
> groups, and use 3rd party CI to validate patches. There are lots of
> details to be worked out yet, but our amazing UC (User Committee) will
> be begin working out the details.
>
> Please take a look at the Etherpad from the session if you'd like to
> see the details. More importantly, if you would like to contribute to
> this effort, please add your name to the list starting on line 133.
>
> https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
>
> Thanks to everyone who participated!
>
> Cheers,
> Erik
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

I'm not a fan of the current proposal. I feel like the discussion jumped into a 
policy/procedure solution without getting much more feedback from operators. 
The room heard "ops want LTS" and we now have a new governance model to work 
out.

What I heard from ops in the room is that they want (to start) one release a 
year who's branch isn't deleted after a year. What if that's exactly what we 
did? I propose that OpenStack only do one release a year instead of two. We 
still keep N-2 stable releases around. We still do backports to all open stable 
branches. We still do all the things we're doing now, we just do it once a year 
instead of twice.

Looking at current deliverables in the openstack releases repo, most (by nearly 
a factor of 2x) are using "cycle-with-intermediary".

john@europa:~/Documents/openstack_releases/deliverables/pike(master)$ grep 
release-model * | cut -d ':' -f 2- | sort | uniq -c
  44 release-model: cycle-trailing
 147 release-model: cycle-with-intermediary
  37 release-model: cycle-with-milestones
   2 release-model: untagged

Any deliverable that using this model is already successfully dealing with 
skip-level upgrades. Skip-level upgrades are already identified as needed and 
prioritized functionality in projects that don't yet support them. Let's keep 
working on getting that functionality supported across all OpenStack 
deliverables. Let's move to one LTS release a year. And let's get all project 
deliverables to start using cycle-with-intermediary releases.

--John




signature.asc
Description: OpenPGP digital signature
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Upstream LTS Releases

[Erik McCormick]

Hello Ops folks,

This morning at the Sydney Summit we had a very well attended and very
productive session about how to go about keeping a selection of past
releases available and maintained for a longer period of time (LTS).

There was agreement in the room that this could be accomplished by
moving the responsibility for those releases from the Stable Branch
team down to those who are already creating and testing patches for
old releases: The distros, deployers, and operators.

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

Please take a look at the Etherpad from the session if you'd like to
see the details. More importantly, if you would like to contribute to
this effort, please add your name to the list starting on line 133.

https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases

Thanks to everyone who participated!

Cheers,
Erik

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators