Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Davanum Srinivas]

+1 to " If there are no contributors for an LTS release, there will be
no LTS release. If there *are* contributors, then we'll find a way to
make some sort of LTS model work within the other constraints we
have."

+1 to let's get some folks who are interested a place to
collaborate and talk to each other and get things up and running. That
would be the first step. If it ends up in LTS and skip-level upgrades
for all projects, we all win!

That's a destination, not the first step!

Thanks,
Dims

On Sun, Nov 12, 2017 at 6:04 AM, Doug Hellmann  wrote:
> Excerpts from Clint Byrum's message of 2017-11-11 08:41:15 -0800:
>> Excerpts from Doug Hellmann's message of 2017-11-10 13:11:45 -0500:
>> > Excerpts from Clint Byrum's message of 2017-11-08 23:15:15 -0800:
>> > > Excerpts from Samuel Cassiba's message of 2017-11-08 08:27:12 -0800:
>> > > > On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
>> > > >  wrote:
>> > > > > Hello Ops folks,
>> > > > >
>> > > > > This morning at the Sydney Summit we had a very well attended and 
>> > > > > very
>> > > > > productive session about how to go about keeping a selection of past
>> > > > > releases available and maintained for a longer period of time (LTS).
>> > > > >
>> > > > > There was agreement in the room that this could be accomplished by
>> > > > > moving the responsibility for those releases from the Stable Branch
>> > > > > team down to those who are already creating and testing patches for
>> > > > > old releases: The distros, deployers, and operators.
>> > > > >
>> > > > > The concept, in general, is to create a new set of cores from these
>> > > > > groups, and use 3rd party CI to validate patches. There are lots of
>> > > > > details to be worked out yet, but our amazing UC (User Committee) 
>> > > > > will
>> > > > > be begin working out the details.
>> > > > >
>> > > > > Please take a look at the Etherpad from the session if you'd like to
>> > > > > see the details. More importantly, if you would like to contribute to
>> > > > > this effort, please add your name to the list starting on line 133.
>> > > > >
>> > > > > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
>> > > > >
>> > > > > Thanks to everyone who participated!
>> > > > >
>> > > > > Cheers,
>> > > > > Erik
>> > > > >
>> > > > > __
>> > > > > OpenStack Development Mailing List (not for usage questions)
>> > > > > Unsubscribe: 
>> > > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > >
>> > > > In advance, pardon the defensive tone. I was not in a position to
>> > > > attend, or even be in Sydney. However, as this comes across the ML, I
>> > > > can't help but get the impression this effort would be forcing more
>> > > > work on already stretched teams, ie. deployment-focused development
>> > > > teams already under a crunch as contributor count continues to decline
>> > > > in favor of other projects inside and out of OpenStack.
>> > > >
>> > >
>> > > I suspect if LTS's become a normal part of OpenStack, most deployment
>> > > projects will decline to support the interim releases. We can infer this
>> > > from the way Ubuntu is used. This might actually be a good thing for the
>> > > chef OpenStack community. 3 out of 3.5 of you can focus on the LTS bits,
>> > > and the 0.5 person can do some best effort to cover the weird corner
>> > > case of "previous stable release to master".
>> > >
>> > > The biggest challenge will be ensuring that the skip-level upgrades
>> > > work. The current grenade based upgrade jobs are already quite a bear to
>> > > keep working IIRC. I've not seen if chef or any of the deployment 
>> > > projects
>> > > test upgrades like that.
>> > >
>> > > However, if people can stop caring much about the interim releases and
>> > > just keep "previous LTS to master" upgrades working, then that might be
>> > > good for casual adoption.
>> > >
>> > > Personally I'd rather we make it easier to run "rolling release"
>> > > OpenStack. Maybe we can do that if we stop cutting stable releases every
>> > > 6 months.
>> > >
>> >
>> > We should stop calling what we're talking about "LTS". It isn't
>> > going to match the expectations of anyone receiving LTS releases
>> > for other products, at least at first. Perhaps "Deployer Supported"
>> > or "User Supported" are better terms for what we're talking about.
>> >
>>
>> I believe this state we're in is a stop-gap on the way to the full
>> LTS. People are already getting stuck. We're going to help them stay stuck
>> by upstreaming bug fixes.  We should be mindful of that and provide a way
>> to get less-stuck. The LTS model from other projects has proven quite
>> popular, and it would make sense for us to embrace it if our operators
>> are hurting with the current model, which I believe they are.
>>
>> > 

Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Doug Hellmann]

Excerpts from Clint Byrum's message of 2017-11-11 08:41:15 -0800:
> Excerpts from Doug Hellmann's message of 2017-11-10 13:11:45 -0500:
> > Excerpts from Clint Byrum's message of 2017-11-08 23:15:15 -0800:
> > > Excerpts from Samuel Cassiba's message of 2017-11-08 08:27:12 -0800:
> > > > On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
> > > >  wrote:
> > > > > Hello Ops folks,
> > > > >
> > > > > This morning at the Sydney Summit we had a very well attended and very
> > > > > productive session about how to go about keeping a selection of past
> > > > > releases available and maintained for a longer period of time (LTS).
> > > > >
> > > > > There was agreement in the room that this could be accomplished by
> > > > > moving the responsibility for those releases from the Stable Branch
> > > > > team down to those who are already creating and testing patches for
> > > > > old releases: The distros, deployers, and operators.
> > > > >
> > > > > The concept, in general, is to create a new set of cores from these
> > > > > groups, and use 3rd party CI to validate patches. There are lots of
> > > > > details to be worked out yet, but our amazing UC (User Committee) will
> > > > > be begin working out the details.
> > > > >
> > > > > Please take a look at the Etherpad from the session if you'd like to
> > > > > see the details. More importantly, if you would like to contribute to
> > > > > this effort, please add your name to the list starting on line 133.
> > > > >
> > > > > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
> > > > >
> > > > > Thanks to everyone who participated!
> > > > >
> > > > > Cheers,
> > > > > Erik
> > > > >
> > > > > __
> > > > > OpenStack Development Mailing List (not for usage questions)
> > > > > Unsubscribe: 
> > > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > > 
> > > > In advance, pardon the defensive tone. I was not in a position to
> > > > attend, or even be in Sydney. However, as this comes across the ML, I
> > > > can't help but get the impression this effort would be forcing more
> > > > work on already stretched teams, ie. deployment-focused development
> > > > teams already under a crunch as contributor count continues to decline
> > > > in favor of other projects inside and out of OpenStack.
> > > > 
> > > 
> > > I suspect if LTS's become a normal part of OpenStack, most deployment
> > > projects will decline to support the interim releases. We can infer this
> > > from the way Ubuntu is used. This might actually be a good thing for the
> > > chef OpenStack community. 3 out of 3.5 of you can focus on the LTS bits,
> > > and the 0.5 person can do some best effort to cover the weird corner
> > > case of "previous stable release to master".
> > > 
> > > The biggest challenge will be ensuring that the skip-level upgrades
> > > work. The current grenade based upgrade jobs are already quite a bear to
> > > keep working IIRC. I've not seen if chef or any of the deployment projects
> > > test upgrades like that.
> > > 
> > > However, if people can stop caring much about the interim releases and
> > > just keep "previous LTS to master" upgrades working, then that might be
> > > good for casual adoption.
> > > 
> > > Personally I'd rather we make it easier to run "rolling release"
> > > OpenStack. Maybe we can do that if we stop cutting stable releases every
> > > 6 months.
> > > 
> > 
> > We should stop calling what we're talking about "LTS". It isn't
> > going to match the expectations of anyone receiving LTS releases
> > for other products, at least at first. Perhaps "Deployer Supported"
> > or "User Supported" are better terms for what we're talking about.
> > 
> 
> I believe this state we're in is a stop-gap on the way to the full
> LTS. People are already getting stuck. We're going to help them stay stuck
> by upstreaming bug fixes.  We should be mindful of that and provide a way
> to get less-stuck. The LTS model from other projects has proven quite
> popular, and it would make sense for us to embrace it if our operators
> are hurting with the current model, which I believe they are.
> 
> > In the "LTS" room we did not agree to stop cutting stable releases
> > or to start supporting upgrades directly from N-2 (or older) to N.
> > Both of those changes would require modifying the support the
> > existing contributor base has committed to provide.
> > 
> 
> Thanks, I am just inferring those things from what was agreed on. However,
> It would make a lot of sense to discuss the plans for the future, even
> if we don't have data from the present proposal.
> 
> > Fast-forward upgrades will still need to run the migration steps
> > of each release in order, one at a time. The team working on that
> > is going to produce a document describing what works today so we
> > can 

Re: [Openstack-operators] Upstream LTS Releases

[Doug Hellmann]

Excerpts from John Dickinson's message of 2017-11-10 14:51:08 -0800:
> On 7 Nov 2017, at 15:28, Erik McCormick wrote:
> 
> > Hello Ops folks,
> >
> > This morning at the Sydney Summit we had a very well attended and very
> > productive session about how to go about keeping a selection of past
> > releases available and maintained for a longer period of time (LTS).
> >
> > There was agreement in the room that this could be accomplished by
> > moving the responsibility for those releases from the Stable Branch
> > team down to those who are already creating and testing patches for
> > old releases: The distros, deployers, and operators.
> >
> > The concept, in general, is to create a new set of cores from these
> > groups, and use 3rd party CI to validate patches. There are lots of
> > details to be worked out yet, but our amazing UC (User Committee) will
> > be begin working out the details.
> >
> > Please take a look at the Etherpad from the session if you'd like to
> > see the details. More importantly, if you would like to contribute to
> > this effort, please add your name to the list starting on line 133.
> >
> > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
> >
> > Thanks to everyone who participated!
> >
> > Cheers,
> > Erik
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> I'm not a fan of the current proposal. I feel like the discussion jumped into 
> a policy/procedure solution without getting much more feedback from 
> operators. The room heard "ops want LTS" and we now have a new governance 
> model to work out.
> 
> What I heard from ops in the room is that they want (to start) one release a 
> year who's branch isn't deleted after a year. What if that's exactly what we 
> did? I propose that OpenStack only do one release a year instead of two. We 
> still keep N-2 stable releases around. We still do backports to all open 
> stable branches. We still do all the things we're doing now, we just do it 
> once a year instead of twice.

We have so far only been able to find people to maintain stable
branches for 12-18 months. Keeping N-2 branches for annual releases
open would mean extending that support period to 2+ years. So, if
we're going to do that, we need to address the fact that we haven't
been able to retain anyone's attention that long up to this point.
Do you think keeping the branches open longer will be sufficient
to attract contributors to actually work on them?

> 
> Looking at current deliverables in the openstack releases repo, most (by 
> nearly a factor of 2x) are using "cycle-with-intermediary".
> 
> john@europa:~/Documents/openstack_releases/deliverables/pike(master)$ 
> grep release-model * | cut -d ':' -f 2- | sort | uniq -c
>   44 release-model: cycle-trailing
>  147 release-model: cycle-with-intermediary
>   37 release-model: cycle-with-milestones
>2 release-model: untagged
> 
> Any deliverable that using this model is already successfully dealing with 
> skip-level upgrades. Skip-level upgrades are already identified as needed and 
> prioritized functionality in projects that don't yet support them. Let's keep 
> working on getting that functionality supported across all OpenStack 
> deliverables. Let's move to one LTS release a year. And let's get all project 
> deliverables to start using cycle-with-intermediary releases.

Most of those deliverables are libraries. I see 16 services released
under pike as cycle-with-intermediary (there are 20 using
cycle-with-milestones).

$ cd openstack/releases
$ tox -e venv -- list-deliverables -v --series pike --type service --model 
cycle-with-intermediary
aodh   Telemetry5.0.0
ceilometer Telemetry9.0.1
cloudkitty cloudkitty   6.0.0
ironic ironic   9.1.2
karbor karbor   0.5.1
magnum magnum   5.0.1
monasca-apimonasca  2.2.0
monasca-log-apimonasca  2.3.0
panko  Telemetry3.0.0
solum  solum5.4.0
swift  swift2.15.1
tacker tacker   0.8.0
tricircle  tricircle4.0.0
vitragevitrage  1.8.2
watcherwatcher  1.4.1
zunzun  0.2.1

Of the cycle-with-intermediary deliverables, only 3 (ceilometer,
ironic, and swift) even claim the assert:supports-upgrade tag [1]
(our simplest upgrade tag, which indicates support for "cold" offline
upgrades).  I would be interested to hear from the other 

Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Thomas Goirand]

On 11/08/2017 05:27 PM, Samuel Cassiba wrote:
> ie. deployment-focused development
> teams already under a crunch as contributor count continues to decline
> in favor of other projects inside and out of OpenStack.

Did you even think that one of the reason for such a decline, is that
OpenStack is moving too fast, and has no LTS? Some major public cloud
(which I will on purpose not name) are still running Kilo, which was
released 3 years ago! 3 or 5 years support for an LTS version is the
industry standard, and OpenStack is doing only 1 year. This has driven
people away, and will continue to do so if nothing is done.

Instead of thinking "this will be more work", why don't you think of the
LTS as an opportunity to only release OpenStack Chef for the LTS? That'd
be a lot less work indeed, and IMO that's a very good opportunity for
you to scale down.

Cheers,

Thomas Goirand (zigo)

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2017-11-10 13:11:45 -0500:
> Excerpts from Clint Byrum's message of 2017-11-08 23:15:15 -0800:
> > Excerpts from Samuel Cassiba's message of 2017-11-08 08:27:12 -0800:
> > > On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
> > >  wrote:
> > > > Hello Ops folks,
> > > >
> > > > This morning at the Sydney Summit we had a very well attended and very
> > > > productive session about how to go about keeping a selection of past
> > > > releases available and maintained for a longer period of time (LTS).
> > > >
> > > > There was agreement in the room that this could be accomplished by
> > > > moving the responsibility for those releases from the Stable Branch
> > > > team down to those who are already creating and testing patches for
> > > > old releases: The distros, deployers, and operators.
> > > >
> > > > The concept, in general, is to create a new set of cores from these
> > > > groups, and use 3rd party CI to validate patches. There are lots of
> > > > details to be worked out yet, but our amazing UC (User Committee) will
> > > > be begin working out the details.
> > > >
> > > > Please take a look at the Etherpad from the session if you'd like to
> > > > see the details. More importantly, if you would like to contribute to
> > > > this effort, please add your name to the list starting on line 133.
> > > >
> > > > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
> > > >
> > > > Thanks to everyone who participated!
> > > >
> > > > Cheers,
> > > > Erik
> > > >
> > > > __
> > > > OpenStack Development Mailing List (not for usage questions)
> > > > Unsubscribe: 
> > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > 
> > > In advance, pardon the defensive tone. I was not in a position to
> > > attend, or even be in Sydney. However, as this comes across the ML, I
> > > can't help but get the impression this effort would be forcing more
> > > work on already stretched teams, ie. deployment-focused development
> > > teams already under a crunch as contributor count continues to decline
> > > in favor of other projects inside and out of OpenStack.
> > > 
> > 
> > I suspect if LTS's become a normal part of OpenStack, most deployment
> > projects will decline to support the interim releases. We can infer this
> > from the way Ubuntu is used. This might actually be a good thing for the
> > chef OpenStack community. 3 out of 3.5 of you can focus on the LTS bits,
> > and the 0.5 person can do some best effort to cover the weird corner
> > case of "previous stable release to master".
> > 
> > The biggest challenge will be ensuring that the skip-level upgrades
> > work. The current grenade based upgrade jobs are already quite a bear to
> > keep working IIRC. I've not seen if chef or any of the deployment projects
> > test upgrades like that.
> > 
> > However, if people can stop caring much about the interim releases and
> > just keep "previous LTS to master" upgrades working, then that might be
> > good for casual adoption.
> > 
> > Personally I'd rather we make it easier to run "rolling release"
> > OpenStack. Maybe we can do that if we stop cutting stable releases every
> > 6 months.
> > 
> 
> We should stop calling what we're talking about "LTS". It isn't
> going to match the expectations of anyone receiving LTS releases
> for other products, at least at first. Perhaps "Deployer Supported"
> or "User Supported" are better terms for what we're talking about.
> 

I believe this state we're in is a stop-gap on the way to the full
LTS. People are already getting stuck. We're going to help them stay stuck
by upstreaming bug fixes.  We should be mindful of that and provide a way
to get less-stuck. The LTS model from other projects has proven quite
popular, and it would make sense for us to embrace it if our operators
are hurting with the current model, which I believe they are.

> In the "LTS" room we did not agree to stop cutting stable releases
> or to start supporting upgrades directly from N-2 (or older) to N.
> Both of those changes would require modifying the support the
> existing contributor base has committed to provide.
> 

Thanks, I am just inferring those things from what was agreed on. However,
It would make a lot of sense to discuss the plans for the future, even
if we don't have data from the present proposal.

> Fast-forward upgrades will still need to run the migration steps
> of each release in order, one at a time. The team working on that
> is going to produce a document describing what works today so we
> can analyze it for ways to improve the upgrade experience, for both
> fast-forward and "regular" upgrades.  That was all discussed in a
> separate session.
> 

We are what we test. If we're going to test fast-forwards, how far into
the past do we test? It