Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Erik McCormick]

The etherpad for the Fast-Forward Upgrades session at the Sydney forum is here:

https://etherpad.openstack.org/p/SYD-forum-fast-forward-upgrades

Please help us flesh it out and frame the discussion to make the best
use of our time. I have included reference materials from previous
sessions to use as a starting point. Thanks to everyone for
participating!

Cheers,
Erik

On Mon, Oct 30, 2017 at 11:25 PM,   wrote:
> See you there Eric.
>
>
>
> From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
> Sent: Monday, October 30, 2017 10:58 AM
> To: Matt Riedemann 
> Cc: OpenStack Development Mailing List ;
> openstack-operators 
> Subject: Re: [openstack-dev] [Openstack-operators]
> [skip-level-upgrades][fast-forward-upgrades] PTG summary
>
>
>
>
>
>
>
> On Oct 30, 2017 11:53 AM, "Matt Riedemann"  wrote:
>
> On 9/20/2017 9:42 AM, arkady.kanev...@dell.com wrote:
>
> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady
>
>
>
> Arkady,
>
> Are you actually moderating the forum session in Sydney because the session
> says Eric McCormick is the session moderator:
>
>
>
> I submitted it so it gets my name on it. I think Arkady and I are going to
> do it together.
>
>
>
> https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20451/fast-forward-upgrades
>
> People are asking in the nova IRC channel about this session and were told
> to ask Jay Pipes about it, but Jay isn't going to be in Sydney and isn't
> involved in fast-forward upgrades, as far as I know anyway.
>
> So whoever is moderating this session, can you please create an etherpad and
> get it linked to the wiki?
>
> https://wiki.openstack.org/wiki/Forum/Sydney2017
>
>
>
> I'll have the etherpad up today and pass it among here and on the wiki.
>
>
>
>
>
> --
>
> Thanks,
>
> Matt
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Arkady.Kanevsky]

See you there Eric.

From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
Sent: Monday, October 30, 2017 10:58 AM
To: Matt Riedemann 
Cc: OpenStack Development Mailing List ; 
openstack-operators 
Subject: Re: [openstack-dev] [Openstack-operators] 
[skip-level-upgrades][fast-forward-upgrades] PTG summary



On Oct 30, 2017 11:53 AM, "Matt Riedemann" 
> wrote:
On 9/20/2017 9:42 AM, arkady.kanev...@dell.com 
wrote:
Lee,
I can chair meeting in Sydney.
Thanks,
Arkady

Arkady,

Are you actually moderating the forum session in Sydney because the session 
says Eric McCormick is the session moderator:

I submitted it so it gets my name on it. I think Arkady and I are going to do 
it together.

https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20451/fast-forward-upgrades

People are asking in the nova IRC channel about this session and were told to 
ask Jay Pipes about it, but Jay isn't going to be in Sydney and isn't involved 
in fast-forward upgrades, as far as I know anyway.

So whoever is moderating this session, can you please create an etherpad and 
get it linked to the wiki?

https://wiki.openstack.org/wiki/Forum/Sydney2017

I'll have the etherpad up today and pass it among here and on the wiki.



--

Thanks,

Matt


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Erik McCormick]

On Oct 30, 2017 11:53 AM, "Matt Riedemann"  wrote:

On 9/20/2017 9:42 AM, arkady.kanev...@dell.com wrote:

> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady
>

Arkady,

Are you actually moderating the forum session in Sydney because the session
says Eric McCormick is the session moderator:


I submitted it so it gets my name on it. I think Arkady and I are going to
do it together.

https://www.openstack.org/summit/sydney-2017/summit-schedule
/events/20451/fast-forward-upgrades

People are asking in the nova IRC channel about this session and were told
to ask Jay Pipes about it, but Jay isn't going to be in Sydney and isn't
involved in fast-forward upgrades, as far as I know anyway.

So whoever is moderating this session, can you please create an etherpad
and get it linked to the wiki?

https://wiki.openstack.org/wiki/Forum/Sydney2017


I'll have the etherpad up today and pass it among here and on the wiki.



-- 

Thanks,

Matt


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Matt Riedemann]

On 9/20/2017 9:42 AM, arkady.kanev...@dell.com wrote:

Lee,
I can chair meeting in Sydney.
Thanks,
Arkady


Arkady,

Are you actually moderating the forum session in Sydney because the 
session says Eric McCormick is the session moderator:


https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20451/fast-forward-upgrades

People are asking in the nova IRC channel about this session and were 
told to ask Jay Pipes about it, but Jay isn't going to be in Sydney and 
isn't involved in fast-forward upgrades, as far as I know anyway.


So whoever is moderating this session, can you please create an etherpad 
and get it linked to the wiki?


https://wiki.openstack.org/wiki/Forum/Sydney2017

--

Thanks,

Matt

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Arkady.Kanevsky]

There are some loose ends that Saverio correctly bringing up.
These perfect points to discuss at Forum.
Suggest we start etherpad to collect agenda for it.

-Original Message-
From: Lee Yarwood [mailto:lyarw...@redhat.com] 
Sent: Friday, September 29, 2017 7:04 AM
To: Saverio Proto 
Cc: OpenStack Development Mailing List (not for usage questions) 
; openstack-operators@lists.openstack.org
Subject: Re: [openstack-dev] [Openstack-operators] 
[skip-level-upgrades][fast-forward-upgrades] PTG summary

On 29-09-17 11:40:21, Saverio Proto wrote:
> Hello,
> 
> sorry I could not make it to the PTG.
> 
> I have an idea that I want to share with the community. I hope this is 
> a good place to start the discussion.
> 
> After years of Openstack operations, upgrading releases from Icehouse 
> to Newton, the feeling is that the control plane upgrade is doable.
> 
> But it is also a lot of pain to upgrade all the compute nodes. This 
> really causes downtime to the VMs that are running.
> I can't always make live migrations, sometimes the VMs are just too 
> big or too busy.
> 
> It would be nice to guarantee the ability to run an updated control 
> plane with compute nodes up to N-3 Release.
> 
> This way even if we have to upgrade the control plane every 6 months, 
> we can keep a longer lifetime for compute nodes. Basically we can 
> never upgrade them until we decommission the hardware.
> 
> If there are new features that require updated compute nodes, we can 
> always organize our datacenter in availability zones, not scheduling 
> new VMs to those compute nodes.
> 
> To my understanding this means having compatibility at least for the 
> nova-compute agent and the neutron-agents running on the compute node.
> 
> Is it a very bad idea ?
> 
> Do other people feel like me that upgrading all the compute nodes is 
> also a big part of the burden regarding the upgrade ?

Yeah, I don't think the Nova community would ever be able or willing to verify 
and maintain that level of backward compatibility. Ultimately there's nothing 
stopping you from upgrading Nova on the computes while also keeping instance 
running.

You only run into issues with kernel, OVS and QEMU (for n-cpu with
libvirt) etc upgrades that require reboots or instances to be restarted (either 
hard or via live-migration). If you're unable or just unwilling to take 
downtime for instances that can't be moved when these components require an 
update then you have bigger problems IMHO.

Regards,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Lee Yarwood]

On 29-09-17 11:40:21, Saverio Proto wrote:
> Hello,
> 
> sorry I could not make it to the PTG.
> 
> I have an idea that I want to share with the community. I hope this is a
> good place to start the discussion.
> 
> After years of Openstack operations, upgrading releases from Icehouse to
> Newton, the feeling is that the control plane upgrade is doable.
> 
> But it is also a lot of pain to upgrade all the compute nodes. This
> really causes downtime to the VMs that are running.
> I can't always make live migrations, sometimes the VMs are just too big
> or too busy.
> 
> It would be nice to guarantee the ability to run an updated control
> plane with compute nodes up to N-3 Release.
> 
> This way even if we have to upgrade the control plane every 6 months, we
> can keep a longer lifetime for compute nodes. Basically we can never
> upgrade them until we decommission the hardware.
> 
> If there are new features that require updated compute nodes, we can
> always organize our datacenter in availability zones, not scheduling new
> VMs to those compute nodes.
> 
> To my understanding this means having compatibility at least for the
> nova-compute agent and the neutron-agents running on the compute node.
> 
> Is it a very bad idea ?
> 
> Do other people feel like me that upgrading all the compute nodes is
> also a big part of the burden regarding the upgrade ?

Yeah, I don't think the Nova community would ever be able or willing to
verify and maintain that level of backward compatibility. Ultimately
there's nothing stopping you from upgrading Nova on the computes while
also keeping instance running.

You only run into issues with kernel, OVS and QEMU (for n-cpu with
libvirt) etc upgrades that require reboots or instances to be restarted
(either hard or via live-migration). If you're unable or just unwilling
to take downtime for instances that can't be moved when these components
require an update then you have bigger problems IMHO.

Regards,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76


signature.asc
Description: PGP signature
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Arkady.Kanevsky]

Erik,
Thanks for setting up a session for it.
Glad it is driven by Operators.
I will be happy to work with you on the session and run it with you.
Thanks,
Arkady

From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
Sent: Thursday, September 28, 2017 7:40 AM
To: Lee Yarwood 
Cc: OpenStack Development Mailing List ; 
openstack-operators 
Subject: Re: [openstack-dev] [Openstack-operators] 
[skip-level-upgrades][fast-forward-upgrades] PTG summary


On Sep 28, 2017 4:31 AM, "Lee Yarwood" 
> wrote:
On 20-09-17 14:56:20, arkady.kanev...@dell.com 
wrote:
> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady
Thanks Arkady!

FYI I see that emccormickva has created the following Forum session to
discuss FF upgrades:

http://forumtopics.openstack.org/cfp/details/19

You might want to reach out to him to help craft the agenda for the
session based on our discussions in Denver.
.
I just didn't want to risk it not getting in, and it was on our etherpad as 
well. I'm happy to help, but would love for you guys to lead.

Thanks,
Erik


Thanks again,

Lee
--
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Lee Yarwood]

On 20-09-17 14:56:20, arkady.kanev...@dell.com wrote:
> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady

Thanks Arkady!

FYI I see that emccormickva has created the following Forum session to
discuss FF upgrades:

http://forumtopics.openstack.org/cfp/details/19

You might want to reach out to him to help craft the agenda for the
session based on our discussions in Denver.

Thanks again,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Arkady.Kanevsky]

Lee,
I can chair meeting in Sydney.
Thanks,
Arkady

-Original Message-
From: Lee Yarwood [mailto:lyarw...@redhat.com] 
Sent: Wednesday, September 20, 2017 8:29 AM
To: openstack-...@lists.openstack.org; openstack-operators@lists.openstack.org
Subject: [openstack-dev] [skip-level-upgrades][fast-forward-upgrades] PTG 
summary

My thanks again to everyone who attended and contributed to the skip-level 
upgrades track over the first two days of last weeks PTG.
I've included a short summary of our discussions below with a list of agreed 
actions for Queens at the end.

tl;dr s/skip-level/fast-forward/g

https://etherpad.openstack.org/p/queens-PTG-skip-level-upgrades

Monday - Define and rename
--

During our first session [1] we briefly discussed the history of the skip-level 
upgrades effort within the community and the various misunderstandings that 
have arisen from previous conversations around this topic at past events.

We agreed that at present the only way to perform upgrades between N and
N+>=2 releases of OpenStack was to upgrade linearly through each major
release, without skipping between the starting and target release of the 
upgrade.

This is contrary to previous discussions on the topic where it had been 
suggested that releases could be skipped if DB migrations for these releases 
were applied in bulk later in the process. As projects within the community 
currently offer no such support for this it was agreed to continue to use the 
supported N to N+1 upgrade jumps, albeit in a minimal, offline way.

The name skip-level upgrades has had an obvious role to play in the confusion 
here and as such the renaming of this effort was discussed at length. Various 
suggestions are listed on the pad but for the time being I'm going to stick 
with the basic `fast-forward upgrades` name (FFOU, OFF, BOFF, FFUD etc were all 
close behind). This removes any notion of releases being skipped and should 
hopefully avoid any further confusion in the future.
 
Support by the projects for offline upgrades was then discussed with a recent 
Ironic issue [2] highlighted as an example where projects have required 
services to run before the upgrade could be considered complete. The additional 
requirement of ensuring both workloads and the data plane remain active during 
the upgrade was also then discussed. It was agreed that both the 
`supports-upgrades` [3] and `supports-accessible-upgrades` [4] tags should be 
updated to reflect these requirements for fast-forward upgrades.

Given the above it was agreed that this new definition of what fast-forward 
upgrades are and the best practices associated with them should be clearly 
documented somewhere. Various operators in the room highlighted that they would 
like to see a high level document outline the steps required to achieve this, 
hopefully written by someone with past experience of running this type of 
upgrade.

I failed to capture the names of the individuals who were interested in helping 
out here. If anyone is interested in helping out here please feel free to add 
your name to the actions either at the end of this mail or at the bottom of the 
pad.

In the afternoon we reviewed the current efforts within the community to 
implement fast-forward upgrades, covering TripleO, Charms (Juju) and 
openstack-ansible. While this was insightful to many in the room there didn't 
appear to be any obvious areas of collaboration outside of sharing best 
practice and defining the high level flow of a fast-forward upgrade.

Tuesday - NFV, SIG and actions
--

Tuesday started with a discussion around NFV considerations with fast-forward 
upgrades. These ranged from the previously mentioned need for the data plane to 
remain active during the upgrade to the restricted nature of upgrades in NFV 
environments in terms of time and number of reboots.

It was highlighted that there are some serious as yet unresolved bugs in Nova 
regarding the live migration of instances using SR-IOV devices.
This currently makes the moving of workloads either prior to or during the 
upgrade particularly difficult.

Rollbacks were also discussed and the need for any best practice documentation 
around fast-forward upgrades to include steps to allow the recovery of 
environments if things fail was also highlighted.

We then revisited an idea from the first day of finding or creating a SIG for 
this effort to call home. It was highlighted that there was a suggestion in the 
packaging room to create a Deployment / Lifecycle SIG.
After speaking with a few individuals later in the week I've taken the action 
to reach out on the openstack-sigs mailing list for further input.

Finally, during a brief discussion on ways we could collaborate and share 
tooling for fast-forward upgrades a new tool to migrate configuration files 
between N to N+>=2 releases was introduced [5]. While interesting it was seen 
as a more generic utility that could also