Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Erik McCormick]

The etherpad for the Fast-Forward Upgrades session at the Sydney forum is here:

https://etherpad.openstack.org/p/SYD-forum-fast-forward-upgrades

Please help us flesh it out and frame the discussion to make the best
use of our time. I have included reference materials from previous
sessions to use as a starting point. Thanks to everyone for
participating!

Cheers,
Erik

On Mon, Oct 30, 2017 at 11:25 PM,  <arkady.kanev...@dell.com> wrote:
> See you there Eric.
>
>
>
> From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
> Sent: Monday, October 30, 2017 10:58 AM
> To: Matt Riedemann <mriede...@gmail.com>
> Cc: OpenStack Development Mailing List <openstack-dev@lists.openstack.org>;
> openstack-operators <openstack-operat...@lists.openstack.org>
> Subject: Re: [openstack-dev] [Openstack-operators]
> [skip-level-upgrades][fast-forward-upgrades] PTG summary
>
>
>
>
>
>
>
> On Oct 30, 2017 11:53 AM, "Matt Riedemann" <mriede...@gmail.com> wrote:
>
> On 9/20/2017 9:42 AM, arkady.kanev...@dell.com wrote:
>
> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady
>
>
>
> Arkady,
>
> Are you actually moderating the forum session in Sydney because the session
> says Eric McCormick is the session moderator:
>
>
>
> I submitted it so it gets my name on it. I think Arkady and I are going to
> do it together.
>
>
>
> https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20451/fast-forward-upgrades
>
> People are asking in the nova IRC channel about this session and were told
> to ask Jay Pipes about it, but Jay isn't going to be in Sydney and isn't
> involved in fast-forward upgrades, as far as I know anyway.
>
> So whoever is moderating this session, can you please create an etherpad and
> get it linked to the wiki?
>
> https://wiki.openstack.org/wiki/Forum/Sydney2017
>
>
>
> I'll have the etherpad up today and pass it among here and on the wiki.
>
>
>
>
>
> --
>
> Thanks,
>
> Matt
>
>
>
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
>
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Arkady.Kanevsky]

See you there Eric.

From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
Sent: Monday, October 30, 2017 10:58 AM
To: Matt Riedemann <mriede...@gmail.com>
Cc: OpenStack Development Mailing List <openstack-dev@lists.openstack.org>; 
openstack-operators <openstack-operat...@lists.openstack.org>
Subject: Re: [openstack-dev] [Openstack-operators] 
[skip-level-upgrades][fast-forward-upgrades] PTG summary



On Oct 30, 2017 11:53 AM, "Matt Riedemann" 
<mriede...@gmail.com<mailto:mriede...@gmail.com>> wrote:
On 9/20/2017 9:42 AM, arkady.kanev...@dell.com<mailto:arkady.kanev...@dell.com> 
wrote:
Lee,
I can chair meeting in Sydney.
Thanks,
Arkady

Arkady,

Are you actually moderating the forum session in Sydney because the session 
says Eric McCormick is the session moderator:

I submitted it so it gets my name on it. I think Arkady and I are going to do 
it together.

https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20451/fast-forward-upgrades

People are asking in the nova IRC channel about this session and were told to 
ask Jay Pipes about it, but Jay isn't going to be in Sydney and isn't involved 
in fast-forward upgrades, as far as I know anyway.

So whoever is moderating this session, can you please create an etherpad and 
get it linked to the wiki?

https://wiki.openstack.org/wiki/Forum/Sydney2017

I'll have the etherpad up today and pass it among here and on the wiki.



--

Thanks,

Matt


___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org<mailto:openstack-operat...@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Erik McCormick]

On Oct 30, 2017 11:53 AM, "Matt Riedemann"  wrote:

On 9/20/2017 9:42 AM, arkady.kanev...@dell.com wrote:

> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady
>

Arkady,

Are you actually moderating the forum session in Sydney because the session
says Eric McCormick is the session moderator:


I submitted it so it gets my name on it. I think Arkady and I are going to
do it together.

https://www.openstack.org/summit/sydney-2017/summit-schedule
/events/20451/fast-forward-upgrades

People are asking in the nova IRC channel about this session and were told
to ask Jay Pipes about it, but Jay isn't going to be in Sydney and isn't
involved in fast-forward upgrades, as far as I know anyway.

So whoever is moderating this session, can you please create an etherpad
and get it linked to the wiki?

https://wiki.openstack.org/wiki/Forum/Sydney2017


I'll have the etherpad up today and pass it among here and on the wiki.



-- 

Thanks,

Matt


___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Arkady.Kanevsky]

There are some loose ends that Saverio correctly bringing up.
These perfect points to discuss at Forum.
Suggest we start etherpad to collect agenda for it.

-Original Message-
From: Lee Yarwood [mailto:lyarw...@redhat.com] 
Sent: Friday, September 29, 2017 7:04 AM
To: Saverio Proto <saverio.pr...@switch.ch>
Cc: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>; openstack-operat...@lists.openstack.org
Subject: Re: [openstack-dev] [Openstack-operators] 
[skip-level-upgrades][fast-forward-upgrades] PTG summary

On 29-09-17 11:40:21, Saverio Proto wrote:
> Hello,
> 
> sorry I could not make it to the PTG.
> 
> I have an idea that I want to share with the community. I hope this is 
> a good place to start the discussion.
> 
> After years of Openstack operations, upgrading releases from Icehouse 
> to Newton, the feeling is that the control plane upgrade is doable.
> 
> But it is also a lot of pain to upgrade all the compute nodes. This 
> really causes downtime to the VMs that are running.
> I can't always make live migrations, sometimes the VMs are just too 
> big or too busy.
> 
> It would be nice to guarantee the ability to run an updated control 
> plane with compute nodes up to N-3 Release.
> 
> This way even if we have to upgrade the control plane every 6 months, 
> we can keep a longer lifetime for compute nodes. Basically we can 
> never upgrade them until we decommission the hardware.
> 
> If there are new features that require updated compute nodes, we can 
> always organize our datacenter in availability zones, not scheduling 
> new VMs to those compute nodes.
> 
> To my understanding this means having compatibility at least for the 
> nova-compute agent and the neutron-agents running on the compute node.
> 
> Is it a very bad idea ?
> 
> Do other people feel like me that upgrading all the compute nodes is 
> also a big part of the burden regarding the upgrade ?

Yeah, I don't think the Nova community would ever be able or willing to verify 
and maintain that level of backward compatibility. Ultimately there's nothing 
stopping you from upgrading Nova on the computes while also keeping instance 
running.

You only run into issues with kernel, OVS and QEMU (for n-cpu with
libvirt) etc upgrades that require reboots or instances to be restarted (either 
hard or via live-migration). If you're unable or just unwilling to take 
downtime for instances that can't be moved when these components require an 
update then you have bigger problems IMHO.

Regards,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Lee Yarwood]

On 29-09-17 11:40:21, Saverio Proto wrote:
> Hello,
> 
> sorry I could not make it to the PTG.
> 
> I have an idea that I want to share with the community. I hope this is a
> good place to start the discussion.
> 
> After years of Openstack operations, upgrading releases from Icehouse to
> Newton, the feeling is that the control plane upgrade is doable.
> 
> But it is also a lot of pain to upgrade all the compute nodes. This
> really causes downtime to the VMs that are running.
> I can't always make live migrations, sometimes the VMs are just too big
> or too busy.
> 
> It would be nice to guarantee the ability to run an updated control
> plane with compute nodes up to N-3 Release.
> 
> This way even if we have to upgrade the control plane every 6 months, we
> can keep a longer lifetime for compute nodes. Basically we can never
> upgrade them until we decommission the hardware.
> 
> If there are new features that require updated compute nodes, we can
> always organize our datacenter in availability zones, not scheduling new
> VMs to those compute nodes.
> 
> To my understanding this means having compatibility at least for the
> nova-compute agent and the neutron-agents running on the compute node.
> 
> Is it a very bad idea ?
> 
> Do other people feel like me that upgrading all the compute nodes is
> also a big part of the burden regarding the upgrade ?

Yeah, I don't think the Nova community would ever be able or willing to
verify and maintain that level of backward compatibility. Ultimately
there's nothing stopping you from upgrading Nova on the computes while
also keeping instance running.

You only run into issues with kernel, OVS and QEMU (for n-cpu with
libvirt) etc upgrades that require reboots or instances to be restarted
(either hard or via live-migration). If you're unable or just unwilling
to take downtime for instances that can't be moved when these components
require an update then you have bigger problems IMHO.

Regards,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Saverio Proto]

Hello,

sorry I could not make it to the PTG.

I have an idea that I want to share with the community. I hope this is a
good place to start the discussion.

After years of Openstack operations, upgrading releases from Icehouse to
Newton, the feeling is that the control plane upgrade is doable.

But it is also a lot of pain to upgrade all the compute nodes. This
really causes downtime to the VMs that are running.
I can't always make live migrations, sometimes the VMs are just too big
or too busy.

It would be nice to guarantee the ability to run an updated control
plane with compute nodes up to N-3 Release.

This way even if we have to upgrade the control plane every 6 months, we
can keep a longer lifetime for compute nodes. Basically we can never
upgrade them until we decommission the hardware.

If there are new features that require updated compute nodes, we can
always organize our datacenter in availability zones, not scheduling new
VMs to those compute nodes.

To my understanding this means having compatibility at least for the
nova-compute agent and the neutron-agents running on the compute node.

Is it a very bad idea ?

Do other people feel like me that upgrading all the compute nodes is
also a big part of the burden regarding the upgrade ?

thanks !

Saverio


On 28.09.17 17:38, arkady.kanev...@dell.com wrote:
> Erik,
> 
> Thanks for setting up a session for it.
> 
> Glad it is driven by Operators.
> 
> I will be happy to work with you on the session and run it with you.
> 
> Thanks,
> 
> Arkady
> 
>  
> 
> *From:*Erik McCormick [mailto:emccorm...@cirrusseven.com]
> *Sent:* Thursday, September 28, 2017 7:40 AM
> *To:* Lee Yarwood <lyarw...@redhat.com>
> *Cc:* OpenStack Development Mailing List
> <openstack-dev@lists.openstack.org>; openstack-operators
> <openstack-operat...@lists.openstack.org>
> *Subject:* Re: [openstack-dev] [Openstack-operators]
> [skip-level-upgrades][fast-forward-upgrades] PTG summary
> 
>  
> 
>  
> 
> On Sep 28, 2017 4:31 AM, "Lee Yarwood" <lyarw...@redhat.com
> <mailto:lyarw...@redhat.com>> wrote:
> 
> On 20-09-17 14:56:20, arkady.kanev...@dell.com
> <mailto:arkady.kanev...@dell.com> wrote:
> > Lee,
> > I can chair meeting in Sydney.
> > Thanks,
> > Arkady
> 
> Thanks Arkady!
> 
> FYI I see that emccormickva has created the following Forum session to
> discuss FF upgrades:
> 
> http://forumtopics.openstack.org/cfp/details/19
> 
> You might want to reach out to him to help craft the agenda for the
> session based on our discussions in Denver.
> 
> .
> 
> I just didn't want to risk it not getting in, and it was on our etherpad
> as well. I'm happy to help, but would love for you guys to lead.
> 
>  
> 
> Thanks,
> 
> Erik
> 
>  
> 
> 
> Thanks again,
> 
> Lee
> 
> --
> Lee Yarwood                 A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33
> F672 2D76
> 
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> <mailto:openstack-operat...@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
>  
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
SWITCH
Saverio Proto, Peta Solutions
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 1573
saverio.pr...@switch.ch, http://www.switch.ch

http://www.switch.ch/stories

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Arkady.Kanevsky]

Erik,
Thanks for setting up a session for it.
Glad it is driven by Operators.
I will be happy to work with you on the session and run it with you.
Thanks,
Arkady

From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
Sent: Thursday, September 28, 2017 7:40 AM
To: Lee Yarwood <lyarw...@redhat.com>
Cc: OpenStack Development Mailing List <openstack-dev@lists.openstack.org>; 
openstack-operators <openstack-operat...@lists.openstack.org>
Subject: Re: [openstack-dev] [Openstack-operators] 
[skip-level-upgrades][fast-forward-upgrades] PTG summary


On Sep 28, 2017 4:31 AM, "Lee Yarwood" 
<lyarw...@redhat.com<mailto:lyarw...@redhat.com>> wrote:
On 20-09-17 14:56:20, arkady.kanev...@dell.com<mailto:arkady.kanev...@dell.com> 
wrote:
> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady
Thanks Arkady!

FYI I see that emccormickva has created the following Forum session to
discuss FF upgrades:

http://forumtopics.openstack.org/cfp/details/19

You might want to reach out to him to help craft the agenda for the
session based on our discussions in Denver.
.
I just didn't want to risk it not getting in, and it was on our etherpad as 
well. I'm happy to help, but would love for you guys to lead.

Thanks,
Erik


Thanks again,

Lee
--
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76
___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org<mailto:openstack-operat...@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [skip-level-upgrades][fast-forward-upgrades] PTG summary

[Erik McCormick]

On Sep 28, 2017 4:31 AM, "Lee Yarwood"  wrote:

On 20-09-17 14:56:20, arkady.kanev...@dell.com wrote:
> Lee,
> I can chair meeting in Sydney.
> Thanks,
> Arkady

Thanks Arkady!

FYI I see that emccormickva has created the following Forum session to
discuss FF upgrades:

http://forumtopics.openstack.org/cfp/details/19

You might want to reach out to him to help craft the agenda for the
session based on our discussions in Denver.

.
I just didn't want to risk it not getting in, and it was on our etherpad as
well. I'm happy to help, but would love for you guys to lead.

Thanks,
Erik


Thanks again,

Lee
--
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672
2D76

___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev