Re: [Openstack-operators] Cloud Upgrade Strategies

[Yuriy Brodskiy]

Database only needed for control operations. During upgrade we disable API 
(mark down on LB or take them down). This will prevent users from making any 
database changes. 
After that flow is "simple"- backup db - do a migration- perform your 
validation tests
If all good, bring up your api, if not, restore db backup to rollback 
I'm over simplifying it here but this is basic concepts. You will find more 
details in the video 






On Wed, Mar 9, 2016 at 10:38 PM -0800, "Xav Paice"  wrote:












On 10 March 2016 at 19:26, Yuriy Brodskiy  wrote:
building a new cloud is not practical for real production environments. even if 
you can afford it, how do you migrate data?
We have been doing upgrades for a while now, and came up with few basic 
principles:1) you don't have to upgrade all at the same time. do it component 
at the time2) stand up a new version along side of an existing one, test it and 
then flip DNS
Take a look at presentation team did during Vancouver summit 
https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/10-minutes-openstack-upgrades-done

(replying to the list this time, and regretting using gmail)
I readily admit to not having watched that video (but will!) - one question.  
How do you deal with the db migration if you have two versions running at the 
same time?







___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cloud Upgrade Strategies

[Xav Paice]

On 10 March 2016 at 19:26, Yuriy Brodskiy  wrote:

> building a new cloud is not practical for real production environments.
> even if you can afford it, how do you migrate data?
>
> We have been doing upgrades for a while now, and came up with few basic
> principles:
> 1) you don't have to upgrade all at the same time. do it component at the
> time
> 2) stand up a new version along side of an existing one, test it and then
> flip DNS
>
> Take a look at presentation team did during Vancouver summit
>
> https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/10-minutes-openstack-upgrades-done
>
>
(replying to the list this time, and regretting using gmail)

I readily admit to not having watched that video (but will!) - one
question.  How do you deal with the db migration if you have two versions
running at the same time?

>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cloud Upgrade Strategies

[Yuriy Brodskiy]

building a new cloud is not practical for real production environments.
even if you can afford it, how do you migrate data?

We have been doing upgrades for a while now, and came up with few basic
principles:
1) you don't have to upgrade all at the same time. do it component at the
time
2) stand up a new version along side of an existing one, test it and then
flip DNS

Take a look at presentation team did during Vancouver summit
https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/10-minutes-openstack-upgrades-done

On Sun, Mar 6, 2016 at 5:41 PM, Kavit Munshi  wrote:

> Shameless plug :)
>
> For people currently running a distro but wanting to run CI, please
> checkout StackBuffet (https://aptira.com/stackbuffet/)
>
> Currently under beta, looking for customers with real use cases to help us
> test
>
> regards,
>
> Kavit
>
> *Kavit Munshi*
>
> *Aptira - Asia Pacific’s leading provider of OpenStack*
>
> Direct/mobile: +91 971 292 9850
>
> General enquiries: +61 2 8030 2333
>
> Australia toll free: 1800 APTIRA
>
> Website aptira.com
>
> Twitter @aptira 
>
>
> On 4 March 2016 at 04:34, Robert Starmer  wrote:
>
>> I have to agree, unless you start with CI/CD as your deployment model,
>> you're going to be doing full upgrades.  And be aware that at least one
>> package model will overwrite your carefully crafted config files if you
>> choose the wrong option.  Having tried an upgrade to a system in the middle
>> of an upstream update, I can say that this is potentially a painful few
>> hours tracking down what's changed, and where my config files got tweaked
>> (yes, I'm looking at you Keystone...).  Fun for all ages.
>>
>> And while I agree that it is difficult to stand up two clouds, you can
>> stand up a smaller  new cloud, migrate those workloads that are cloud
>> native (just heard your cattle that-a-way), and as the load increases,
>> migrate the hardware from one cloud to another.  It's almost a square
>> dance, and it's never perfect, but it is doable...
>>
>> On Wed, Mar 2, 2016 at 2:58 PM, Silence Dogood 
>> wrote:
>>
>>>
>>>- In-place Full Release upgrades (upgrade an entire cloud from
>>>Icehouse to Kilo for instance)
>>>
>>> This tends to be the most likely scenario with CI/CD being almost
>>> impossible for anyone using supported openstack components ( such as SDN /
>>> NAS / other hardware integration pieces ).
>>>
>>> That's not to say people don't almost always test on a test environment
>>> ( other cloud ) first.
>>>
>>> On Wed, Mar 2, 2016 at 4:34 PM, Adam Lawson  wrote:
>>>
 Hey all,

 So I've been discussing cloud design with the team and of course the
 topic comes up about how upgrades will be handled.

 Handling OpenStack code updates generally consists of three paths in my
 experience:

- CI/CD (continuous incremental upgrades)
- In-place Full Release upgrades (upgrade an entire cloud from
Icehouse to Kilo for instance)
- Migrating old cloud to new cloud

 Is there a cloud maintenance strategy I'm missing that doesn't fall
 into the categories above? How are the rest of you adopting your cloud
 upgrade strategies and how has cloud size impacted whatever strategy you
 ultimately selected? Migrating workloads from an Icehouse cloud with 1000
 nodes to a Liberty cloud with similar capacity isn't always a realistic
 option due to cost, upgrading a cloud in place is super-risky and CI/CD
 takes a lot of development and testing overhead.

 For CI/CD strategies, I'm also curious how the rest of you are handling
 disruptive tasks (for example replacing a vRouter with a newer version,
 updating the SQL schema etc etc)? Just looking to learn from everyone's
 experiences to hopefully keep my own thinking on where it needs to be.

 Thanks!!!

 //adam

 *Adam Lawson*

 AQORN, Inc.
 427 North Tatnall Street
 Ste. 58461
 Wilmington, Delaware 19801-2230
 Toll-free: (844) 4-AQORN-NOW ext. 101
 International: +1 302-387-4660
 Direct: +1 916-246-2072

 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


>>>
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
> ___
> OpenStack-operators mailing list
> 

Re: [Openstack-operators] Cloud Upgrade Strategies

[Kavit Munshi]

Shameless plug :)

For people currently running a distro but wanting to run CI, please
checkout StackBuffet (https://aptira.com/stackbuffet/)

Currently under beta, looking for customers with real use cases to help us
test

regards,

Kavit

*Kavit Munshi*

*Aptira - Asia Pacific’s leading provider of OpenStack*

Direct/mobile: +91 971 292 9850

General enquiries: +61 2 8030 2333

Australia toll free: 1800 APTIRA

Website aptira.com

Twitter @aptira 


On 4 March 2016 at 04:34, Robert Starmer  wrote:

> I have to agree, unless you start with CI/CD as your deployment model,
> you're going to be doing full upgrades.  And be aware that at least one
> package model will overwrite your carefully crafted config files if you
> choose the wrong option.  Having tried an upgrade to a system in the middle
> of an upstream update, I can say that this is potentially a painful few
> hours tracking down what's changed, and where my config files got tweaked
> (yes, I'm looking at you Keystone...).  Fun for all ages.
>
> And while I agree that it is difficult to stand up two clouds, you can
> stand up a smaller  new cloud, migrate those workloads that are cloud
> native (just heard your cattle that-a-way), and as the load increases,
> migrate the hardware from one cloud to another.  It's almost a square
> dance, and it's never perfect, but it is doable...
>
> On Wed, Mar 2, 2016 at 2:58 PM, Silence Dogood 
> wrote:
>
>>
>>- In-place Full Release upgrades (upgrade an entire cloud from
>>Icehouse to Kilo for instance)
>>
>> This tends to be the most likely scenario with CI/CD being almost
>> impossible for anyone using supported openstack components ( such as SDN /
>> NAS / other hardware integration pieces ).
>>
>> That's not to say people don't almost always test on a test environment (
>> other cloud ) first.
>>
>> On Wed, Mar 2, 2016 at 4:34 PM, Adam Lawson  wrote:
>>
>>> Hey all,
>>>
>>> So I've been discussing cloud design with the team and of course the
>>> topic comes up about how upgrades will be handled.
>>>
>>> Handling OpenStack code updates generally consists of three paths in my
>>> experience:
>>>
>>>- CI/CD (continuous incremental upgrades)
>>>- In-place Full Release upgrades (upgrade an entire cloud from
>>>Icehouse to Kilo for instance)
>>>- Migrating old cloud to new cloud
>>>
>>> Is there a cloud maintenance strategy I'm missing that doesn't fall into
>>> the categories above? How are the rest of you adopting your cloud upgrade
>>> strategies and how has cloud size impacted whatever strategy you ultimately
>>> selected? Migrating workloads from an Icehouse cloud with 1000 nodes to a
>>> Liberty cloud with similar capacity isn't always a realistic option due to
>>> cost, upgrading a cloud in place is super-risky and CI/CD takes a lot of
>>> development and testing overhead.
>>>
>>> For CI/CD strategies, I'm also curious how the rest of you are handling
>>> disruptive tasks (for example replacing a vRouter with a newer version,
>>> updating the SQL schema etc etc)? Just looking to learn from everyone's
>>> experiences to hopefully keep my own thinking on where it needs to be.
>>>
>>> Thanks!!!
>>>
>>> //adam
>>>
>>> *Adam Lawson*
>>>
>>> AQORN, Inc.
>>> 427 North Tatnall Street
>>> Ste. 58461
>>> Wilmington, Delaware 19801-2230
>>> Toll-free: (844) 4-AQORN-NOW ext. 101
>>> International: +1 302-387-4660
>>> Direct: +1 916-246-2072
>>>
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cloud Upgrade Strategies

[Robert Starmer]

I have to agree, unless you start with CI/CD as your deployment model,
you're going to be doing full upgrades.  And be aware that at least one
package model will overwrite your carefully crafted config files if you
choose the wrong option.  Having tried an upgrade to a system in the middle
of an upstream update, I can say that this is potentially a painful few
hours tracking down what's changed, and where my config files got tweaked
(yes, I'm looking at you Keystone...).  Fun for all ages.

And while I agree that it is difficult to stand up two clouds, you can
stand up a smaller  new cloud, migrate those workloads that are cloud
native (just heard your cattle that-a-way), and as the load increases,
migrate the hardware from one cloud to another.  It's almost a square
dance, and it's never perfect, but it is doable...

On Wed, Mar 2, 2016 at 2:58 PM, Silence Dogood  wrote:

>
>- In-place Full Release upgrades (upgrade an entire cloud from
>Icehouse to Kilo for instance)
>
> This tends to be the most likely scenario with CI/CD being almost
> impossible for anyone using supported openstack components ( such as SDN /
> NAS / other hardware integration pieces ).
>
> That's not to say people don't almost always test on a test environment (
> other cloud ) first.
>
> On Wed, Mar 2, 2016 at 4:34 PM, Adam Lawson  wrote:
>
>> Hey all,
>>
>> So I've been discussing cloud design with the team and of course the
>> topic comes up about how upgrades will be handled.
>>
>> Handling OpenStack code updates generally consists of three paths in my
>> experience:
>>
>>- CI/CD (continuous incremental upgrades)
>>- In-place Full Release upgrades (upgrade an entire cloud from
>>Icehouse to Kilo for instance)
>>- Migrating old cloud to new cloud
>>
>> Is there a cloud maintenance strategy I'm missing that doesn't fall into
>> the categories above? How are the rest of you adopting your cloud upgrade
>> strategies and how has cloud size impacted whatever strategy you ultimately
>> selected? Migrating workloads from an Icehouse cloud with 1000 nodes to a
>> Liberty cloud with similar capacity isn't always a realistic option due to
>> cost, upgrading a cloud in place is super-risky and CI/CD takes a lot of
>> development and testing overhead.
>>
>> For CI/CD strategies, I'm also curious how the rest of you are handling
>> disruptive tasks (for example replacing a vRouter with a newer version,
>> updating the SQL schema etc etc)? Just looking to learn from everyone's
>> experiences to hopefully keep my own thinking on where it needs to be.
>>
>> Thanks!!!
>>
>> //adam
>>
>> *Adam Lawson*
>>
>> AQORN, Inc.
>> 427 North Tatnall Street
>> Ste. 58461
>> Wilmington, Delaware 19801-2230
>> Toll-free: (844) 4-AQORN-NOW ext. 101
>> International: +1 302-387-4660
>> Direct: +1 916-246-2072
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cloud Upgrade Strategies

[Silence Dogood]

   - In-place Full Release upgrades (upgrade an entire cloud from Icehouse
   to Kilo for instance)

This tends to be the most likely scenario with CI/CD being almost
impossible for anyone using supported openstack components ( such as SDN /
NAS / other hardware integration pieces ).

That's not to say people don't almost always test on a test environment (
other cloud ) first.

On Wed, Mar 2, 2016 at 4:34 PM, Adam Lawson  wrote:

> Hey all,
>
> So I've been discussing cloud design with the team and of course the topic
> comes up about how upgrades will be handled.
>
> Handling OpenStack code updates generally consists of three paths in my
> experience:
>
>- CI/CD (continuous incremental upgrades)
>- In-place Full Release upgrades (upgrade an entire cloud from
>Icehouse to Kilo for instance)
>- Migrating old cloud to new cloud
>
> Is there a cloud maintenance strategy I'm missing that doesn't fall into
> the categories above? How are the rest of you adopting your cloud upgrade
> strategies and how has cloud size impacted whatever strategy you ultimately
> selected? Migrating workloads from an Icehouse cloud with 1000 nodes to a
> Liberty cloud with similar capacity isn't always a realistic option due to
> cost, upgrading a cloud in place is super-risky and CI/CD takes a lot of
> development and testing overhead.
>
> For CI/CD strategies, I'm also curious how the rest of you are handling
> disruptive tasks (for example replacing a vRouter with a newer version,
> updating the SQL schema etc etc)? Just looking to learn from everyone's
> experiences to hopefully keep my own thinking on where it needs to be.
>
> Thanks!!!
>
> //adam
>
> *Adam Lawson*
>
> AQORN, Inc.
> 427 North Tatnall Street
> Ste. 58461
> Wilmington, Delaware 19801-2230
> Toll-free: (844) 4-AQORN-NOW ext. 101
> International: +1 302-387-4660
> Direct: +1 916-246-2072
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Cloud Upgrade Strategies

[Adam Lawson]

Hey all,

So I've been discussing cloud design with the team and of course the topic
comes up about how upgrades will be handled.

Handling OpenStack code updates generally consists of three paths in my
experience:

   - CI/CD (continuous incremental upgrades)
   - In-place Full Release upgrades (upgrade an entire cloud from Icehouse
   to Kilo for instance)
   - Migrating old cloud to new cloud

Is there a cloud maintenance strategy I'm missing that doesn't fall into
the categories above? How are the rest of you adopting your cloud upgrade
strategies and how has cloud size impacted whatever strategy you ultimately
selected? Migrating workloads from an Icehouse cloud with 1000 nodes to a
Liberty cloud with similar capacity isn't always a realistic option due to
cost, upgrading a cloud in place is super-risky and CI/CD takes a lot of
development and testing overhead.

For CI/CD strategies, I'm also curious how the rest of you are handling
disruptive tasks (for example replacing a vRouter with a newer version,
updating the SQL schema etc etc)? Just looking to learn from everyone's
experiences to hopefully keep my own thinking on where it needs to be.

Thanks!!!

//adam

*Adam Lawson*

AQORN, Inc.
427 North Tatnall Street
Ste. 58461
Wilmington, Delaware 19801-2230
Toll-free: (844) 4-AQORN-NOW ext. 101
International: +1 302-387-4660
Direct: +1 916-246-2072
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators