Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Day, Phil's message of 2014-01-24 04:24:11 -0800:
> > >
> > > Cool. I like this a good bit better as it avoids the reboot. Still, this 
> > > is a rather
> > large amount of data to copy around if I'm only changing a single file in 
> > Nova.
> > >
> > 
> > I think in most cases transfer cost is worth it to know you're deploying 
> > what
> > you tested. Also it is pretty easy to just do this optimization but still be
> > rsyncing the contents of the image. Instead of downloading the whole thing
> > we could have a box expose the mounted image via rsync and then all of the
> > machines can just rsync changes. Also rsync has a batch mode where if you
> > know for sure the end-state of machines you can pre-calculate that rsync and
> > just ship that. Lots of optimization possible that will work fine in your 
> > just-
> > update-one-file scenario.
> > 
> > But really, how much does downtime cost? How much do 10Gb NICs and
> > switches cost?
> > 
> 
> It's not as simple as just saying "buy better hardware" (although I do have a 
> vested interest in that approach ;-)  - on a compute node the Network and 
> Disk bandwidth is already doing useful work for paying customers.   The more 
> overhead you put into that for updates, the more disruptive it becomes.
> 

Agreed. The question becomes whether you should reserve a portion of
your resources for updates or let them push you into over-subscription.
Either way, those are business decisions.

And once we have a working system and we can say "this costs X bitcoins",
we can make a clear justification for somebody to spend developer time
to push X downward.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Day, Phil's message of 2014-01-24 04:39:10 -0800:
> > On 01/22/2014 12:17 PM, Dan Prince wrote:
> > > I've been thinking a bit more about how TripleO updates are developing
> > specifically with regards to compute nodes. What is commonly called the
> > "update story" I think.
> > >
> > > As I understand it we expect people to actually have to reboot a compute
> > node in the cluster in order to deploy an update. This really worries me
> > because it seems like way overkill for such a simple operation. Lets say 
> > all I
> > need to deploy is a simple change to Nova's libvirt driver. And I need to
> > deploy it to *all* my compute instances. Do we really expect people to
> > actually have to reboot every single compute node in their cluster for such 
> > a
> > thing. And then do this again and again for each update they deploy?
> > 
> > FWIW, I agree that this is going to be considered unacceptable by most
> > people.  Hopefully everyone is on the same page with that.  It sounds like
> > that's the case so far in this thread, at least...
> > 
> > If you have to reboot the compute node, ideally you also have support for
> > live migrating all running VMs on that compute node elsewhere before doing
> > so.  That's not something you want to have to do for *every* little change 
> > to
> > *every* compute node.
> >
> 
> Yep, my reading is the same as yours Russell, everyone agreed that there 
> needs to be an update that avoids the reboot where possible (other parts of 
> the thread seem to be focused on how much further the update can be 
> optimized).
> 
> What's not clear to me is when the plan is to have that support in TripleO - 
> I tried looking for a matching Blueprint to see if it was targeted for 
> Icehouse but can't match it against the five listed.   Perhaps Rob or Clint 
> can clarify ?
> Feels to me that this is a must have before anyone will really be able to use 
> TripleO beyond a PoC for initial deployment.
> 

Right now we are focused on the hard case, updates requiring
reboot. Avoiding the reboot is a bit more than an optimization, but it
is something we will get to once we've nailed the harder case of handling
a new kernel and reboot gracefully.

I for one have a fear that if we start with the easy case, we'll just
avoid the hard one, spend less time on it, and thus do it poorly.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Day, Phil]

> On 01/22/2014 12:17 PM, Dan Prince wrote:
> > I've been thinking a bit more about how TripleO updates are developing
> specifically with regards to compute nodes. What is commonly called the
> "update story" I think.
> >
> > As I understand it we expect people to actually have to reboot a compute
> node in the cluster in order to deploy an update. This really worries me
> because it seems like way overkill for such a simple operation. Lets say all I
> need to deploy is a simple change to Nova's libvirt driver. And I need to
> deploy it to *all* my compute instances. Do we really expect people to
> actually have to reboot every single compute node in their cluster for such a
> thing. And then do this again and again for each update they deploy?
> 
> FWIW, I agree that this is going to be considered unacceptable by most
> people.  Hopefully everyone is on the same page with that.  It sounds like
> that's the case so far in this thread, at least...
> 
> If you have to reboot the compute node, ideally you also have support for
> live migrating all running VMs on that compute node elsewhere before doing
> so.  That's not something you want to have to do for *every* little change to
> *every* compute node.
>

Yep, my reading is the same as yours Russell, everyone agreed that there needs 
to be an update that avoids the reboot where possible (other parts of the 
thread seem to be focused on how much further the update can be optimized).

What's not clear to me is when the plan is to have that support in TripleO - I 
tried looking for a matching Blueprint to see if it was targeted for Icehouse 
but can't match it against the five listed.   Perhaps Rob or Clint can clarify ?
Feels to me that this is a must have before anyone will really be able to use 
TripleO beyond a PoC for initial deployment.






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Day, Phil]

> >
> > Cool. I like this a good bit better as it avoids the reboot. Still, this is 
> > a rather
> large amount of data to copy around if I'm only changing a single file in 
> Nova.
> >
> 
> I think in most cases transfer cost is worth it to know you're deploying what
> you tested. Also it is pretty easy to just do this optimization but still be
> rsyncing the contents of the image. Instead of downloading the whole thing
> we could have a box expose the mounted image via rsync and then all of the
> machines can just rsync changes. Also rsync has a batch mode where if you
> know for sure the end-state of machines you can pre-calculate that rsync and
> just ship that. Lots of optimization possible that will work fine in your 
> just-
> update-one-file scenario.
> 
> But really, how much does downtime cost? How much do 10Gb NICs and
> switches cost?
> 

It's not as simple as just saying "buy better hardware" (although I do have a 
vested interest in that approach ;-)  - on a compute node the Network and Disk 
bandwidth is already doing useful work for paying customers.   The more 
overhead you put into that for updates, the more disruptive it becomes.

Phil 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Robert Collins]

On 23 January 2014 09:27, Keith Basil  wrote:

>>> 3) Deploy specific application level updates via packages or tarballs. 
>>> (only selected applications/packages get deployed)
>>
>> ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD
>> environments, so this level of optimization will be frequently used.
>> And, as I've said before, optimizing for frequently-used scenarios is
>> worth spending the time on. Optimizing for infrequently-occurring
>> things... not so much. :)
>
> I don't understand the aversion to using existing, well-known tools to handle 
> this?

If you're referring to operating system packages, I laid out the
overall case point by point in this talk at LCA -
http://mirror.linux.org.au/linux.conf.au/2014/Friday/111-Diskimage-builder_deep_dive_into_a_machine_compiler_-_Robert_Collins.mp4
- I'm not sure why the video is not linked in the programme
(https://lca2014.linux.org.au/programme/schedule/friday) yet.

> A hybrid model (blending 2 and 3, above) here I think would work best where
> TripleO lays down a baseline image and the cloud operator would employ an 
> well-known
> and support configuration tool for any small diffs.

In doing that they would sacrifice all testing of the 'small diffs'.
Which is a great way to end up running a combination that doesn't
work.

> The operator would then be empowered to make the call for any major upgrades 
> that
> would adversely impact the infrastructure (and ultimately the users/apps).  
> He/She
> could say, this is a major release, let's deploy the image.
>
> Something logically like this, seems reasonable:
>
> if (system_change > 10%) {
>   use TripleO;
>   } else {
>   use Existing_Config_Management;
> }
>
> It seems disruptive to force compute (or other) nodes to reboot on trivial 
> updates.

We won't be doing that :), though it is much simpler to reason about,
so I believe some users will want it.

> If we are to get further enterprise adoption of OpenStack, this seems like a 
> huge
> blocker.  This will be a very hard sell to get traditional IT folk to buy into
> this approach:
>
> "Wait, *every* time I have to make a system change, I need to reboot 
> my
>  entire cloud?"
>
> Elastic cloud concepts are already trying enough for the enterprise.

Sure, if you frame it like that it would be a scary and unreasonable
concept - they would quite reasonably ask 'what about all my running
VMs' and other such questions :). Also it would be a massive
exaggeration as we *already* *only* do reboots on software changes,
not on last-mile configuration changes.

With rolling upgrades there would be no end user visible downtime in
the *un*optimised case. The optimised case merely deploys faster.

Finally, remember that the undercloud machines have small (~170K)
numbers of files - which takes 28 seconds to stat cold - so rsyncing a
single changed file should be ~30seconds. Thats comparable (if not
better :)) than the time to run yum or apt.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

If we're ready to head down the path of trying to isolate things and
have zero-downtime-on-one-box deploys, then CoreOS has basically solved
this with Docker.

https://coreos.com/using-coreos/

However, let's solve one thing at a time. Even with CoreOS, we still
need HA, and we still want to have rolling deploys that are driven by
tests passing and monitoring.

So once we get those things, sure, lets grind our downtime windows down
to next to nothing. :)

(Also, really if we end up using Docker on top of TripleO, we'd want to
use the OpenStack driver, and that would mean we're using OpenStack on
OpenStack on OpenStack.. which is _QUINT-O_ ;)

Excerpts from Fox, Kevin M's message of 2014-01-23 13:10:09 -0800:
> Would docker work for this?
> 
> Assume every service gets its own docker container. A deployed node is then a 
> docker base image with a set of service containers. Updating an image could 
> be:
> Check if base part of image updated (kernel, docker). if so, full redeploy 
> the node.
> Sync each container image, restart container only if different?
> 
> Maybe the same trick could be done with python virtualenvs instead...
> 
> Kevin
> 
> From: Chris Jones [c...@tenshu.net]
> Sent: Thursday, January 23, 2014 7:17 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
>   with it?
> 
> Hi
> 
> Not a tarball. The node would notice from Heat metadata that it should update 
> to a new image and would fetch that image and sync the contents to its /. 
> This would leave it bit for bit identical to a fresh deployment of the new 
> image, at least on disk. The running state would differ and that still 
> requires some design and implementation to figure out.
> 
> Cheers,
> --
> Chris Jones
> 
> > On 23 Jan 2014, at 12:57, Angus Thomas  wrote:
> >
> > On 22/01/14 20:54, Clint Byrum wrote:
> >>> >
> >>> >I don't understand the aversion to using existing, well-known tools to 
> >>> >handle this?
> >>> >
> >> These tools are of course available to users and nobody is stopping them
> >> from using them. We are optimizing for not needing them. They are there
> >> and we're not going to explode if you use them. You just lose one aspect
> >> of what we're aiming at. I believe that having image based deploys will
> >> be well received as long as it is simple to understand.
> >>
> >>> >A hybrid model (blending 2 and 3, above) here I think would work best 
> >>> >where
> >>> >TripleO lays down a baseline image and the cloud operator would employ 
> >>> >an well-known
> >>> >and support configuration tool for any small diffs.
> >>> >
> >> These tools are popular because they control entropy and make it at
> >> least more likely that what you tested ends up on the boxes.
> >>
> >> A read-only root partition is a much stronger control on entropy.
> >>
> >>> >The operator would then be empowered to make the call for any major 
> >>> >upgrades that
> >>> >would adversely impact the infrastructure (and ultimately the 
> >>> >users/apps).  He/She
> >>> >could say, this is a major release, let's deploy the image.
> >>> >
> >>> >Something logically like this, seems reasonable:
> >>> >
> >>> > if (system_change > 10%) {
> >>> >   use TripleO;
> >>> >   } else {
> >>> >   use Existing_Config_Management;
> >>> > }
> >>> >
> >> I think we can make deploying minor updates minimally invasive.
> >>
> >> We've kept it simple enough, this should be a fairly straight forward
> >> optimization cycle. And the win there is that we also improve things
> >> for the 11% change.
> >
> > Hi Clint,
> >
> > For deploying minimally-invasive minor updates, the idea, if I've 
> > understood it correctly, would be to deploy a tarball which replaced 
> > selected files on the (usually read-only) root filesystem. That would allow 
> > for selective restarting of only the services which are directly affected. 
> > The alternative, pushing out a complete root filesystem image, would 
> > necessitate the same amount of disruption in all cases.
> >
> > There are a handful of costs with that approach which concern me: It 
> > simplifies the deployment itself, but inc

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Robert Collins]

On 23 January 2014 09:19, Fox, Kevin M  wrote:
> I think most of the time taken to reboot is spent in bringing down/up the 
> services though, so I'm not sure what it really buys you if you do it all. It 
> may let you skip the crazy long bootup time on "enterprise" hardware, but 
> that could be worked around with kexec on the full reboot method too.

kexec is not the answer you are looking for - we did give it a spin,
but because the hardware is not left in the state the new OS expects,
it is super fragile: when it works, it works, when it doesn't its OMG
pain.

Also no, restarting all the openstack related services is a few seconds:
real0m41.683s
user0m3.747s
sys 0m1.451s

(thats on a control plane node)

OCP might power on faster than that, but still, its only slightly
longer than SYN timeouts.

And yes, enterprise hardware boot times are crazy long :)


-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Robert Collins]

On 23 January 2014 07:55, Clint Byrum  wrote:
> Agreed, it is tricky if we try to only restart what we've changed.

man checkrestart

> OR, just restart everything. We can make endpoints HA and use rolling
> updates to avoid spurious faults.

+1 for simple.

> There are complex ways to handle things even smoother.. but I go back to
> "What does complexity cost?"

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Chris Jones]

Hi

There is certainly a lot of interest around Docker and TripleO. There is the 
tricky aspect of it adding another layer of deployment, but how hard can it be? 
:)

Cheers.

Chris Jones
   c...@tenshu.net
www.tenshu.net




On 23 Jan 2014, at 21:10, Fox, Kevin M  wrote:

> Would docker work for this?
> 
> Assume every service gets its own docker container. A deployed node is then a 
> docker base image with a set of service containers. Updating an image could 
> be:
> Check if base part of image updated (kernel, docker). if so, full redeploy 
> the node.
> Sync each container image, restart container only if different?
> 
> Maybe the same trick could be done with python virtualenvs instead...
> 
> Kevin
> 
> From: Chris Jones [c...@tenshu.net]
> Sent: Thursday, January 23, 2014 7:17 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
>   with it?
> 
> Hi
> 
> Not a tarball. The node would notice from Heat metadata that it should update 
> to a new image and would fetch that image and sync the contents to its /. 
> This would leave it bit for bit identical to a fresh deployment of the new 
> image, at least on disk. The running state would differ and that still 
> requires some design and implementation to figure out.
> 
> Cheers,
> --
> Chris Jones
> 
>> On 23 Jan 2014, at 12:57, Angus Thomas  wrote:
>> 
>> On 22/01/14 20:54, Clint Byrum wrote:
>>>>> 
>>>>> I don't understand the aversion to using existing, well-known tools to 
>>>>> handle this?
>>>>> 
>>> These tools are of course available to users and nobody is stopping them
>>> from using them. We are optimizing for not needing them. They are there
>>> and we're not going to explode if you use them. You just lose one aspect
>>> of what we're aiming at. I believe that having image based deploys will
>>> be well received as long as it is simple to understand.
>>> 
>>>>> A hybrid model (blending 2 and 3, above) here I think would work best 
>>>>> where
>>>>> TripleO lays down a baseline image and the cloud operator would employ an 
>>>>> well-known
>>>>> and support configuration tool for any small diffs.
>>>>> 
>>> These tools are popular because they control entropy and make it at
>>> least more likely that what you tested ends up on the boxes.
>>> 
>>> A read-only root partition is a much stronger control on entropy.
>>> 
>>>>> The operator would then be empowered to make the call for any major 
>>>>> upgrades that
>>>>> would adversely impact the infrastructure (and ultimately the 
>>>>> users/apps).  He/She
>>>>> could say, this is a major release, let's deploy the image.
>>>>> 
>>>>> Something logically like this, seems reasonable:
>>>>> 
>>>>>if (system_change > 10%) {
>>>>>  use TripleO;
>>>>>  } else {
>>>>>  use Existing_Config_Management;
>>>>>}
>>>>> 
>>> I think we can make deploying minor updates minimally invasive.
>>> 
>>> We've kept it simple enough, this should be a fairly straight forward
>>> optimization cycle. And the win there is that we also improve things
>>> for the 11% change.
>> 
>> Hi Clint,
>> 
>> For deploying minimally-invasive minor updates, the idea, if I've understood 
>> it correctly, would be to deploy a tarball which replaced selected files on 
>> the (usually read-only) root filesystem. That would allow for selective 
>> restarting of only the services which are directly affected. The 
>> alternative, pushing out a complete root filesystem image, would necessitate 
>> the same amount of disruption in all cases.
>> 
>> There are a handful of costs with that approach which concern me: It 
>> simplifies the deployment itself, but increases the complexity of preparing 
>> the deployment. The administrator is going to have to identify the services 
>> which need to be restarted, based on the particular set of libraries which 
>> are touched in their partial update, and put together the service restart 
>> scripts accordingly.
>> 
>> We're also making the administrator responsible for managing the sequence in 
>> which incremental updates are deployed. Since each incremetal update will 
>> re-writ

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Fox, Kevin M]

Would docker work for this?

Assume every service gets its own docker container. A deployed node is then a 
docker base image with a set of service containers. Updating an image could be:
Check if base part of image updated (kernel, docker). if so, full redeploy the 
node.
Sync each container image, restart container only if different?

Maybe the same trick could be done with python virtualenvs instead...

Kevin

From: Chris Jones [c...@tenshu.net]
Sent: Thursday, January 23, 2014 7:17 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] our update story: can people live
with it?

Hi

Not a tarball. The node would notice from Heat metadata that it should update 
to a new image and would fetch that image and sync the contents to its /. This 
would leave it bit for bit identical to a fresh deployment of the new image, at 
least on disk. The running state would differ and that still requires some 
design and implementation to figure out.

Cheers,
--
Chris Jones

> On 23 Jan 2014, at 12:57, Angus Thomas  wrote:
>
> On 22/01/14 20:54, Clint Byrum wrote:
>>> >
>>> >I don't understand the aversion to using existing, well-known tools to 
>>> >handle this?
>>> >
>> These tools are of course available to users and nobody is stopping them
>> from using them. We are optimizing for not needing them. They are there
>> and we're not going to explode if you use them. You just lose one aspect
>> of what we're aiming at. I believe that having image based deploys will
>> be well received as long as it is simple to understand.
>>
>>> >A hybrid model (blending 2 and 3, above) here I think would work best where
>>> >TripleO lays down a baseline image and the cloud operator would employ an 
>>> >well-known
>>> >and support configuration tool for any small diffs.
>>> >
>> These tools are popular because they control entropy and make it at
>> least more likely that what you tested ends up on the boxes.
>>
>> A read-only root partition is a much stronger control on entropy.
>>
>>> >The operator would then be empowered to make the call for any major 
>>> >upgrades that
>>> >would adversely impact the infrastructure (and ultimately the users/apps). 
>>> > He/She
>>> >could say, this is a major release, let's deploy the image.
>>> >
>>> >Something logically like this, seems reasonable:
>>> >
>>> > if (system_change > 10%) {
>>> >   use TripleO;
>>> >   } else {
>>> >   use Existing_Config_Management;
>>> > }
>>> >
>> I think we can make deploying minor updates minimally invasive.
>>
>> We've kept it simple enough, this should be a fairly straight forward
>> optimization cycle. And the win there is that we also improve things
>> for the 11% change.
>
> Hi Clint,
>
> For deploying minimally-invasive minor updates, the idea, if I've understood 
> it correctly, would be to deploy a tarball which replaced selected files on 
> the (usually read-only) root filesystem. That would allow for selective 
> restarting of only the services which are directly affected. The alternative, 
> pushing out a complete root filesystem image, would necessitate the same 
> amount of disruption in all cases.
>
> There are a handful of costs with that approach which concern me: It 
> simplifies the deployment itself, but increases the complexity of preparing 
> the deployment. The administrator is going to have to identify the services 
> which need to be restarted, based on the particular set of libraries which 
> are touched in their partial update, and put together the service restart 
> scripts accordingly.
>
> We're also making the administrator responsible for managing the sequence in 
> which incremental updates are deployed. Since each incremetal update will 
> re-write a particular set of files, any machine which gets updates 1,2, 3, 
> there's an oversight, and then update 5 is deployed would end up in an odd 
> state, which would require additional tooling to detect. Package based 
> updates, with versioning and dependency tracking on each package, mitigate 
> that risk.
>
> Then there's the relationship between the state of running machines, with 
> applied partial updates, and the images which are put onto new machines by 
> Ironic. We would need to apply the partial updates to the images which Ironic 
> writes, or to have the tooling to ensure that newly deployed machines 
> immediately apply

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Angus Thomas's message of 2014-01-23 04:57:20 -0800:
> On 22/01/14 20:54, Clint Byrum wrote:
> >> >
> >> >I don't understand the aversion to using existing, well-known tools to 
> >> >handle this?
> >> >
> > These tools are of course available to users and nobody is stopping them
> > from using them. We are optimizing for not needing them. They are there
> > and we're not going to explode if you use them. You just lose one aspect
> > of what we're aiming at. I believe that having image based deploys will
> > be well received as long as it is simple to understand.
> >
> >> >A hybrid model (blending 2 and 3, above) here I think would work best 
> >> >where
> >> >TripleO lays down a baseline image and the cloud operator would employ an 
> >> >well-known
> >> >and support configuration tool for any small diffs.
> >> >
> > These tools are popular because they control entropy and make it at
> > least more likely that what you tested ends up on the boxes.
> >
> > A read-only root partition is a much stronger control on entropy.
> >
> >> >The operator would then be empowered to make the call for any major 
> >> >upgrades that
> >> >would adversely impact the infrastructure (and ultimately the 
> >> >users/apps).  He/She
> >> >could say, this is a major release, let's deploy the image.
> >> >
> >> >Something logically like this, seems reasonable:
> >> >
> >> > if (system_change > 10%) {
> >> >   use TripleO;
> >> >   } else {
> >> >   use Existing_Config_Management;
> >> > }
> >> >
> > I think we can make deploying minor updates minimally invasive.
> >
> > We've kept it simple enough, this should be a fairly straight forward
> > optimization cycle. And the win there is that we also improve things
> > for the 11% change.
> >
> 
> Hi Clint,
> 
> For deploying minimally-invasive minor updates, the idea, if I've 
> understood it correctly, would be to deploy a tarball which replaced 
> selected files on the (usually read-only) root filesystem. That would 
> allow for selective restarting of only the services which are directly 
> affected. The alternative, pushing out a complete root filesystem image, 
> would necessitate the same amount of disruption in all cases.
> 

FTR, I have not said that chopping updates into things like tars of only
the difference between old and new image is a bad idea. Its a great idea
to avoid transfer costs. It just doesn't solve the system state problem,
nor do package updates.

How exactly would we determine which running things have imported
/usr/lib/python2.7/wsgiref/__init__.py ? Assuming we have packages only,
are we going to walk through its reverse dependencies and find any init
scripts/systemd unit files, and restart them? Do we just pray there
are no transitive dependencies or do we get really broad and do a first
level deep just to be sure? This is fine, but it is complexity, and for
what gain? If we make it not disruptive to restart any given service,
we don't have to be complex, and we can spend our energy on other
things.

I don't think a tarball or packages make it any easier to know that we
have deployed the software and ensured there are no stale copies of code
or configuration running. Any optimizations that can be done will need
to be able to resolve system state with the whole software collection or
will have to carry the risk of stale software continuing to run.

I'm suggesting we focus our efforts on straight forward, easy to
understand things: HA and tests.  Those things will have a much greater
benefit on the user than trying to disect diffs and figure out what
things to restart, IMO.

> There are a handful of costs with that approach which concern me: It 
> simplifies the deployment itself, but increases the complexity of 
> preparing the deployment. The administrator is going to have to identify 
> the services which need to be restarted, based on the particular set of 
> libraries which are touched in their partial update, and put together 
> the service restart scripts accordingly.
> 

The cloud is for automation. Said administrator should be writing tests
that ensure they're running the software and configuration that they
think they are. We should also be writing tests for them and shipping
said tests with each and every update, as that is something generally
consumable that will have a massive positive impact on trust in OpenStack.

> We're also making the administrator responsible for managing the 
> sequence in which incremental updates are deployed. Since each 
> incremetal update will re-write a particular set of files, any machine 
> which gets updates 1,2, 3, there's an oversight, and then update 5 is 
> deployed would end up in an odd state, which would require additional 
> tooling to detect. Package based updates, with versioning and dependency 
> tracking on each package, mitigate that risk.
> 

Yes, a carefully constructed set of packages can know all of the
version combinations and they have asserted their dependen

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Chris Jones]

Hi

Not a tarball. The node would notice from Heat metadata that it should update 
to a new image and would fetch that image and sync the contents to its /. This 
would leave it bit for bit identical to a fresh deployment of the new image, at 
least on disk. The running state would differ and that still requires some 
design and implementation to figure out.

Cheers,
--
Chris Jones

> On 23 Jan 2014, at 12:57, Angus Thomas  wrote:
> 
> On 22/01/14 20:54, Clint Byrum wrote:
>>> >
>>> >I don't understand the aversion to using existing, well-known tools to 
>>> >handle this?
>>> >
>> These tools are of course available to users and nobody is stopping them
>> from using them. We are optimizing for not needing them. They are there
>> and we're not going to explode if you use them. You just lose one aspect
>> of what we're aiming at. I believe that having image based deploys will
>> be well received as long as it is simple to understand.
>> 
>>> >A hybrid model (blending 2 and 3, above) here I think would work best where
>>> >TripleO lays down a baseline image and the cloud operator would employ an 
>>> >well-known
>>> >and support configuration tool for any small diffs.
>>> >
>> These tools are popular because they control entropy and make it at
>> least more likely that what you tested ends up on the boxes.
>> 
>> A read-only root partition is a much stronger control on entropy.
>> 
>>> >The operator would then be empowered to make the call for any major 
>>> >upgrades that
>>> >would adversely impact the infrastructure (and ultimately the users/apps). 
>>> > He/She
>>> >could say, this is a major release, let's deploy the image.
>>> >
>>> >Something logically like this, seems reasonable:
>>> >
>>> > if (system_change > 10%) {
>>> >   use TripleO;
>>> >   } else {
>>> >   use Existing_Config_Management;
>>> > }
>>> >
>> I think we can make deploying minor updates minimally invasive.
>> 
>> We've kept it simple enough, this should be a fairly straight forward
>> optimization cycle. And the win there is that we also improve things
>> for the 11% change.
> 
> Hi Clint,
> 
> For deploying minimally-invasive minor updates, the idea, if I've understood 
> it correctly, would be to deploy a tarball which replaced selected files on 
> the (usually read-only) root filesystem. That would allow for selective 
> restarting of only the services which are directly affected. The alternative, 
> pushing out a complete root filesystem image, would necessitate the same 
> amount of disruption in all cases.
> 
> There are a handful of costs with that approach which concern me: It 
> simplifies the deployment itself, but increases the complexity of preparing 
> the deployment. The administrator is going to have to identify the services 
> which need to be restarted, based on the particular set of libraries which 
> are touched in their partial update, and put together the service restart 
> scripts accordingly.
> 
> We're also making the administrator responsible for managing the sequence in 
> which incremental updates are deployed. Since each incremetal update will 
> re-write a particular set of files, any machine which gets updates 1,2, 3, 
> there's an oversight, and then update 5 is deployed would end up in an odd 
> state, which would require additional tooling to detect. Package based 
> updates, with versioning and dependency tracking on each package, mitigate 
> that risk.
> 
> Then there's the relationship between the state of running machines, with 
> applied partial updates, and the images which are put onto new machines by 
> Ironic. We would need to apply the partial updates to the images which Ironic 
> writes, or to have the tooling to ensure that newly deployed machines 
> immediately apply the set of applicable partial updates, in sequence.
> 
> Solving these issues feels like it'll require quite a lot of additional 
> tooling.
> 
> 
> Angus
> 
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Angus Thomas]

On 22/01/14 20:54, Clint Byrum wrote:

>
>I don't understand the aversion to using existing, well-known tools to handle 
this?
>

These tools are of course available to users and nobody is stopping them
from using them. We are optimizing for not needing them. They are there
and we're not going to explode if you use them. You just lose one aspect
of what we're aiming at. I believe that having image based deploys will
be well received as long as it is simple to understand.


>A hybrid model (blending 2 and 3, above) here I think would work best where
>TripleO lays down a baseline image and the cloud operator would employ an 
well-known
>and support configuration tool for any small diffs.
>

These tools are popular because they control entropy and make it at
least more likely that what you tested ends up on the boxes.

A read-only root partition is a much stronger control on entropy.


>The operator would then be empowered to make the call for any major upgrades 
that
>would adversely impact the infrastructure (and ultimately the users/apps).  
He/She
>could say, this is a major release, let's deploy the image.
>
>Something logically like this, seems reasonable:
>
> if (system_change > 10%) {
>   use TripleO;
>   } else {
>   use Existing_Config_Management;
> }
>

I think we can make deploying minor updates minimally invasive.

We've kept it simple enough, this should be a fairly straight forward
optimization cycle. And the win there is that we also improve things
for the 11% change.



Hi Clint,

For deploying minimally-invasive minor updates, the idea, if I've 
understood it correctly, would be to deploy a tarball which replaced 
selected files on the (usually read-only) root filesystem. That would 
allow for selective restarting of only the services which are directly 
affected. The alternative, pushing out a complete root filesystem image, 
would necessitate the same amount of disruption in all cases.


There are a handful of costs with that approach which concern me: It 
simplifies the deployment itself, but increases the complexity of 
preparing the deployment. The administrator is going to have to identify 
the services which need to be restarted, based on the particular set of 
libraries which are touched in their partial update, and put together 
the service restart scripts accordingly.


We're also making the administrator responsible for managing the 
sequence in which incremental updates are deployed. Since each 
incremetal update will re-write a particular set of files, any machine 
which gets updates 1,2, 3, there's an oversight, and then update 5 is 
deployed would end up in an odd state, which would require additional 
tooling to detect. Package based updates, with versioning and dependency 
tracking on each package, mitigate that risk.


Then there's the relationship between the state of running machines, 
with applied partial updates, and the images which are put onto new 
machines by Ironic. We would need to apply the partial updates to the 
images which Ironic writes, or to have the tooling to ensure that newly 
deployed machines immediately apply the set of applicable partial 
updates, in sequence.


Solving these issues feels like it'll require quite a lot of additional 
tooling.



Angus




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Chris Jones]

Hi

On 22 January 2014 21:33, Fox, Kevin M  wrote:

> I think we're pretty far off in a tangent though. My main point was, if
> you can't selectively restart services as needed, I'm not sure how useful
> patching the image really is over a full reboot. It should take on the same
> order of magnitude service unavailability I think.


The in-place upgrade (currently known as "takeovernode") is not yet well
designed and while there is a script in tripleo-incubator called
takeovernode, nobody is likely to resume working on it until a little later
in our roadmap. The crude hack we have atm does no detection of services
that need to be restarted, but that is not intended to suggest that we
don't care about such a feature :)

I think Clint has covered pretty much all the bases here, but I would
re-iterate that in no way do we think the kind of upgrade we're working on
at the moment (i.e. a nova rebuild driven instance reboot) is the only one
that should exist. We know that in-place upgrades need to happen for
tripleo's full story to be taken seriously, and we will get to it. If folk
have suggestions for behaviours/techniques/tools, those would be great to
capture, probably in https://etherpad.openstack.org/p/tripleo-image-updates
. http://manpages.ubuntu.com/manpages/oneiric/man1/checkrestart.1.html is
one such tool that we turned up in earlier research about how to detect
services that need to be restarted after an upgrade. It's not a complete
solution on its own, but it gets us some of the way.

(Also, just because we favour low-entropy golden images for all software
changes, doesn't mean that any given user can't choose to roll out an
upgrade to some piece(s) of software via any other mechanism they choose,
if that is what they feel is right for their operation. A combination of
the two strategies is entirely possible).

-- 
Cheers,

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Fox, Kevin M]

Maybe I misunderstand, but I thought:

kexec - lets you boot a new kernel/initrd starting at the point a boot loader 
would skipping the bios init. All previous running processes are not running in 
the new boot just like a normal reboot.

CRIU - Lets you snapshot/restart running processes.

While you could use both together to upgrades kernel while leaving all the 
processes running after the reboot,  I don't think that's very tested at the 
moment. checkpointing the system memory is not without cost too. Restarting the 
services may be faster.

I think we're pretty far off in a tangent though. My main point was, if you 
can't selectively restart services as needed, I'm not sure how useful patching 
the image really is over a full reboot. It should take on the same order of 
magnitude service unavailability I think.

Thanks,
Kevin


From: Clint Byrum [cl...@fewbar.com]
Sent: Wednesday, January 22, 2014 12:36 PM
To: openstack-dev
Subject: Re: [openstack-dev] [TripleO] our update story: can people live
with it?

Excerpts from Fox, Kevin M's message of 2014-01-22 12:19:56 -0800:
> I think most of the time taken to reboot is spent in bringing down/up the 
> services though, so I'm not sure what it really buys you if you do it all. It 
> may let you skip the crazy long bootup time on "enterprise" hardware, but 
> that could be worked around with kexec on the full reboot method too.
>

If we could get kexec reliable.. but I have no evidence that it is
anything but a complete flake.

What it saves you is losing running processes that you don't end up
killing, which is expensive on many types of services.. Nova Compute
being a notable example.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Keith Basil's message of 2014-01-22 12:27:50 -0800:
> On Jan 22, 2014, at 1:53 PM, Jay Pipes wrote:
> 
> > On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote:
> >> 
> >> - Original Message -
> >>> From: "Clint Byrum" 
> >>> To: "openstack-dev" 
> >>> Sent: Wednesday, January 22, 2014 12:45:45 PM
> >>> Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
> >>>   with it?
> >>> 
> >>> Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> >>>> I've been thinking a bit more about how TripleO updates are developing
> >>>> specifically with regards to compute nodes. What is commonly called the
> >>>> "update story" I think.
> >>>> 
> >>>> As I understand it we expect people to actually have to reboot a compute
> >>>> node in the cluster in order to deploy an update. This really worries me
> >>>> because it seems like way overkill for such a simple operation. Lets say
> >>>> all I need to deploy is a simple change to Nova's libvirt driver. And
> >>>> I need to deploy it to *all* my compute instances. Do we really expect
> >>>> people to actually have to reboot every single compute node in their
> >>>> cluster for such a thing. And then do this again and again for each
> >>>> update they deploy?
> >>>> 
> >>> 
> >>> Agreed, if we make everybody reboot to push out a patch to libvirt, we
> >>> have failed. And thus far, we are failing to do that, but with good
> >>> reason.
> >>> 
> >>> Right at this very moment, we are leaning on 'rebuild' in Nova, which
> >>> reboots the instance. But this is so that we handle the hardest thing
> >>> well first (rebooting to have a new kernel).
> >>> 
> >>> For small updates we need to decouple things a bit more. There is a
> >>> notion of the image ID in Nova, versus the image ID that is actually
> >>> running. Right now we update it with a nova rebuild command only.
> >>> 
> >>> But ideally we would give operators a tool to optimize and avoid the
> >>> reboot when it is appropriate. The heuristic should be as simple as
> >>> comparing kernels.
> >> 
> >> When we get to implementing such a thing I might prefer it not to be 
> >> auto-magic. I can see a case where I want the new image but maybe not the 
> >> new kernel. Perhaps this should be addressed when building the image (by 
> >> using the older kernel)... but still. I could see a case for explicitly 
> >> not wanting to reboot here as well.
> > 
> > ++
> > 
> >>> Once we have determined that a new image does not
> >>> need a reboot, we can just change the ID in Metadata, and an
> >>> os-refresh-config script will do something like this:
> >>> 
> >>> if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
> >>> then;
> >>>download_new_image
> >>>mount_image /tmp/new_image
> >>>mount / -o remount,rw # Assuming we've achieved ro root
> >>>rsync --one-file-system -a /tmp/new_image/ /
> >>>mount / -o remount,ro # ditto
> >>> fi
> >>> 
> >>> No reboot required. This would run early in configure.d, so that any
> >>> pre-configure.d scripts will have run to quiesce services that can't
> >>> handle having their binaries removed out from under them (read:
> >>> non-Unix services). Then configure.d runs as usual, configures things,
> >>> restarts services, and we are now running the new image.
> >> 
> >> Cool. I like this a good bit better as it avoids the reboot. Still, this 
> >> is a rather large amount of data to copy around if I'm only changing a 
> >> single file in Nova.
> > 
> > Right.
> > 
> >>> 
> >>>> I understand the whole read only images thing plays into this too... but
> >>>> I'm wondering if there is a middle ground where things might work
> >>>> better. Perhaps we have a mechanism where we can tar up individual venvs
> >>>> from /opt/stack/ or perhaps also this is an area where real OpenStack
> >>>> packages could shine. It seems like we could certainly come up with some
> >>>> simple mechanisms 

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Jay Pipes]

On Wed, 2014-01-22 at 12:12 -0800, Clint Byrum wrote:
> Excerpts from Jay Pipes's message of 2014-01-22 10:53:14 -0800:
> > On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote:
> > > 
> > > - Original Message -
> > > > From: "Clint Byrum" 
> > > > To: "openstack-dev" 
> > > > Sent: Wednesday, January 22, 2014 12:45:45 PM
> > > > Subject: Re: [openstack-dev] [TripleO] our update story: can people 
> > > > livewith it?
> > > > 
> > > > Given the scenario above, that would be a further optimization. I don't
> > > > think it makes sense to specialize for venvs or openstack services
> > > > though, so just "ensure the root filesystems match" seems like a
> > > > workable, highly efficient system. Note that we've talked about having
> > > > highly efficient ways to widely distribute the new images as well.
> > > 
> > > Yes. Optimization! In the big scheme of things I could see 3 approaches 
> > > being useful:
> > > 
> > > 1) Deploy a full image and reboot if you have a kernel update. (entire 
> > > image is copied)
> > > 
> > > 2) Deploy a full image if you change a bunch of things and/or you prefer 
> > > to do that. (entire image is copied)
> > > 
> > > 3) Deploy specific application level updates via packages or tarballs. 
> > > (only selected applications/packages get deployed)
> > 
> > ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD
> > environments, so this level of optimization will be frequently used.
> > And, as I've said before, optimizing for frequently-used scenarios is
> > worth spending the time on. Optimizing for infrequently-occurring
> > things... not so much. :)
> 
> I do understand that little tweaks are more common than whole software
> updates.
> 
> I also think that little tweaks must be tested just like big ones.
> 
> So I would argue that it is more important to optimize for trusting that
> what you tested is what is in production, and then to address any issues
> if that work-flow needs optimization. A system that leaves operators
> afraid to do a big update because "it will trigger the bad path" is a
> system that doesn't handle big updates well.
> 
> Ideally we'd optimize all 3 in all of the obvious ways before determining
> that the "one file" update just isn't fast enough.

Well said. No disagreement from me.

Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2014-01-22 12:19:56 -0800:
> I think most of the time taken to reboot is spent in bringing down/up the 
> services though, so I'm not sure what it really buys you if you do it all. It 
> may let you skip the crazy long bootup time on "enterprise" hardware, but 
> that could be worked around with kexec on the full reboot method too.
> 

If we could get kexec reliable.. but I have no evidence that it is
anything but a complete flake.

What it saves you is losing running processes that you don't end up
killing, which is expensive on many types of services.. Nova Compute
being a notable example.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Keith Basil]

On Jan 22, 2014, at 1:53 PM, Jay Pipes wrote:

> On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote:
>> 
>> - Original Message -
>>> From: "Clint Byrum" 
>>> To: "openstack-dev" 
>>> Sent: Wednesday, January 22, 2014 12:45:45 PM
>>> Subject: Re: [openstack-dev] [TripleO] our update story: can people live
>>> with it?
>>> 
>>> Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
>>>> I've been thinking a bit more about how TripleO updates are developing
>>>> specifically with regards to compute nodes. What is commonly called the
>>>> "update story" I think.
>>>> 
>>>> As I understand it we expect people to actually have to reboot a compute
>>>> node in the cluster in order to deploy an update. This really worries me
>>>> because it seems like way overkill for such a simple operation. Lets say
>>>> all I need to deploy is a simple change to Nova's libvirt driver. And
>>>> I need to deploy it to *all* my compute instances. Do we really expect
>>>> people to actually have to reboot every single compute node in their
>>>> cluster for such a thing. And then do this again and again for each
>>>> update they deploy?
>>>> 
>>> 
>>> Agreed, if we make everybody reboot to push out a patch to libvirt, we
>>> have failed. And thus far, we are failing to do that, but with good
>>> reason.
>>> 
>>> Right at this very moment, we are leaning on 'rebuild' in Nova, which
>>> reboots the instance. But this is so that we handle the hardest thing
>>> well first (rebooting to have a new kernel).
>>> 
>>> For small updates we need to decouple things a bit more. There is a
>>> notion of the image ID in Nova, versus the image ID that is actually
>>> running. Right now we update it with a nova rebuild command only.
>>> 
>>> But ideally we would give operators a tool to optimize and avoid the
>>> reboot when it is appropriate. The heuristic should be as simple as
>>> comparing kernels.
>> 
>> When we get to implementing such a thing I might prefer it not to be 
>> auto-magic. I can see a case where I want the new image but maybe not the 
>> new kernel. Perhaps this should be addressed when building the image (by 
>> using the older kernel)... but still. I could see a case for explicitly not 
>> wanting to reboot here as well.
> 
> ++
> 
>>> Once we have determined that a new image does not
>>> need a reboot, we can just change the ID in Metadata, and an
>>> os-refresh-config script will do something like this:
>>> 
>>> if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
>>> then;
>>>download_new_image
>>>mount_image /tmp/new_image
>>>mount / -o remount,rw # Assuming we've achieved ro root
>>>rsync --one-file-system -a /tmp/new_image/ /
>>>mount / -o remount,ro # ditto
>>> fi
>>> 
>>> No reboot required. This would run early in configure.d, so that any
>>> pre-configure.d scripts will have run to quiesce services that can't
>>> handle having their binaries removed out from under them (read:
>>> non-Unix services). Then configure.d runs as usual, configures things,
>>> restarts services, and we are now running the new image.
>> 
>> Cool. I like this a good bit better as it avoids the reboot. Still, this is 
>> a rather large amount of data to copy around if I'm only changing a single 
>> file in Nova.
> 
> Right.
> 
>>> 
>>>> I understand the whole read only images thing plays into this too... but
>>>> I'm wondering if there is a middle ground where things might work
>>>> better. Perhaps we have a mechanism where we can tar up individual venvs
>>>> from /opt/stack/ or perhaps also this is an area where real OpenStack
>>>> packages could shine. It seems like we could certainly come up with some
>>>> simple mechanisms to deploy these sorts of changes with Heat such that
>>>> compute host reboot can be avoided for each new deploy.
>>> 
>>> Given the scenario above, that would be a further optimization. I don't
>>> think it makes sense to specialize for venvs or openstack services
>>> though, so just "ensure the root filesystems match" seems like a
>>> workable, highly efficient system. Note that we've talked about havi

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Fox, Kevin M]

I think most of the time taken to reboot is spent in bringing down/up the 
services though, so I'm not sure what it really buys you if you do it all. It 
may let you skip the crazy long bootup time on "enterprise" hardware, but that 
could be worked around with kexec on the full reboot method too.

Thanks,
Kevin

From: Clint Byrum [cl...@fewbar.com]
Sent: Wednesday, January 22, 2014 10:55 AM
To: openstack-dev
Subject: Re: [openstack-dev] [TripleO] our update story: can people live    
with it?

Agreed, it is tricky if we try to only restart what we've changed.

OR, just restart everything. We can make endpoints HA and use rolling
updates to avoid spurious faults.

There are complex ways to handle things even smoother.. but I go back to
"What does complexity cost?"

Excerpts from Fox, Kevin M's message of 2014-01-22 10:32:02 -0800:
> Another tricky bit left is how to handle service restarts as needed?
>
> Thanks,
> Kevin
> 
> From: Dan Prince [dpri...@redhat.com]
> Sent: Wednesday, January 22, 2014 10:15 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
>   with it?
>
> - Original Message -
> > From: "Clint Byrum" 
> > To: "openstack-dev" 
> > Sent: Wednesday, January 22, 2014 12:45:45 PM
> > Subject: Re: [openstack-dev] [TripleO] our update story: can people live
> >   with it?
> >
> > Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> > > I've been thinking a bit more about how TripleO updates are developing
> > > specifically with regards to compute nodes. What is commonly called the
> > > "update story" I think.
> > >
> > > As I understand it we expect people to actually have to reboot a compute
> > > node in the cluster in order to deploy an update. This really worries me
> > > because it seems like way overkill for such a simple operation. Lets say
> > > all I need to deploy is a simple change to Nova's libvirt driver. And
> > > I need to deploy it to *all* my compute instances. Do we really expect
> > > people to actually have to reboot every single compute node in their
> > > cluster for such a thing. And then do this again and again for each
> > > update they deploy?
> > >
> >
> > Agreed, if we make everybody reboot to push out a patch to libvirt, we
> > have failed. And thus far, we are failing to do that, but with good
> > reason.
> >
> > Right at this very moment, we are leaning on 'rebuild' in Nova, which
> > reboots the instance. But this is so that we handle the hardest thing
> > well first (rebooting to have a new kernel).
> >
> > For small updates we need to decouple things a bit more. There is a
> > notion of the image ID in Nova, versus the image ID that is actually
> > running. Right now we update it with a nova rebuild command only.
> >
> > But ideally we would give operators a tool to optimize and avoid the
> > reboot when it is appropriate. The heuristic should be as simple as
> > comparing kernels.
>
> When we get to implementing such a thing I might prefer it not to be 
> auto-magic. I can see a case where I want the new image but maybe not the new 
> kernel. Perhaps this should be addressed when building the image (by using 
> the older kernel)... but still. I could see a case for explicitly not wanting 
> to reboot here as well.
>
> > Once we have determined that a new image does not
> > need a reboot, we can just change the ID in Metadata, and an
> > os-refresh-config script will do something like this:
> >
> > if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
> > then;
> > download_new_image
> > mount_image /tmp/new_image
> > mount / -o remount,rw # Assuming we've achieved ro root
> > rsync --one-file-system -a /tmp/new_image/ /
> > mount / -o remount,ro # ditto
> > fi
> >
> > No reboot required. This would run early in configure.d, so that any
> > pre-configure.d scripts will have run to quiesce services that can't
> > handle having their binaries removed out from under them (read:
> > non-Unix services). Then configure.d runs as usual, configures things,
> > restarts services, and we are now running the new image.
>
> Cool. I like this a good bit better as it avoids the reboot. Still, this is a 
> rather large amount of data to copy around if I'm only changing a single file 
>

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Jay Pipes's message of 2014-01-22 10:53:14 -0800:
> On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote:
> > 
> > - Original Message -
> > > From: "Clint Byrum" 
> > > To: "openstack-dev" 
> > > Sent: Wednesday, January 22, 2014 12:45:45 PM
> > > Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
> > >   with it?
> > > 
> > > Given the scenario above, that would be a further optimization. I don't
> > > think it makes sense to specialize for venvs or openstack services
> > > though, so just "ensure the root filesystems match" seems like a
> > > workable, highly efficient system. Note that we've talked about having
> > > highly efficient ways to widely distribute the new images as well.
> > 
> > Yes. Optimization! In the big scheme of things I could see 3 approaches 
> > being useful:
> > 
> > 1) Deploy a full image and reboot if you have a kernel update. (entire 
> > image is copied)
> > 
> > 2) Deploy a full image if you change a bunch of things and/or you prefer to 
> > do that. (entire image is copied)
> > 
> > 3) Deploy specific application level updates via packages or tarballs. 
> > (only selected applications/packages get deployed)
> 
> ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD
> environments, so this level of optimization will be frequently used.
> And, as I've said before, optimizing for frequently-used scenarios is
> worth spending the time on. Optimizing for infrequently-occurring
> things... not so much. :)

I do understand that little tweaks are more common than whole software
updates.

I also think that little tweaks must be tested just like big ones.

So I would argue that it is more important to optimize for trusting that
what you tested is what is in production, and then to address any issues
if that work-flow needs optimization. A system that leaves operators
afraid to do a big update because "it will trigger the bad path" is a
system that doesn't handle big updates well.

Ideally we'd optimize all 3 in all of the obvious ways before determining
that the "one file" update just isn't fast enough.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Agreed, it is tricky if we try to only restart what we've changed.

OR, just restart everything. We can make endpoints HA and use rolling
updates to avoid spurious faults.

There are complex ways to handle things even smoother.. but I go back to
"What does complexity cost?"

Excerpts from Fox, Kevin M's message of 2014-01-22 10:32:02 -0800:
> Another tricky bit left is how to handle service restarts as needed?
> 
> Thanks,
> Kevin
> 
> From: Dan Prince [dpri...@redhat.com]
> Sent: Wednesday, January 22, 2014 10:15 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
>   with it?
> 
> - Original Message -
> > From: "Clint Byrum" 
> > To: "openstack-dev" 
> > Sent: Wednesday, January 22, 2014 12:45:45 PM
> > Subject: Re: [openstack-dev] [TripleO] our update story: can people live
> >   with it?
> >
> > Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> > > I've been thinking a bit more about how TripleO updates are developing
> > > specifically with regards to compute nodes. What is commonly called the
> > > "update story" I think.
> > >
> > > As I understand it we expect people to actually have to reboot a compute
> > > node in the cluster in order to deploy an update. This really worries me
> > > because it seems like way overkill for such a simple operation. Lets say
> > > all I need to deploy is a simple change to Nova's libvirt driver. And
> > > I need to deploy it to *all* my compute instances. Do we really expect
> > > people to actually have to reboot every single compute node in their
> > > cluster for such a thing. And then do this again and again for each
> > > update they deploy?
> > >
> >
> > Agreed, if we make everybody reboot to push out a patch to libvirt, we
> > have failed. And thus far, we are failing to do that, but with good
> > reason.
> >
> > Right at this very moment, we are leaning on 'rebuild' in Nova, which
> > reboots the instance. But this is so that we handle the hardest thing
> > well first (rebooting to have a new kernel).
> >
> > For small updates we need to decouple things a bit more. There is a
> > notion of the image ID in Nova, versus the image ID that is actually
> > running. Right now we update it with a nova rebuild command only.
> >
> > But ideally we would give operators a tool to optimize and avoid the
> > reboot when it is appropriate. The heuristic should be as simple as
> > comparing kernels.
> 
> When we get to implementing such a thing I might prefer it not to be 
> auto-magic. I can see a case where I want the new image but maybe not the new 
> kernel. Perhaps this should be addressed when building the image (by using 
> the older kernel)... but still. I could see a case for explicitly not wanting 
> to reboot here as well.
> 
> > Once we have determined that a new image does not
> > need a reboot, we can just change the ID in Metadata, and an
> > os-refresh-config script will do something like this:
> >
> > if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
> > then;
> > download_new_image
> > mount_image /tmp/new_image
> > mount / -o remount,rw # Assuming we've achieved ro root
> > rsync --one-file-system -a /tmp/new_image/ /
> > mount / -o remount,ro # ditto
> > fi
> >
> > No reboot required. This would run early in configure.d, so that any
> > pre-configure.d scripts will have run to quiesce services that can't
> > handle having their binaries removed out from under them (read:
> > non-Unix services). Then configure.d runs as usual, configures things,
> > restarts services, and we are now running the new image.
> 
> Cool. I like this a good bit better as it avoids the reboot. Still, this is a 
> rather large amount of data to copy around if I'm only changing a single file 
> in Nova.
> 
> >
> > > I understand the whole read only images thing plays into this too... but
> > > I'm wondering if there is a middle ground where things might work
> > > better. Perhaps we have a mechanism where we can tar up individual venvs
> > > from /opt/stack/ or perhaps also this is an area where real OpenStack
> > > packages could shine. It seems like we could certainly come up with some
> > > simple mechanisms to deploy these sorts of changes with Heat such that
> &

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Jay Pipes]

On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote:
> 
> - Original Message -
> > From: "Clint Byrum" 
> > To: "openstack-dev" 
> > Sent: Wednesday, January 22, 2014 12:45:45 PM
> > Subject: Re: [openstack-dev] [TripleO] our update story: can people live
> > with it?
> > 
> > Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> > > I've been thinking a bit more about how TripleO updates are developing
> > > specifically with regards to compute nodes. What is commonly called the
> > > "update story" I think.
> > > 
> > > As I understand it we expect people to actually have to reboot a compute
> > > node in the cluster in order to deploy an update. This really worries me
> > > because it seems like way overkill for such a simple operation. Lets say
> > > all I need to deploy is a simple change to Nova's libvirt driver. And
> > > I need to deploy it to *all* my compute instances. Do we really expect
> > > people to actually have to reboot every single compute node in their
> > > cluster for such a thing. And then do this again and again for each
> > > update they deploy?
> > > 
> > 
> > Agreed, if we make everybody reboot to push out a patch to libvirt, we
> > have failed. And thus far, we are failing to do that, but with good
> > reason.
> > 
> > Right at this very moment, we are leaning on 'rebuild' in Nova, which
> > reboots the instance. But this is so that we handle the hardest thing
> > well first (rebooting to have a new kernel).
> > 
> > For small updates we need to decouple things a bit more. There is a
> > notion of the image ID in Nova, versus the image ID that is actually
> > running. Right now we update it with a nova rebuild command only.
> > 
> > But ideally we would give operators a tool to optimize and avoid the
> > reboot when it is appropriate. The heuristic should be as simple as
> > comparing kernels.
> 
> When we get to implementing such a thing I might prefer it not to be 
> auto-magic. I can see a case where I want the new image but maybe not the new 
> kernel. Perhaps this should be addressed when building the image (by using 
> the older kernel)... but still. I could see a case for explicitly not wanting 
> to reboot here as well.

++

> > Once we have determined that a new image does not
> > need a reboot, we can just change the ID in Metadata, and an
> > os-refresh-config script will do something like this:
> > 
> > if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
> > then;
> > download_new_image
> > mount_image /tmp/new_image
> > mount / -o remount,rw # Assuming we've achieved ro root
> > rsync --one-file-system -a /tmp/new_image/ /
> > mount / -o remount,ro # ditto
> > fi
> > 
> > No reboot required. This would run early in configure.d, so that any
> > pre-configure.d scripts will have run to quiesce services that can't
> > handle having their binaries removed out from under them (read:
> > non-Unix services). Then configure.d runs as usual, configures things,
> > restarts services, and we are now running the new image.
> 
> Cool. I like this a good bit better as it avoids the reboot. Still, this is a 
> rather large amount of data to copy around if I'm only changing a single file 
> in Nova.

Right.

> > 
> > > I understand the whole read only images thing plays into this too... but
> > > I'm wondering if there is a middle ground where things might work
> > > better. Perhaps we have a mechanism where we can tar up individual venvs
> > > from /opt/stack/ or perhaps also this is an area where real OpenStack
> > > packages could shine. It seems like we could certainly come up with some
> > > simple mechanisms to deploy these sorts of changes with Heat such that
> > > compute host reboot can be avoided for each new deploy.
> > 
> > Given the scenario above, that would be a further optimization. I don't
> > think it makes sense to specialize for venvs or openstack services
> > though, so just "ensure the root filesystems match" seems like a
> > workable, highly efficient system. Note that we've talked about having
> > highly efficient ways to widely distribute the new images as well.
> 
> Yes. Optimization! In the big scheme of things I could see 3 approaches being 
> useful:
> 
> 1) Deploy a full image and reboot if you have a kernel update. (entire image 
> is copied)
> 
> 2)

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Dan Prince's message of 2014-01-22 10:15:20 -0800:
> 
> - Original Message -
> > From: "Clint Byrum" 
> > To: "openstack-dev" 
> > Sent: Wednesday, January 22, 2014 12:45:45 PM
> > Subject: Re: [openstack-dev] [TripleO] our update story: can people live
> > with it?
> > 
> > Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> > > I've been thinking a bit more about how TripleO updates are developing
> > > specifically with regards to compute nodes. What is commonly called the
> > > "update story" I think.
> > > 
> > > As I understand it we expect people to actually have to reboot a compute
> > > node in the cluster in order to deploy an update. This really worries me
> > > because it seems like way overkill for such a simple operation. Lets say
> > > all I need to deploy is a simple change to Nova's libvirt driver. And
> > > I need to deploy it to *all* my compute instances. Do we really expect
> > > people to actually have to reboot every single compute node in their
> > > cluster for such a thing. And then do this again and again for each
> > > update they deploy?
> > > 
> > 
> > Agreed, if we make everybody reboot to push out a patch to libvirt, we
> > have failed. And thus far, we are failing to do that, but with good
> > reason.
> > 
> > Right at this very moment, we are leaning on 'rebuild' in Nova, which
> > reboots the instance. But this is so that we handle the hardest thing
> > well first (rebooting to have a new kernel).
> > 
> > For small updates we need to decouple things a bit more. There is a
> > notion of the image ID in Nova, versus the image ID that is actually
> > running. Right now we update it with a nova rebuild command only.
> > 
> > But ideally we would give operators a tool to optimize and avoid the
> > reboot when it is appropriate. The heuristic should be as simple as
> > comparing kernels.
> 
> When we get to implementing such a thing I might prefer it not to be 
> auto-magic. I can see a case where I want the new image but maybe not the new 
> kernel. Perhaps this should be addressed when building the image (by using 
> the older kernel)... but still. I could see a case for explicitly not wanting 
> to reboot here as well.
> 

I prefer choosing what to update at image build time. This is the time
where it is most clear how, from a developer and deployer standpoint,
to influence the update. I can diff images, I can freeze mirrors.. etc.
etc, all decoupled from anybody else and from production or test cycles.

That said, I do think it would be good for deployers to be able have
a way to control when reboots are and aren't allowed. That seems like
policy, which may be best handled in Nova.. so we can have a user that
can do updates to Heat Metadata/stacks, but not rebuilds in Nova.

I have no idea of Heat's trust model will allow us to have such separation
though.

> > Once we have determined that a new image does not
> > need a reboot, we can just change the ID in Metadata, and an
> > os-refresh-config script will do something like this:
> > 
> > if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
> > then;
> > download_new_image
> > mount_image /tmp/new_image
> > mount / -o remount,rw # Assuming we've achieved ro root
> > rsync --one-file-system -a /tmp/new_image/ /
> > mount / -o remount,ro # ditto
> > fi
> > 
> > No reboot required. This would run early in configure.d, so that any
> > pre-configure.d scripts will have run to quiesce services that can't
> > handle having their binaries removed out from under them (read:
> > non-Unix services). Then configure.d runs as usual, configures things,
> > restarts services, and we are now running the new image.
> 
> Cool. I like this a good bit better as it avoids the reboot. Still, this is a 
> rather large amount of data to copy around if I'm only changing a single file 
> in Nova.
> 

I think in most cases transfer cost is worth it to know you're deploying
what you tested. Also it is pretty easy to just do this optimization
but still be rsyncing the contents of the image. Instead of downloading
the whole thing we could have a box expose the mounted image via rsync
and then all of the machines can just rsync changes. Also rsync has a
batch mode where if you know for sure the end-state of machines you can
pre-calculate that rsync and just ship that. Lots of optimization
possible that will work fine in your just-update-one-file scenario.

But re

[openstack-dev] [TripleO] our update story: can people live with it?

[Charles Crouch]

Clint wrote:
> I would call your e-mail a documentation/roadmap bug. This plan may
> have been recorded somewhere, but for me it has just always been in my
> head as the end goal (thanks to Robert Collins for drilling the hole
> and pouring it in there btw ;).

I found the following links helpful in getting a better idea of how updates 
could work:

https://etherpad.openstack.org/p/tripleo-image-updates
https://github.com/openstack/tripleo-incubator/blob/master/scripts/takeovernode

/hat-tip to lifeless for the pointers.

Cheers
Charles

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Russell Bryant]

On 01/22/2014 12:17 PM, Dan Prince wrote:
> I've been thinking a bit more about how TripleO updates are developing 
> specifically with regards to compute nodes. What is commonly called the 
> "update story" I think.
> 
> As I understand it we expect people to actually have to reboot a compute node 
> in the cluster in order to deploy an update. This really worries me because 
> it seems like way overkill for such a simple operation. Lets say all I need 
> to deploy is a simple change to Nova's libvirt driver. And I need to deploy 
> it to *all* my compute instances. Do we really expect people to actually have 
> to reboot every single compute node in their cluster for such a thing. And 
> then do this again and again for each update they deploy?

FWIW, I agree that this is going to be considered unacceptable by most
people.  Hopefully everyone is on the same page with that.  It sounds
like that's the case so far in this thread, at least...

If you have to reboot the compute node, ideally you also have support
for live migrating all running VMs on that compute node elsewhere before
doing so.  That's not something you want to have to do for *every*
little change to *every* compute node.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Fox, Kevin M]

Another tricky bit left is how to handle service restarts as needed?

Thanks,
Kevin

From: Dan Prince [dpri...@redhat.com]
Sent: Wednesday, January 22, 2014 10:15 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] our update story: can people live
with it?

- Original Message -
> From: "Clint Byrum" 
> To: "openstack-dev" 
> Sent: Wednesday, January 22, 2014 12:45:45 PM
> Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
> with it?
>
> Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> > I've been thinking a bit more about how TripleO updates are developing
> > specifically with regards to compute nodes. What is commonly called the
> > "update story" I think.
> >
> > As I understand it we expect people to actually have to reboot a compute
> > node in the cluster in order to deploy an update. This really worries me
> > because it seems like way overkill for such a simple operation. Lets say
> > all I need to deploy is a simple change to Nova's libvirt driver. And
> > I need to deploy it to *all* my compute instances. Do we really expect
> > people to actually have to reboot every single compute node in their
> > cluster for such a thing. And then do this again and again for each
> > update they deploy?
> >
>
> Agreed, if we make everybody reboot to push out a patch to libvirt, we
> have failed. And thus far, we are failing to do that, but with good
> reason.
>
> Right at this very moment, we are leaning on 'rebuild' in Nova, which
> reboots the instance. But this is so that we handle the hardest thing
> well first (rebooting to have a new kernel).
>
> For small updates we need to decouple things a bit more. There is a
> notion of the image ID in Nova, versus the image ID that is actually
> running. Right now we update it with a nova rebuild command only.
>
> But ideally we would give operators a tool to optimize and avoid the
> reboot when it is appropriate. The heuristic should be as simple as
> comparing kernels.

When we get to implementing such a thing I might prefer it not to be 
auto-magic. I can see a case where I want the new image but maybe not the new 
kernel. Perhaps this should be addressed when building the image (by using the 
older kernel)... but still. I could see a case for explicitly not wanting to 
reboot here as well.

> Once we have determined that a new image does not
> need a reboot, we can just change the ID in Metadata, and an
> os-refresh-config script will do something like this:
>
> if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
> then;
> download_new_image
> mount_image /tmp/new_image
> mount / -o remount,rw # Assuming we've achieved ro root
> rsync --one-file-system -a /tmp/new_image/ /
> mount / -o remount,ro # ditto
> fi
>
> No reboot required. This would run early in configure.d, so that any
> pre-configure.d scripts will have run to quiesce services that can't
> handle having their binaries removed out from under them (read:
> non-Unix services). Then configure.d runs as usual, configures things,
> restarts services, and we are now running the new image.

Cool. I like this a good bit better as it avoids the reboot. Still, this is a 
rather large amount of data to copy around if I'm only changing a single file 
in Nova.

>
> > I understand the whole read only images thing plays into this too... but
> > I'm wondering if there is a middle ground where things might work
> > better. Perhaps we have a mechanism where we can tar up individual venvs
> > from /opt/stack/ or perhaps also this is an area where real OpenStack
> > packages could shine. It seems like we could certainly come up with some
> > simple mechanisms to deploy these sorts of changes with Heat such that
> > compute host reboot can be avoided for each new deploy.
>
> Given the scenario above, that would be a further optimization. I don't
> think it makes sense to specialize for venvs or openstack services
> though, so just "ensure the root filesystems match" seems like a
> workable, highly efficient system. Note that we've talked about having
> highly efficient ways to widely distribute the new images as well.

Yes. Optimization! In the big scheme of things I could see 3 approaches being 
useful:

1) Deploy a full image and reboot if you have a kernel update. (entire image is 
copied)

2) Deploy a full image if you change a bunch of things and/or you prefer to do 
that. (entire image is copied)

3) Deploy specific applicatio

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Dan Prince]

- Original Message -
> From: "Clint Byrum" 
> To: "openstack-dev" 
> Sent: Wednesday, January 22, 2014 12:45:45 PM
> Subject: Re: [openstack-dev] [TripleO] our update story: can people live  
> with it?
> 
> Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> > I've been thinking a bit more about how TripleO updates are developing
> > specifically with regards to compute nodes. What is commonly called the
> > "update story" I think.
> > 
> > As I understand it we expect people to actually have to reboot a compute
> > node in the cluster in order to deploy an update. This really worries me
> > because it seems like way overkill for such a simple operation. Lets say
> > all I need to deploy is a simple change to Nova's libvirt driver. And
> > I need to deploy it to *all* my compute instances. Do we really expect
> > people to actually have to reboot every single compute node in their
> > cluster for such a thing. And then do this again and again for each
> > update they deploy?
> > 
> 
> Agreed, if we make everybody reboot to push out a patch to libvirt, we
> have failed. And thus far, we are failing to do that, but with good
> reason.
> 
> Right at this very moment, we are leaning on 'rebuild' in Nova, which
> reboots the instance. But this is so that we handle the hardest thing
> well first (rebooting to have a new kernel).
> 
> For small updates we need to decouple things a bit more. There is a
> notion of the image ID in Nova, versus the image ID that is actually
> running. Right now we update it with a nova rebuild command only.
> 
> But ideally we would give operators a tool to optimize and avoid the
> reboot when it is appropriate. The heuristic should be as simple as
> comparing kernels.

When we get to implementing such a thing I might prefer it not to be 
auto-magic. I can see a case where I want the new image but maybe not the new 
kernel. Perhaps this should be addressed when building the image (by using the 
older kernel)... but still. I could see a case for explicitly not wanting to 
reboot here as well.

> Once we have determined that a new image does not
> need a reboot, we can just change the ID in Metadata, and an
> os-refresh-config script will do something like this:
> 
> if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
> then;
> download_new_image
> mount_image /tmp/new_image
> mount / -o remount,rw # Assuming we've achieved ro root
> rsync --one-file-system -a /tmp/new_image/ /
> mount / -o remount,ro # ditto
> fi
> 
> No reboot required. This would run early in configure.d, so that any
> pre-configure.d scripts will have run to quiesce services that can't
> handle having their binaries removed out from under them (read:
> non-Unix services). Then configure.d runs as usual, configures things,
> restarts services, and we are now running the new image.

Cool. I like this a good bit better as it avoids the reboot. Still, this is a 
rather large amount of data to copy around if I'm only changing a single file 
in Nova.

> 
> > I understand the whole read only images thing plays into this too... but
> > I'm wondering if there is a middle ground where things might work
> > better. Perhaps we have a mechanism where we can tar up individual venvs
> > from /opt/stack/ or perhaps also this is an area where real OpenStack
> > packages could shine. It seems like we could certainly come up with some
> > simple mechanisms to deploy these sorts of changes with Heat such that
> > compute host reboot can be avoided for each new deploy.
> 
> Given the scenario above, that would be a further optimization. I don't
> think it makes sense to specialize for venvs or openstack services
> though, so just "ensure the root filesystems match" seems like a
> workable, highly efficient system. Note that we've talked about having
> highly efficient ways to widely distribute the new images as well.

Yes. Optimization! In the big scheme of things I could see 3 approaches being 
useful:

1) Deploy a full image and reboot if you have a kernel update. (entire image is 
copied)

2) Deploy a full image if you change a bunch of things and/or you prefer to do 
that. (entire image is copied)

3) Deploy specific application level updates via packages or tarballs. (only 
selected applications/packages get deployed)

> 
> I would call your e-mail a documentation/roadmap bug.

Fair enough. Thanks for the info.

> This plan may have been recorded somewhere, but for me it has just always 
> been in my
> head as the end goal (thanks to Robert Collins for drilling the hole
> and pouring it in there btw ;).
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] our update story: can people live with it?

[Clint Byrum]

Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800:
> I've been thinking a bit more about how TripleO updates are developing
> specifically with regards to compute nodes. What is commonly called the
> "update story" I think.
> 
> As I understand it we expect people to actually have to reboot a compute
> node in the cluster in order to deploy an update. This really worries me
> because it seems like way overkill for such a simple operation. Lets say
> all I need to deploy is a simple change to Nova's libvirt driver. And
> I need to deploy it to *all* my compute instances. Do we really expect
> people to actually have to reboot every single compute node in their
> cluster for such a thing. And then do this again and again for each
> update they deploy?
> 

Agreed, if we make everybody reboot to push out a patch to libvirt, we
have failed. And thus far, we are failing to do that, but with good
reason.

Right at this very moment, we are leaning on 'rebuild' in Nova, which
reboots the instance. But this is so that we handle the hardest thing
well first (rebooting to have a new kernel).

For small updates we need to decouple things a bit more. There is a
notion of the image ID in Nova, versus the image ID that is actually
running. Right now we update it with a nova rebuild command only.

But ideally we would give operators a tool to optimize and avoid the
reboot when it is appropriate. The heuristic should be as simple as
comparing kernels. Once we have determined that a new image does not
need a reboot, we can just change the ID in Metadata, and an
os-refresh-config script will do something like this:

if [ "$(cat /etc/image_id)" != "$(os-apply-config --key image_id)" ] ;
then;
download_new_image
mount_image /tmp/new_image
mount / -o remount,rw # Assuming we've achieved ro root
rsync --one-file-system -a /tmp/new_image/ /
mount / -o remount,ro # ditto
fi

No reboot required. This would run early in configure.d, so that any
pre-configure.d scripts will have run to quiesce services that can't
handle having their binaries removed out from under them (read:
non-Unix services). Then configure.d runs as usual, configures things,
restarts services, and we are now running the new image.

> I understand the whole read only images thing plays into this too... but
> I'm wondering if there is a middle ground where things might work
> better. Perhaps we have a mechanism where we can tar up individual venvs
> from /opt/stack/ or perhaps also this is an area where real OpenStack
> packages could shine. It seems like we could certainly come up with some
> simple mechanisms to deploy these sorts of changes with Heat such that
> compute host reboot can be avoided for each new deploy.

Given the scenario above, that would be a further optimization. I don't
think it makes sense to specialize for venvs or openstack services
though, so just "ensure the root filesystems match" seems like a
workable, highly efficient system. Note that we've talked about having
highly efficient ways to widely distribute the new images as well.

I would call your e-mail a documentation/roadmap bug. This plan may
have been recorded somewhere, but for me it has just always been in my
head as the end goal (thanks to Robert Collins for drilling the hole
and pouring it in there btw ;).

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] our update story: can people live with it?

[Dan Prince]

I've been thinking a bit more about how TripleO updates are developing 
specifically with regards to compute nodes. What is commonly called the "update 
story" I think.

As I understand it we expect people to actually have to reboot a compute node 
in the cluster in order to deploy an update. This really worries me because it 
seems like way overkill for such a simple operation. Lets say all I need to 
deploy is a simple change to Nova's libvirt driver. And I need to deploy it to 
*all* my compute instances. Do we really expect people to actually have to 
reboot every single compute node in their cluster for such a thing. And then do 
this again and again for each update they deploy?

I understand the whole read only images thing plays into this too... but I'm 
wondering if there is a middle ground where things might work better. Perhaps 
we have a mechanism where we can tar up individual venvs from /opt/stack/ or 
perhaps also this is an area where real OpenStack packages could shine. It 
seems like we could certainly come up with some simple mechanisms to deploy 
these sorts of changes with Heat such that compute host reboot can be avoided 
for each new deploy.

Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev