Re: [openstack-dev] [nova] Automatic Evacuation
Are there any resources available or proven examples on using external tools which call nova evacuate? For example use a monitoring tool to detect node failure and let the monitoring tool call evacuate on the instances which were running on the failed compute node. On Mon, Mar 3, 2014 at 11:28 PM, Jay Lau wrote: > > Yes, it would be great if we can have a simple framework for future run > time policy plugins. ;-) > > 2014-03-03 23:12 GMT+08:00 laserjetyang : > >> there are a lot of rules for HA or LB, so I think it might be a better >> idea to scope the framework and leave the policy as plugins. >> >> >> On Mon, Mar 3, 2014 at 10:30 PM, Andrew Laski > > wrote: >> >>> On 03/01/14 at 07:24am, Jay Lau wrote: >>> Hey, Sorry to bring this up again. There are also some discussions here: http://markmail.org/message/5zotly4qktaf34ei You can also search [Runtime Policy] in your email list. Not sure if we can put this to Gantt and enable Gantt provide both initial placement and rum time polices like HA, load balance etc. >>> >>> I don't have an opinion at the moment as to whether or not this sort of >>> functionality belongs in Gantt, but there's still a long way to go just to >>> get the scheduling functionality we want out of Gantt and I would like to >>> see the focus stay on that. >>> >>> >>> >>> >>> Thanks, Jay 2014-02-21 21:31 GMT+08:00 Russell Bryant : On 02/20/2014 06:04 PM, Sean Dague wrote: > > On 02/20/2014 05:32 PM, Russell Bryant wrote: > >> On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: > >>> Hi, > >>> > >>> Would like to know if there's any interest on having > >>> 'automatic evacuation' feature when a compute node goes down. I > >>> found 3 bps related to this topic: [1] Adding a periodic task > >>> and using ServiceGroup API for compute-node status [2] Using > >>> ceilometer to trigger the evacuate api. [3] Include some kind > >>> of H/A plugin by using a 'resource optimization service' > >>> > >>> Most of those BP's have comments like 'this logic should not > >>> reside in nova', so that's why i am asking what should be the > >>> best approach to have something like that. > >>> > >>> Should this be ignored, and just rely on external monitoring > >>> tools to trigger the evacuation? There are complex scenarios > >>> that require lot of logic that won't fit into nova nor any > >>> other OS component. (For instance: sometimes it will be faster > >>> to reboot the node or compute-nova than starting the > >>> evacuation, but if it fail X times then trigger an evacuation, > >>> etc ) > >>> > >>> Any thought/comment// about this? > >>> > >>> Regards Leandro > >>> > >>> [1] > >>> > https://blueprints.launchpad.net/nova/+spec/vm-auto-ha- > when-host-broken > >>> > >>> > [2] > >>> > https://blueprints.launchpad.net/nova/+spec/evacuate- > instance-automatically > >>> > >>> > [3] > >>> > https://blueprints.launchpad.net/nova/+spec/resource- > optimization-service > >> > >> > >>> > My opinion is that I would like to see this logic done outside of Nova. > > > > Right now Nova is the only service that really understands the > > compute topology of hosts, though it's understanding of liveness is > > really not sufficient to handle this kind of HA thing anyway. > > > > I think that's the real problem to solve. How to provide > > notifications to somewhere outside of Nova on host death. And the > > question is, should Nova be involved in just that part, keeping > > track of node liveness and signaling up for someone else to deal > > with it? Honestly that part I'm more on the fence about. Because > > putting another service in place to just handle that monitoring > > seems overkill. > > > > I 100% agree that all the policy, reacting, logic for this should > > be outside of Nova. Be it Heat or somewhere else. > > I think we agree. I'm very interested in continuing to enhance Nova > to make sure that the thing outside of Nova has all of the APIs it > needs to get the job done. > > -- > Russell Bryant > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Thanks, Jay >>> >>> ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/l
Re: [openstack-dev] [nova] Automatic Evacuation
Yes, it would be great if we can have a simple framework for future run time policy plugins. ;-) 2014-03-03 23:12 GMT+08:00 laserjetyang : > there are a lot of rules for HA or LB, so I think it might be a better > idea to scope the framework and leave the policy as plugins. > > > On Mon, Mar 3, 2014 at 10:30 PM, Andrew Laski > wrote: > >> On 03/01/14 at 07:24am, Jay Lau wrote: >> >>> Hey, >>> >>> Sorry to bring this up again. There are also some discussions here: >>> http://markmail.org/message/5zotly4qktaf34ei >>> >>> You can also search [Runtime Policy] in your email list. >>> >>> Not sure if we can put this to Gantt and enable Gantt provide both >>> initial >>> placement and rum time polices like HA, load balance etc. >>> >> >> I don't have an opinion at the moment as to whether or not this sort of >> functionality belongs in Gantt, but there's still a long way to go just to >> get the scheduling functionality we want out of Gantt and I would like to >> see the focus stay on that. >> >> >> >> >> >>> Thanks, >>> >>> Jay >>> >>> >>> >>> 2014-02-21 21:31 GMT+08:00 Russell Bryant : >>> >>> On 02/20/2014 06:04 PM, Sean Dague wrote: > On 02/20/2014 05:32 PM, Russell Bryant wrote: >> On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: >>> Hi, >>> >>> Would like to know if there's any interest on having >>> 'automatic evacuation' feature when a compute node goes down. I >>> found 3 bps related to this topic: [1] Adding a periodic task >>> and using ServiceGroup API for compute-node status [2] Using >>> ceilometer to trigger the evacuate api. [3] Include some kind >>> of H/A plugin by using a 'resource optimization service' >>> >>> Most of those BP's have comments like 'this logic should not >>> reside in nova', so that's why i am asking what should be the >>> best approach to have something like that. >>> >>> Should this be ignored, and just rely on external monitoring >>> tools to trigger the evacuation? There are complex scenarios >>> that require lot of logic that won't fit into nova nor any >>> other OS component. (For instance: sometimes it will be faster >>> to reboot the node or compute-nova than starting the >>> evacuation, but if it fail X times then trigger an evacuation, >>> etc ) >>> >>> Any thought/comment// about this? >>> >>> Regards Leandro >>> >>> [1] >>> https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken >>> >>> [2] >>> https://blueprints.launchpad.net/nova/+spec/evacuate- instance-automatically >>> >>> [3] >>> https://blueprints.launchpad.net/nova/+spec/resource- optimization-service >> >> >>> My opinion is that I would like to see this logic done outside of Nova. > > Right now Nova is the only service that really understands the > compute topology of hosts, though it's understanding of liveness is > really not sufficient to handle this kind of HA thing anyway. > > I think that's the real problem to solve. How to provide > notifications to somewhere outside of Nova on host death. And the > question is, should Nova be involved in just that part, keeping > track of node liveness and signaling up for someone else to deal > with it? Honestly that part I'm more on the fence about. Because > putting another service in place to just handle that monitoring > seems overkill. > > I 100% agree that all the policy, reacting, logic for this should > be outside of Nova. Be it Heat or somewhere else. I think we agree. I'm very interested in continuing to enhance Nova to make sure that the thing outside of Nova has all of the APIs it needs to get the job done. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> -- >>> Thanks, >>> >>> Jay >>> >> >> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
there are a lot of rules for HA or LB, so I think it might be a better idea to scope the framework and leave the policy as plugins. On Mon, Mar 3, 2014 at 10:30 PM, Andrew Laski wrote: > On 03/01/14 at 07:24am, Jay Lau wrote: > >> Hey, >> >> Sorry to bring this up again. There are also some discussions here: >> http://markmail.org/message/5zotly4qktaf34ei >> >> You can also search [Runtime Policy] in your email list. >> >> Not sure if we can put this to Gantt and enable Gantt provide both initial >> placement and rum time polices like HA, load balance etc. >> > > I don't have an opinion at the moment as to whether or not this sort of > functionality belongs in Gantt, but there's still a long way to go just to > get the scheduling functionality we want out of Gantt and I would like to > see the focus stay on that. > > > > > >> Thanks, >> >> Jay >> >> >> >> 2014-02-21 21:31 GMT+08:00 Russell Bryant : >> >> On 02/20/2014 06:04 PM, Sean Dague wrote: >>> > On 02/20/2014 05:32 PM, Russell Bryant wrote: >>> >> On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: >>> >>> Hi, >>> >>> >>> >>> Would like to know if there's any interest on having >>> >>> 'automatic evacuation' feature when a compute node goes down. I >>> >>> found 3 bps related to this topic: [1] Adding a periodic task >>> >>> and using ServiceGroup API for compute-node status [2] Using >>> >>> ceilometer to trigger the evacuate api. [3] Include some kind >>> >>> of H/A plugin by using a 'resource optimization service' >>> >>> >>> >>> Most of those BP's have comments like 'this logic should not >>> >>> reside in nova', so that's why i am asking what should be the >>> >>> best approach to have something like that. >>> >>> >>> >>> Should this be ignored, and just rely on external monitoring >>> >>> tools to trigger the evacuation? There are complex scenarios >>> >>> that require lot of logic that won't fit into nova nor any >>> >>> other OS component. (For instance: sometimes it will be faster >>> >>> to reboot the node or compute-nova than starting the >>> >>> evacuation, but if it fail X times then trigger an evacuation, >>> >>> etc ) >>> >>> >>> >>> Any thought/comment// about this? >>> >>> >>> >>> Regards Leandro >>> >>> >>> >>> [1] >>> >>> >>> https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken >>> >>> >>> >>> >>> [2] >>> >>> >>> https://blueprints.launchpad.net/nova/+spec/evacuate- >>> instance-automatically >>> >>> >>> >>> >>> [3] >>> >>> >>> https://blueprints.launchpad.net/nova/+spec/resource- >>> optimization-service >>> >> >>> >> >>> >>> >>> My opinion is that I would like to see this logic done outside of Nova. >>> > >>> > Right now Nova is the only service that really understands the >>> > compute topology of hosts, though it's understanding of liveness is >>> > really not sufficient to handle this kind of HA thing anyway. >>> > >>> > I think that's the real problem to solve. How to provide >>> > notifications to somewhere outside of Nova on host death. And the >>> > question is, should Nova be involved in just that part, keeping >>> > track of node liveness and signaling up for someone else to deal >>> > with it? Honestly that part I'm more on the fence about. Because >>> > putting another service in place to just handle that monitoring >>> > seems overkill. >>> > >>> > I 100% agree that all the policy, reacting, logic for this should >>> > be outside of Nova. Be it Heat or somewhere else. >>> >>> I think we agree. I'm very interested in continuing to enhance Nova >>> to make sure that the thing outside of Nova has all of the APIs it >>> needs to get the job done. >>> >>> -- >>> Russell Bryant >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> -- >> Thanks, >> >> Jay >> > > ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
On 03/01/14 at 07:24am, Jay Lau wrote: Hey, Sorry to bring this up again. There are also some discussions here: http://markmail.org/message/5zotly4qktaf34ei You can also search [Runtime Policy] in your email list. Not sure if we can put this to Gantt and enable Gantt provide both initial placement and rum time polices like HA, load balance etc. I don't have an opinion at the moment as to whether or not this sort of functionality belongs in Gantt, but there's still a long way to go just to get the scheduling functionality we want out of Gantt and I would like to see the focus stay on that. Thanks, Jay 2014-02-21 21:31 GMT+08:00 Russell Bryant : On 02/20/2014 06:04 PM, Sean Dague wrote: > On 02/20/2014 05:32 PM, Russell Bryant wrote: >> On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: >>> Hi, >>> >>> Would like to know if there's any interest on having >>> 'automatic evacuation' feature when a compute node goes down. I >>> found 3 bps related to this topic: [1] Adding a periodic task >>> and using ServiceGroup API for compute-node status [2] Using >>> ceilometer to trigger the evacuate api. [3] Include some kind >>> of H/A plugin by using a 'resource optimization service' >>> >>> Most of those BP's have comments like 'this logic should not >>> reside in nova', so that's why i am asking what should be the >>> best approach to have something like that. >>> >>> Should this be ignored, and just rely on external monitoring >>> tools to trigger the evacuation? There are complex scenarios >>> that require lot of logic that won't fit into nova nor any >>> other OS component. (For instance: sometimes it will be faster >>> to reboot the node or compute-nova than starting the >>> evacuation, but if it fail X times then trigger an evacuation, >>> etc ) >>> >>> Any thought/comment// about this? >>> >>> Regards Leandro >>> >>> [1] >>> https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken >>> >>> [2] >>> https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically >>> >>> [3] >>> https://blueprints.launchpad.net/nova/+spec/resource-optimization-service >> >> >>> My opinion is that I would like to see this logic done outside of Nova. > > Right now Nova is the only service that really understands the > compute topology of hosts, though it's understanding of liveness is > really not sufficient to handle this kind of HA thing anyway. > > I think that's the real problem to solve. How to provide > notifications to somewhere outside of Nova on host death. And the > question is, should Nova be involved in just that part, keeping > track of node liveness and signaling up for someone else to deal > with it? Honestly that part I'm more on the fence about. Because > putting another service in place to just handle that monitoring > seems overkill. > > I 100% agree that all the policy, reacting, logic for this should > be outside of Nova. Be it Heat or somewhere else. I think we agree. I'm very interested in continuing to enhance Nova to make sure that the thing outside of Nova has all of the APIs it needs to get the job done. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
Hey, Sorry to bring this up again. There are also some discussions here: http://markmail.org/message/5zotly4qktaf34ei You can also search [Runtime Policy] in your email list. Not sure if we can put this to Gantt and enable Gantt provide both initial placement and rum time polices like HA, load balance etc. Thanks, Jay 2014-02-21 21:31 GMT+08:00 Russell Bryant : > On 02/20/2014 06:04 PM, Sean Dague wrote: > > On 02/20/2014 05:32 PM, Russell Bryant wrote: > >> On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: > >>> Hi, > >>> > >>> Would like to know if there's any interest on having > >>> 'automatic evacuation' feature when a compute node goes down. I > >>> found 3 bps related to this topic: [1] Adding a periodic task > >>> and using ServiceGroup API for compute-node status [2] Using > >>> ceilometer to trigger the evacuate api. [3] Include some kind > >>> of H/A plugin by using a 'resource optimization service' > >>> > >>> Most of those BP's have comments like 'this logic should not > >>> reside in nova', so that's why i am asking what should be the > >>> best approach to have something like that. > >>> > >>> Should this be ignored, and just rely on external monitoring > >>> tools to trigger the evacuation? There are complex scenarios > >>> that require lot of logic that won't fit into nova nor any > >>> other OS component. (For instance: sometimes it will be faster > >>> to reboot the node or compute-nova than starting the > >>> evacuation, but if it fail X times then trigger an evacuation, > >>> etc ) > >>> > >>> Any thought/comment// about this? > >>> > >>> Regards Leandro > >>> > >>> [1] > >>> > https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken > >>> > >>> > [2] > >>> > https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically > >>> > >>> > [3] > >>> > https://blueprints.launchpad.net/nova/+spec/resource-optimization-service > >> > >> > >>> > My opinion is that I would like to see this logic done outside of Nova. > > > > Right now Nova is the only service that really understands the > > compute topology of hosts, though it's understanding of liveness is > > really not sufficient to handle this kind of HA thing anyway. > > > > I think that's the real problem to solve. How to provide > > notifications to somewhere outside of Nova on host death. And the > > question is, should Nova be involved in just that part, keeping > > track of node liveness and signaling up for someone else to deal > > with it? Honestly that part I'm more on the fence about. Because > > putting another service in place to just handle that monitoring > > seems overkill. > > > > I 100% agree that all the policy, reacting, logic for this should > > be outside of Nova. Be it Heat or somewhere else. > > I think we agree. I'm very interested in continuing to enhance Nova > to make sure that the thing outside of Nova has all of the APIs it > needs to get the job done. > > -- > Russell Bryant > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
On 02/20/2014 06:04 PM, Sean Dague wrote: > On 02/20/2014 05:32 PM, Russell Bryant wrote: >> On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: >>> Hi, >>> >>> Would like to know if there's any interest on having >>> 'automatic evacuation' feature when a compute node goes down. I >>> found 3 bps related to this topic: [1] Adding a periodic task >>> and using ServiceGroup API for compute-node status [2] Using >>> ceilometer to trigger the evacuate api. [3] Include some kind >>> of H/A plugin by using a 'resource optimization service' >>> >>> Most of those BP's have comments like 'this logic should not >>> reside in nova', so that's why i am asking what should be the >>> best approach to have something like that. >>> >>> Should this be ignored, and just rely on external monitoring >>> tools to trigger the evacuation? There are complex scenarios >>> that require lot of logic that won't fit into nova nor any >>> other OS component. (For instance: sometimes it will be faster >>> to reboot the node or compute-nova than starting the >>> evacuation, but if it fail X times then trigger an evacuation, >>> etc ) >>> >>> Any thought/comment// about this? >>> >>> Regards Leandro >>> >>> [1] >>> https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken >>> >>> [2] >>> https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically >>> >>> [3] >>> https://blueprints.launchpad.net/nova/+spec/resource-optimization-service >> >> >>> My opinion is that I would like to see this logic done outside of Nova. > > Right now Nova is the only service that really understands the > compute topology of hosts, though it's understanding of liveness is > really not sufficient to handle this kind of HA thing anyway. > > I think that's the real problem to solve. How to provide > notifications to somewhere outside of Nova on host death. And the > question is, should Nova be involved in just that part, keeping > track of node liveness and signaling up for someone else to deal > with it? Honestly that part I'm more on the fence about. Because > putting another service in place to just handle that monitoring > seems overkill. > > I 100% agree that all the policy, reacting, logic for this should > be outside of Nova. Be it Heat or somewhere else. I think we agree. I'm very interested in continuing to enhance Nova to make sure that the thing outside of Nova has all of the APIs it needs to get the job done. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
Hi, If I am not mistaken Mistral team listed Live migration as a potential use case for workflow engine. There is no much details though. https://wiki.openstack.org/wiki/Mistral#Live_migration As I know Mistral plan to implement generic event handling mechanism when one can bind any kind of workflow with external event triggered by Ceilometer or other monitoring system. This bound workflow can actually define live migration logic. Thanks Georgy On Thu, Feb 20, 2014 at 3:04 PM, Sean Dague wrote: > On 02/20/2014 05:32 PM, Russell Bryant wrote: > > On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: > >> Hi, > >> > >> Would like to know if there's any interest on having 'automatic > >> evacuation' feature when a compute node goes down. > >> I found 3 bps related to this topic: > >>[1] Adding a periodic task and using ServiceGroup API for > >> compute-node status > >>[2] Using ceilometer to trigger the evacuate api. > >>[3] Include some kind of H/A plugin by using a 'resource > >> optimization service' > >> > >> Most of those BP's have comments like 'this logic should not reside in > >> nova', so that's > >> why i am asking what should be the best approach to have something like > >> that. > >> > >> Should this be ignored, and just rely on external monitoring tools to > >> trigger the evacuation? > >> There are complex scenarios that require lot of logic that won't fit > >> into nova nor any other OS component. (For instance: sometimes it will > >> be faster to reboot the node or compute-nova than starting the > >> evacuation, but if it fail X times then trigger an evacuation, etc ) > >> > >> Any thought/comment// about this? > >> > >> Regards > >> Leandro > >> > >> [1] > https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken > >> [2] > >> > https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically > >> [3] > >> > https://blueprints.launchpad.net/nova/+spec/resource-optimization-service > > > > My opinion is that I would like to see this logic done outside of Nova. > > Right now Nova is the only service that really understands the compute > topology of hosts, though it's understanding of liveness is really not > sufficient to handle this kind of HA thing anyway. > > I think that's the real problem to solve. How to provide notifications > to somewhere outside of Nova on host death. And the question is, should > Nova be involved in just that part, keeping track of node liveness and > signaling up for someone else to deal with it? Honestly that part I'm > more on the fence about. Because putting another service in place to > just handle that monitoring seems overkill. > > I 100% agree that all the policy, reacting, logic for this should be > outside of Nova. Be it Heat or somewhere else. > > -Sean > > -- > Sean Dague > Samsung Research America > s...@dague.net / sean.da...@samsung.com > http://dague.net > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
On 02/20/2014 05:32 PM, Russell Bryant wrote: > On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: >> Hi, >> >> Would like to know if there's any interest on having 'automatic >> evacuation' feature when a compute node goes down. >> I found 3 bps related to this topic: >>[1] Adding a periodic task and using ServiceGroup API for >> compute-node status >>[2] Using ceilometer to trigger the evacuate api. >>[3] Include some kind of H/A plugin by using a 'resource >> optimization service' >> >> Most of those BP's have comments like 'this logic should not reside in >> nova', so that's >> why i am asking what should be the best approach to have something like >> that. >> >> Should this be ignored, and just rely on external monitoring tools to >> trigger the evacuation? >> There are complex scenarios that require lot of logic that won't fit >> into nova nor any other OS component. (For instance: sometimes it will >> be faster to reboot the node or compute-nova than starting the >> evacuation, but if it fail X times then trigger an evacuation, etc ) >> >> Any thought/comment// about this? >> >> Regards >> Leandro >> >> [1] https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken >> [2] >> https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically >> [3] >> https://blueprints.launchpad.net/nova/+spec/resource-optimization-service > > My opinion is that I would like to see this logic done outside of Nova. Right now Nova is the only service that really understands the compute topology of hosts, though it's understanding of liveness is really not sufficient to handle this kind of HA thing anyway. I think that's the real problem to solve. How to provide notifications to somewhere outside of Nova on host death. And the question is, should Nova be involved in just that part, keeping track of node liveness and signaling up for someone else to deal with it? Honestly that part I'm more on the fence about. Because putting another service in place to just handle that monitoring seems overkill. I 100% agree that all the policy, reacting, logic for this should be outside of Nova. Be it Heat or somewhere else. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
On 02/20/2014 05:05 PM, Costantino, Leandro I wrote: > Hi, > > Would like to know if there's any interest on having 'automatic > evacuation' feature when a compute node goes down. > I found 3 bps related to this topic: >[1] Adding a periodic task and using ServiceGroup API for > compute-node status >[2] Using ceilometer to trigger the evacuate api. >[3] Include some kind of H/A plugin by using a 'resource > optimization service' > > Most of those BP's have comments like 'this logic should not reside in > nova', so that's > why i am asking what should be the best approach to have something like > that. > > Should this be ignored, and just rely on external monitoring tools to > trigger the evacuation? > There are complex scenarios that require lot of logic that won't fit > into nova nor any other OS component. (For instance: sometimes it will > be faster to reboot the node or compute-nova than starting the > evacuation, but if it fail X times then trigger an evacuation, etc ) > > Any thought/comment// about this? > > Regards > Leandro > > [1] https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken > [2] > https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically > [3] > https://blueprints.launchpad.net/nova/+spec/resource-optimization-service My opinion is that I would like to see this logic done outside of Nova. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Automatic Evacuation
Im also curious about that. I think it is proper that ceilometer should have the role of Mornitoring vm. Nova just opens auto-evacuate API, when a trigger calls the API nova calls like shelve api for the vm re-spawn. What do you think of this? 2014. 2. 21. 오전 7:13에 "Costantino, Leandro I" < leandro.i.costant...@intel.com>님이 작성: > Hi, > > Would like to know if there's any interest on having 'automatic > evacuation' feature when a compute node goes down. > I found 3 bps related to this topic: >[1] Adding a periodic task and using ServiceGroup API for compute-node > status >[2] Using ceilometer to trigger the evacuate api. >[3] Include some kind of H/A plugin by using a 'resource optimization > service' > > Most of those BP's have comments like 'this logic should not reside in > nova', so that's > why i am asking what should be the best approach to have something like > that. > > Should this be ignored, and just rely on external monitoring tools to > trigger the evacuation? > There are complex scenarios that require lot of logic that won't fit into > nova nor any other OS component. (For instance: sometimes it will be faster > to reboot the node or compute-nova than starting the evacuation, but if it > fail X times then trigger an evacuation, etc ) > > Any thought/comment// about this? > > Regards > Leandro > > [1] https://blueprints.launchpad.net/nova/+spec/vm-auto-ha- > when-host-broken > [2] https://blueprints.launchpad.net/nova/+spec/evacuate- > instance-automatically > [3] https://blueprints.launchpad.net/nova/+spec/resource- > optimization-service > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
