Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On 10/08/2013 02:19 PM, Mike Wilson wrote: > +1 to what Chris suggested. Zombie state that doesn't affect quota, but > doesn't create more problems by trying to reuse resources that aren't > available. That way we can tell the customer that things are deleted, > but we don't need to break our cloud by screwing up future schedule > requests. Right. We don't want to punish users for an operational error or mistake. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
+1 to what Chris suggested. Zombie state that doesn't affect quota, but doesn't create more problems by trying to reuse resources that aren't available. That way we can tell the customer that things are deleted, but we don't need to break our cloud by screwing up future schedule requests. -Mike On Tue, Oct 8, 2013 at 11:58 AM, Joshua Harlow wrote: > Sure, basically a way around this is to do migration of the VM's on the > host u are doing maintenance on. > > That¹s one way y! has its ops folks work around this. > > Another solution is just don't do local_deletes :-P > > It sounds like your 'zombie' state would be useful as a way to solve this > also. > > To me though any solution that creates 2 sets of the same resources in > your cloud though isn't a good way (which afaik the current local_delete > aims for) as it causes maintenance and operator pain (and needless > problems that a person has to go in and figure out & resolve). I'd rather > have the delete fail, leave the quota of the user alone, and tell the user > the hypervisor where the VM is on is currently under maintenance (ideally > the `host-update` resolves this, as long as its supported on all > hypervisor types). At least that gives a sane operational experience and > doesn't cause support bugs that are hard to resolve. > > But maybe this type of action should be more configurable. Allow or > disallow local deletes. > > On 10/7/13 11:50 PM, "Chris Friesen" wrote: > > >On 10/07/2013 05:30 PM, Joshua Harlow wrote: > >> A scenario that I've seen: > >> > >> Take 'nova-compute' down for software upgrade, API still accessible > >>since > >> you want to provide API uptime (aka not taking the whole cluster > >>offline). > >> > >> User Y deletes VM on that hypervisor where nova-compute is currently > >>down, > >> DB locally deletes, at this point VM 'A' is still active but nova thinks > >> its not. > > > >Isn't this sort of thing exactly what "nova host-update --maintenance > >enable " was intended for? I.e., push all the VMs off that > >compute node so you can take down the services without causing problems. > > > >Its kind of a pain that the host-update stuff is implemented at the > >hypervisor level though (and isn't available for libvirt), it seems like > >it could be implemented at a more generic level. (And on that note, why > >isn't there a "host" table in the database since we can have multiple > >services running on one host and we might want to take them all down?) > > > >Alternately, maybe we need to have a 2-stage delete, where the VM gets > >put into a "zombie" state in the database and the resources can't be > >reused until the compute service confirms that the VM has been killed. > > > >Chris > > > > > > > >___ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
Sure, basically a way around this is to do migration of the VM's on the host u are doing maintenance on. That¹s one way y! has its ops folks work around this. Another solution is just don't do local_deletes :-P It sounds like your 'zombie' state would be useful as a way to solve this also. To me though any solution that creates 2 sets of the same resources in your cloud though isn't a good way (which afaik the current local_delete aims for) as it causes maintenance and operator pain (and needless problems that a person has to go in and figure out & resolve). I'd rather have the delete fail, leave the quota of the user alone, and tell the user the hypervisor where the VM is on is currently under maintenance (ideally the `host-update` resolves this, as long as its supported on all hypervisor types). At least that gives a sane operational experience and doesn't cause support bugs that are hard to resolve. But maybe this type of action should be more configurable. Allow or disallow local deletes. On 10/7/13 11:50 PM, "Chris Friesen" wrote: >On 10/07/2013 05:30 PM, Joshua Harlow wrote: >> A scenario that I've seen: >> >> Take 'nova-compute' down for software upgrade, API still accessible >>since >> you want to provide API uptime (aka not taking the whole cluster >>offline). >> >> User Y deletes VM on that hypervisor where nova-compute is currently >>down, >> DB locally deletes, at this point VM 'A' is still active but nova thinks >> its not. > >Isn't this sort of thing exactly what "nova host-update --maintenance >enable " was intended for? I.e., push all the VMs off that >compute node so you can take down the services without causing problems. > >Its kind of a pain that the host-update stuff is implemented at the >hypervisor level though (and isn't available for libvirt), it seems like >it could be implemented at a more generic level. (And on that note, why >isn't there a "host" table in the database since we can have multiple >services running on one host and we might want to take them all down?) > >Alternately, maybe we need to have a 2-stage delete, where the VM gets >put into a "zombie" state in the database and the resources can't be >reused until the compute service confirms that the VM has been killed. > >Chris > > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On 08/10/13 01:01, Russell Bryant wrote: > On 10/07/2013 06:34 PM, Vishvananda Ishaya wrote: >> There is a configuration option stating what to do with instances that are >> still in the hypervisor but have been deleted from the database. I think you >> want: >> >> running_deleted_instance_action=reap >> >> You probably also want >> >> resume_guests_state_on_host_boot=true >> >> to bring back the instances that were running before the node was powered >> off. We should definitely consider changing the default of these two values >> since I think the default values are probably not what most people would >> want. > > Thanks, vish. Those defaults keep biting ... > > We tried changing the default of resume_guests_state_on_host_boot in the > RDO packages at one point, but ended up turning it back off. I believe > we had some problems with nova-compute trying to start instances before > nova-network was fully initialized, sometimes leaving instances in a > broken state. That kind of stuff is all fixable though, and if we go > ahead and change the defaults early in Icehouse dev, we should have > plenty of time to deal with any fallout before Icehouse is released. > IIRC - the consensus was that resume_guests_state_on_host_boot is not what cloud users would expect actually. If the instance went down, it is likely that another one took it's place and bringing the old one back up might cause problems. I will comment on the review and we can take the discussion there if more appropriate. Cheers, N. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
In case of vm evacuate with Grizzly code you will have those disks leftovers due to this bug: https://bugs.launchpad.net/nova/+bug/1156269 It's solved for Havana. Pavel Vishvananda Ishaya wrote on 08/10/2013 01:34:26 AM: > From: Vishvananda Ishaya > To: OpenStack Development Mailing List , > Date: 08/10/2013 01:36 AM > Subject: Re: [openstack-dev] [nova] BUG? nova-compute should delete > unused instance files on boot > > There is a configuration option stating what to do with instances > that are still in the hypervisor but have been deleted from the > database. I think you want: > > running_deleted_instance_action=reap > > You probably also want > > resume_guests_state_on_host_boot=true > > to bring back the instances that were running before the node was > powered off. We should definitely consider changing the default of > these two values since I think the default values are probably not > what most people would want. > > Vish > On Oct 7, 2013, at 1:24 PM, Chris Friesen wrote: > > > On 10/07/2013 12:44 PM, Russell Bryant wrote: > >> On 10/07/2013 02:28 PM, Chris Friesen wrote: > >>> > >>> I've been doing a lot of instance creation/deletion/evacuate and I've > >>> noticed that if I > >>> > >>> 1)create an instance > >>> 2) power off the compute node it was running on > >>> 3) delete the instance > >>> 4) boot up the compute node > >>> > >>> then the instance rootfs stays around in /var/lib/nova/instances/. > >>> Eventually this could add up to significant amounts of space. > >>> > >>> > >>> Is this expected behaviour? (This is on grizzly, so maybe havana is > >>> different.) If not, should I file a bug for it? > >>> > >>> I think it would make sense for the compute node to come up, query all > >>> the instances in /var/lib/nova/instances/, and delete the ones for > >>> instances that aren't in the database. > >> > >> How long are you waiting after starting up the compute node? I would > >> expect it to get cleaned up by a periodic task, so you might have to > >> wait roughly 10 minutes (by default). > > > > This is nearly 50 minutes after booting up the compute node: > > > > cfriesen@compute2:/var/lib/nova/instances$ ls -1 > > 39e459b1-3878-41db-aaaf-7c7d0dfa2b19 > > 41a60975-d6b8-468e-90bc-d7de58c2124d > > 46aec2ae-b6de-4503-a238-af736f81f1a4 > > 50ec3d89-1c9d-4c28-adaf-26c924dfa3ed > > _base > > c6ec71a3-658c-4c7c-aa42-cc26296ce7fb > > c72845e9-0d34-459f-b602-bb2ee409728b > > compute_nodes > > locks > > > > Of these, only two show up in "nova list". > > > > Chris > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On 10/07/2013 05:30 PM, Joshua Harlow wrote: A scenario that I've seen: Take 'nova-compute' down for software upgrade, API still accessible since you want to provide API uptime (aka not taking the whole cluster offline). User Y deletes VM on that hypervisor where nova-compute is currently down, DB locally deletes, at this point VM 'A' is still active but nova thinks its not. Isn't this sort of thing exactly what "nova host-update --maintenance enable " was intended for? I.e., push all the VMs off that compute node so you can take down the services without causing problems. Its kind of a pain that the host-update stuff is implemented at the hypervisor level though (and isn't available for libvirt), it seems like it could be implemented at a more generic level. (And on that note, why isn't there a "host" table in the database since we can have multiple services running on one host and we might want to take them all down?) Alternately, maybe we need to have a 2-stage delete, where the VM gets put into a "zombie" state in the database and the resources can't be reused until the compute service confirms that the VM has been killed. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
A scenario that I've seen: Take 'nova-compute' down for software upgrade, API still accessible since you want to provide API uptime (aka not taking the whole cluster offline). User Y deletes VM on that hypervisor where nova-compute is currently down, DB locally deletes, at this point VM 'A' is still active but nova thinks its not. User X requests VM, gets allocated 'A' hostname, ip (or other resource). Uh oh, u now have two of the same IP/hostname in the same network (second 'A' is in damaged state since it likely can't even 'ifup' its network). Upgrade software on that hypervisor (yum install xyz...), service nova-compute restart (back to normal). Now the first 'A' could get deleted, but u still have second broken 'A'. Now what? On 10/7/13 4:21 PM, "Vishvananda Ishaya" wrote: > >On Oct 7, 2013, at 3:49 PM, Joshua Harlow wrote: > >> This brings up another question, do people actually like/use the >> 'local_delete' feature in nova? >> >> In general it seems to free resources that are not actually capable of >> being freed and has been problematic for y! usage. >> >> Deleting from the DB allows another request to actually take those >> resources over, yet the previous VM (+network,volumes...) that wasn't >> deleted still has those resources (likely attached to it in the case of >>a >> volume, or in the case of a hypervisor the VM resource is still active, >> but maybe nova-compute is down) so u end up in a conflict. How are >>others >> using this code? Has it been working out? > >We haven't had any trouble with the two settings set as below. Users seem >to get far more frustrated when they have things that they cannot delete, >especially when it is using up their precious quota. > >Vish > >> >> -Josh >> >> On 10/7/13 3:34 PM, "Vishvananda Ishaya" wrote: >> >>> There is a configuration option stating what to do with instances that >>> are still in the hypervisor but have been deleted from the database. I >>> think you want: >>> >>> running_deleted_instance_action=reap >>> >>> You probably also want >>> >>> resume_guests_state_on_host_boot=true >>> >>> to bring back the instances that were running before the node was >>>powered >>> off. We should definitely consider changing the default of these two >>> values since I think the default values are probably not what most >>>people >>> would want. >>> >>> Vish >>> On Oct 7, 2013, at 1:24 PM, Chris Friesen >>> wrote: >>> On 10/07/2013 12:44 PM, Russell Bryant wrote: > On 10/07/2013 02:28 PM, Chris Friesen wrote: >> >> I've been doing a lot of instance creation/deletion/evacuate and >>I've >> noticed that if I >> >> 1)create an instance >> 2) power off the compute node it was running on >> 3) delete the instance >> 4) boot up the compute node >> >> then the instance rootfs stays around in /var/lib/nova/instances/. >> Eventually this could add up to significant amounts of space. >> >> >> Is this expected behaviour? (This is on grizzly, so maybe havana is >> different.) If not, should I file a bug for it? >> >> I think it would make sense for the compute node to come up, query >>all >> the instances in /var/lib/nova/instances/, and delete the ones for >> instances that aren't in the database. > > How long are you waiting after starting up the compute node? I would > expect it to get cleaned up by a periodic task, so you might have to > wait roughly 10 minutes (by default). This is nearly 50 minutes after booting up the compute node: cfriesen@compute2:/var/lib/nova/instances$ ls -1 39e459b1-3878-41db-aaaf-7c7d0dfa2b19 41a60975-d6b8-468e-90bc-d7de58c2124d 46aec2ae-b6de-4503-a238-af736f81f1a4 50ec3d89-1c9d-4c28-adaf-26c924dfa3ed _base c6ec71a3-658c-4c7c-aa42-cc26296ce7fb c72845e9-0d34-459f-b602-bb2ee409728b compute_nodes locks Of these, only two show up in "nova list". Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On Oct 7, 2013, at 3:49 PM, Joshua Harlow wrote: > This brings up another question, do people actually like/use the > 'local_delete' feature in nova? > > In general it seems to free resources that are not actually capable of > being freed and has been problematic for y! usage. > > Deleting from the DB allows another request to actually take those > resources over, yet the previous VM (+network,volumes...) that wasn't > deleted still has those resources (likely attached to it in the case of a > volume, or in the case of a hypervisor the VM resource is still active, > but maybe nova-compute is down) so u end up in a conflict. How are others > using this code? Has it been working out? We haven't had any trouble with the two settings set as below. Users seem to get far more frustrated when they have things that they cannot delete, especially when it is using up their precious quota. Vish > > -Josh > > On 10/7/13 3:34 PM, "Vishvananda Ishaya" wrote: > >> There is a configuration option stating what to do with instances that >> are still in the hypervisor but have been deleted from the database. I >> think you want: >> >> running_deleted_instance_action=reap >> >> You probably also want >> >> resume_guests_state_on_host_boot=true >> >> to bring back the instances that were running before the node was powered >> off. We should definitely consider changing the default of these two >> values since I think the default values are probably not what most people >> would want. >> >> Vish >> On Oct 7, 2013, at 1:24 PM, Chris Friesen >> wrote: >> >>> On 10/07/2013 12:44 PM, Russell Bryant wrote: On 10/07/2013 02:28 PM, Chris Friesen wrote: > > I've been doing a lot of instance creation/deletion/evacuate and I've > noticed that if I > > 1)create an instance > 2) power off the compute node it was running on > 3) delete the instance > 4) boot up the compute node > > then the instance rootfs stays around in /var/lib/nova/instances/. > Eventually this could add up to significant amounts of space. > > > Is this expected behaviour? (This is on grizzly, so maybe havana is > different.) If not, should I file a bug for it? > > I think it would make sense for the compute node to come up, query all > the instances in /var/lib/nova/instances/, and delete the ones for > instances that aren't in the database. How long are you waiting after starting up the compute node? I would expect it to get cleaned up by a periodic task, so you might have to wait roughly 10 minutes (by default). >>> >>> This is nearly 50 minutes after booting up the compute node: >>> >>> cfriesen@compute2:/var/lib/nova/instances$ ls -1 >>> 39e459b1-3878-41db-aaaf-7c7d0dfa2b19 >>> 41a60975-d6b8-468e-90bc-d7de58c2124d >>> 46aec2ae-b6de-4503-a238-af736f81f1a4 >>> 50ec3d89-1c9d-4c28-adaf-26c924dfa3ed >>> _base >>> c6ec71a3-658c-4c7c-aa42-cc26296ce7fb >>> c72845e9-0d34-459f-b602-bb2ee409728b >>> compute_nodes >>> locks >>> >>> Of these, only two show up in "nova list". >>> >>> Chris >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On 10/07/2013 06:34 PM, Vishvananda Ishaya wrote: > There is a configuration option stating what to do with instances that are > still in the hypervisor but have been deleted from the database. I think you > want: > > running_deleted_instance_action=reap > > You probably also want > > resume_guests_state_on_host_boot=true > > to bring back the instances that were running before the node was powered > off. We should definitely consider changing the default of these two values > since I think the default values are probably not what most people would want. https://review.openstack.org/50188 -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On 10/07/2013 06:34 PM, Vishvananda Ishaya wrote: > There is a configuration option stating what to do with instances that are > still in the hypervisor but have been deleted from the database. I think you > want: > > running_deleted_instance_action=reap > > You probably also want > > resume_guests_state_on_host_boot=true > > to bring back the instances that were running before the node was powered > off. We should definitely consider changing the default of these two values > since I think the default values are probably not what most people would want. Thanks, vish. Those defaults keep biting ... We tried changing the default of resume_guests_state_on_host_boot in the RDO packages at one point, but ended up turning it back off. I believe we had some problems with nova-compute trying to start instances before nova-network was fully initialized, sometimes leaving instances in a broken state. That kind of stuff is all fixable though, and if we go ahead and change the defaults early in Icehouse dev, we should have plenty of time to deal with any fallout before Icehouse is released. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
This brings up another question, do people actually like/use the 'local_delete' feature in nova? In general it seems to free resources that are not actually capable of being freed and has been problematic for y! usage. Deleting from the DB allows another request to actually take those resources over, yet the previous VM (+network,volumes...) that wasn't deleted still has those resources (likely attached to it in the case of a volume, or in the case of a hypervisor the VM resource is still active, but maybe nova-compute is down) so u end up in a conflict. How are others using this code? Has it been working out? -Josh On 10/7/13 3:34 PM, "Vishvananda Ishaya" wrote: >There is a configuration option stating what to do with instances that >are still in the hypervisor but have been deleted from the database. I >think you want: > >running_deleted_instance_action=reap > >You probably also want > >resume_guests_state_on_host_boot=true > >to bring back the instances that were running before the node was powered >off. We should definitely consider changing the default of these two >values since I think the default values are probably not what most people >would want. > >Vish >On Oct 7, 2013, at 1:24 PM, Chris Friesen >wrote: > >> On 10/07/2013 12:44 PM, Russell Bryant wrote: >>> On 10/07/2013 02:28 PM, Chris Friesen wrote: I've been doing a lot of instance creation/deletion/evacuate and I've noticed that if I 1)create an instance 2) power off the compute node it was running on 3) delete the instance 4) boot up the compute node then the instance rootfs stays around in /var/lib/nova/instances/. Eventually this could add up to significant amounts of space. Is this expected behaviour? (This is on grizzly, so maybe havana is different.) If not, should I file a bug for it? I think it would make sense for the compute node to come up, query all the instances in /var/lib/nova/instances/, and delete the ones for instances that aren't in the database. >>> >>> How long are you waiting after starting up the compute node? I would >>> expect it to get cleaned up by a periodic task, so you might have to >>> wait roughly 10 minutes (by default). >> >> This is nearly 50 minutes after booting up the compute node: >> >> cfriesen@compute2:/var/lib/nova/instances$ ls -1 >> 39e459b1-3878-41db-aaaf-7c7d0dfa2b19 >> 41a60975-d6b8-468e-90bc-d7de58c2124d >> 46aec2ae-b6de-4503-a238-af736f81f1a4 >> 50ec3d89-1c9d-4c28-adaf-26c924dfa3ed >> _base >> c6ec71a3-658c-4c7c-aa42-cc26296ce7fb >> c72845e9-0d34-459f-b602-bb2ee409728b >> compute_nodes >> locks >> >> Of these, only two show up in "nova list". >> >> Chris >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
There is a configuration option stating what to do with instances that are still in the hypervisor but have been deleted from the database. I think you want: running_deleted_instance_action=reap You probably also want resume_guests_state_on_host_boot=true to bring back the instances that were running before the node was powered off. We should definitely consider changing the default of these two values since I think the default values are probably not what most people would want. Vish On Oct 7, 2013, at 1:24 PM, Chris Friesen wrote: > On 10/07/2013 12:44 PM, Russell Bryant wrote: >> On 10/07/2013 02:28 PM, Chris Friesen wrote: >>> >>> I've been doing a lot of instance creation/deletion/evacuate and I've >>> noticed that if I >>> >>> 1)create an instance >>> 2) power off the compute node it was running on >>> 3) delete the instance >>> 4) boot up the compute node >>> >>> then the instance rootfs stays around in /var/lib/nova/instances/. >>> Eventually this could add up to significant amounts of space. >>> >>> >>> Is this expected behaviour? (This is on grizzly, so maybe havana is >>> different.) If not, should I file a bug for it? >>> >>> I think it would make sense for the compute node to come up, query all >>> the instances in /var/lib/nova/instances/, and delete the ones for >>> instances that aren't in the database. >> >> How long are you waiting after starting up the compute node? I would >> expect it to get cleaned up by a periodic task, so you might have to >> wait roughly 10 minutes (by default). > > This is nearly 50 minutes after booting up the compute node: > > cfriesen@compute2:/var/lib/nova/instances$ ls -1 > 39e459b1-3878-41db-aaaf-7c7d0dfa2b19 > 41a60975-d6b8-468e-90bc-d7de58c2124d > 46aec2ae-b6de-4503-a238-af736f81f1a4 > 50ec3d89-1c9d-4c28-adaf-26c924dfa3ed > _base > c6ec71a3-658c-4c7c-aa42-cc26296ce7fb > c72845e9-0d34-459f-b602-bb2ee409728b > compute_nodes > locks > > Of these, only two show up in "nova list". > > Chris > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On 10/07/2013 12:44 PM, Russell Bryant wrote: On 10/07/2013 02:28 PM, Chris Friesen wrote: I've been doing a lot of instance creation/deletion/evacuate and I've noticed that if I 1)create an instance 2) power off the compute node it was running on 3) delete the instance 4) boot up the compute node then the instance rootfs stays around in /var/lib/nova/instances/. Eventually this could add up to significant amounts of space. Is this expected behaviour? (This is on grizzly, so maybe havana is different.) If not, should I file a bug for it? I think it would make sense for the compute node to come up, query all the instances in /var/lib/nova/instances/, and delete the ones for instances that aren't in the database. How long are you waiting after starting up the compute node? I would expect it to get cleaned up by a periodic task, so you might have to wait roughly 10 minutes (by default). This is nearly 50 minutes after booting up the compute node: cfriesen@compute2:/var/lib/nova/instances$ ls -1 39e459b1-3878-41db-aaaf-7c7d0dfa2b19 41a60975-d6b8-468e-90bc-d7de58c2124d 46aec2ae-b6de-4503-a238-af736f81f1a4 50ec3d89-1c9d-4c28-adaf-26c924dfa3ed _base c6ec71a3-658c-4c7c-aa42-cc26296ce7fb c72845e9-0d34-459f-b602-bb2ee409728b compute_nodes locks Of these, only two show up in "nova list". Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
On 10/07/2013 02:28 PM, Chris Friesen wrote: > > I've been doing a lot of instance creation/deletion/evacuate and I've > noticed that if I > > 1)create an instance > 2) power off the compute node it was running on > 3) delete the instance > 4) boot up the compute node > > then the instance rootfs stays around in /var/lib/nova/instances/. > Eventually this could add up to significant amounts of space. > > > Is this expected behaviour? (This is on grizzly, so maybe havana is > different.) If not, should I file a bug for it? > > I think it would make sense for the compute node to come up, query all > the instances in /var/lib/nova/instances/, and delete the ones for > instances that aren't in the database. How long are you waiting after starting up the compute node? I would expect it to get cleaned up by a periodic task, so you might have to wait roughly 10 minutes (by default). -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] BUG? nova-compute should delete unused instance files on boot
I've been doing a lot of instance creation/deletion/evacuate and I've noticed that if I 1)create an instance 2) power off the compute node it was running on 3) delete the instance 4) boot up the compute node then the instance rootfs stays around in /var/lib/nova/instances/. Eventually this could add up to significant amounts of space. Is this expected behaviour? (This is on grizzly, so maybe havana is different.) If not, should I file a bug for it? I think it would make sense for the compute node to come up, query all the instances in /var/lib/nova/instances/, and delete the ones for instances that aren't in the database. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
