Re: [Openstack] Compute Node Down!
Thanks, that's what I ended up doing (by intuition rather than knowledge)
yesterday. I didn't know about nova rescue either.
I think this is a Big Big room for improvement here. In the best case this
should be discovered automatically and the switchover should be done without
human Intervention.
Wolfgang
--
Sent from my mobile device
On 20.09.2012, at 07:26, Tom Fifield fifie...@unimelb.edu.au wrote:
On 20/09/12 13:50, Vishvananda Ishaya wrote:
**
On Wed, Sep 19, 2012 at 4:03 AM, Wolfgang Hennerbichler
wolfgang.hennerbich...@risc-software.at
mailto:wolfgang.hennerbich...@risc-software.at wrote:
Hello Folks,
Although it seems a pretty straightforward scenario I have a hard
time finding documentation on this.
One of my compute nodes broke down. All the instances are on shared
storage, so no troubles here, but I don't know how to tell openstack
that the VM should be deployed on another compute node. I tried
fiddling around in the mysql-db with no success.
Any help is really appreciated.
Wolfgang
== Dead compute host ==
Working with the host information
pre
i-15b9 at3-ui02 running nectarkey (376, np-rcc54) 0 m1.xxlarge
2012-06-19T00:48:11.000Z 115.146.93.60
/pre
# review the status of the host using the nova database, some of the
important information is highlighted below.
pre
SELECT * FROM instances WHERE id = CONV('15b9', 16, 10) \G;
*** 1. row ***
created_at: 2012-06-19 00:48:11
updated_at: 2012-07-03 00:35:11
deleted_at: NULL
...
id: 5561
...
power_state: 5
vm_state: shutoff
...
hostname: at3-ui02
host: np-rcc54
...
uuid: 3f57699a-e773-4650-a443-b4b37eed5a06
...
task_state: NULL
...
/pre
Update the vm's compute host.
pre
UPDATE instances SET host = 'np-rcc46' WHERE uuid =
'3f57699a-e773-4650-a443-b4b37eed5a06';
/pre
Update the libvirt xml
* change the DHCPSERVER value to the host ip address.
* possibly the VNC IP if it isn't already 0.0.0.0
Dump a copy of a nwfilter to use as a template for creating the missing
nwfilter.
pre
virsh nwfilter-list
vrish nwfilter-dumpxml nova-instance-instance-.
/pre
Example of the template file
pre
filter name='nova-instance-instance-1cc6-fa163e003b43' chain='root'
uuidd5f6f610-d0b8-4407-ae00-5dabef80677a/uuid
filterref filter='nova-base'/
/filter
/pre
The filter name value is available from the instances.xml file (filterref
filter=nova-instance-instance-1cc6-fa163e003b43).
*Note the filter name must be exact!
Generate a new uuid and replace it at the uuid value.
Update filter to match id from instance xml
pre
virsh nwfilter-define /tmp/filter.xml
virsh define libvirt.xml
virsh list --all
/pre
Kill all dnsmasq and restart nova services.
pre
killall dnsmasq; service nova-network restart; service nova-compute restart
/pre
Start the vm
pre
virsh start instance-0
/pre
On the nova DB
pre
UPDATE instances SET vm_state = 'active', power_state = 1 WHERE uuid =
'3f57699a-e773-4650-a443-b4b37eed5a06';
/pre
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Compute Node Down!
if you are on essex, you can issue a nova rescue, if in cactus, you have to manipulate the instances table to tell where the new instance will be running, and then from the new compute node issue a : virsh define /path/to/XML virsh start instance_name From that moment, you can manage the instance using euca / nova * * *Ale* On Wed, Sep 19, 2012 at 4:03 AM, Wolfgang Hennerbichler wolfgang.hennerbich...@risc-software.at wrote: Hello Folks, Although it seems a pretty straightforward scenario I have a hard time finding documentation on this. One of my compute nodes broke down. All the instances are on shared storage, so no troubles here, but I don't know how to tell openstack that the VM should be deployed on another compute node. I tried fiddling around in the mysql-db with no success. Any help is really appreciated. Wolfgang ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Compute Node Down!
Hi Ale, It's the first time I see nova rescue, maybe this should be somewhere else in the documentation. Maybe the part related to migration, that's where I looked and tried. I first tried to do migration of the VM on a dead host, but that didn't work then after searching I stumbled upon this patch to enable the functionality to move VMs from a dead host: https://review.openstack.org/#/c/11086/12 But it wasn't available for Essex, and won't be in Folsom either. So I thought this functionality was not there yet either. Searched for ours documentation and google, but never saw anything about rescue. Is the functionality the same as the above patch? I looked at the code, and the code seems much smaller. The above patch seems to do more cleaning up also. Because I didn't know about nova rescue I already copied the patch to Essex successfully, but maybe it's safe to use nova rescue. This is the evacuate patch for stable essex: https://review.openstack.org/#/c/13282/ Do I understand right that this is the workflow: nova rescue instance1 Look if the state changed to RESCUED if it's rescued do a nova unrescue instance1 which cleans up the resources used for rescuing and changes the state back to ACTIVE on the vm? Thanks a lot, sure is helpful. Sam On Wed, Sep 19, 2012 at 9:12 PM, Alejandro Comisario alejandro.comisa...@mercadolibre.com wrote: if you are on essex, you can issue a nova rescue, if in cactus, you have to manipulate the instances table to tell where the new instance will be running, and then from the new compute node issue a : virsh define /path/to/XML virsh start instance_name From that moment, you can manage the instance using euca / nova * * *Ale* On Wed, Sep 19, 2012 at 4:03 AM, Wolfgang Hennerbichler wolfgang.hennerbich...@risc-software.at wrote: Hello Folks, Although it seems a pretty straightforward scenario I have a hard time finding documentation on this. One of my compute nodes broke down. All the instances are on shared storage, so no troubles here, but I don't know how to tell openstack that the VM should be deployed on another compute node. I tried fiddling around in the mysql-db with no success. Any help is really appreciated. Wolfgang ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Compute Node Down!
rescue does not bring back nodes from dead hosts. It reboots the instance with a fresh copy of the image it booted with as the base drive and attaches the current drive as a second drive so you can mount it and fix things. Unrescue switches back to the original drive after you have made your repairs. Vish On Sep 19, 2012, at 7:13 PM, Sam Stoelinga sammiest...@gmail.com wrote: Hi Ale, It's the first time I see nova rescue, maybe this should be somewhere else in the documentation. Maybe the part related to migration, that's where I looked and tried. I first tried to do migration of the VM on a dead host, but that didn't work then after searching I stumbled upon this patch to enable the functionality to move VMs from a dead host: https://review.openstack.org/#/c/11086/12 But it wasn't available for Essex, and won't be in Folsom either. So I thought this functionality was not there yet either. Searched for ours documentation and google, but never saw anything about rescue. Is the functionality the same as the above patch? I looked at the code, and the code seems much smaller. The above patch seems to do more cleaning up also. Because I didn't know about nova rescue I already copied the patch to Essex successfully, but maybe it's safe to use nova rescue. This is the evacuate patch for stable essex: https://review.openstack.org/#/c/13282/ Do I understand right that this is the workflow: nova rescue instance1 Look if the state changed to RESCUED if it's rescued do a nova unrescue instance1 which cleans up the resources used for rescuing and changes the state back to ACTIVE on the vm? Thanks a lot, sure is helpful. Sam On Wed, Sep 19, 2012 at 9:12 PM, Alejandro Comisario alejandro.comisa...@mercadolibre.com wrote: if you are on essex, you can issue a nova rescue, if in cactus, you have to manipulate the instances table to tell where the new instance will be running, and then from the new compute node issue a : virsh define /path/to/XML virsh start instance_name From that moment, you can manage the instance using euca / nova Ale On Wed, Sep 19, 2012 at 4:03 AM, Wolfgang Hennerbichler wolfgang.hennerbich...@risc-software.at wrote: Hello Folks, Although it seems a pretty straightforward scenario I have a hard time finding documentation on this. One of my compute nodes broke down. All the instances are on shared storage, so no troubles here, but I don't know how to tell openstack that the VM should be deployed on another compute node. I tried fiddling around in the mysql-db with no success. Any help is really appreciated. Wolfgang ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Compute Node Down!
On 20/09/12 13:50, Vishvananda Ishaya wrote:
**
On Wed, Sep 19, 2012 at 4:03 AM, Wolfgang Hennerbichler
wolfgang.hennerbich...@risc-software.at
mailto:wolfgang.hennerbich...@risc-software.at wrote:
Hello Folks,
Although it seems a pretty straightforward scenario I have a hard
time finding documentation on this.
One of my compute nodes broke down. All the instances are on shared
storage, so no troubles here, but I don't know how to tell openstack
that the VM should be deployed on another compute node. I tried
fiddling around in the mysql-db with no success.
Any help is really appreciated.
Wolfgang
== Dead compute host ==
Working with the host information
pre
i-15b9 at3-ui02 running nectarkey (376, np-rcc54) 0 m1.xxlarge
2012-06-19T00:48:11.000Z 115.146.93.60
/pre
# review the status of the host using the nova database, some of the
important information is highlighted below.
pre
SELECT * FROM instances WHERE id = CONV('15b9', 16, 10) \G;
*** 1. row ***
created_at: 2012-06-19 00:48:11
updated_at: 2012-07-03 00:35:11
deleted_at: NULL
...
id: 5561
...
power_state: 5
vm_state: shutoff
...
hostname: at3-ui02
host: np-rcc54
...
uuid: 3f57699a-e773-4650-a443-b4b37eed5a06
...
task_state: NULL
...
/pre
Update the vm's compute host.
pre
UPDATE instances SET host = 'np-rcc46' WHERE uuid =
'3f57699a-e773-4650-a443-b4b37eed5a06';
/pre
Update the libvirt xml
* change the DHCPSERVER value to the host ip address.
* possibly the VNC IP if it isn't already 0.0.0.0
Dump a copy of a nwfilter to use as a template for creating the missing
nwfilter.
pre
virsh nwfilter-list
vrish nwfilter-dumpxml nova-instance-instance-.
/pre
Example of the template file
pre
filter name='nova-instance-instance-1cc6-fa163e003b43' chain='root'
uuidd5f6f610-d0b8-4407-ae00-5dabef80677a/uuid
filterref filter='nova-base'/
/filter
/pre
The filter name value is available from the instances.xml file
(filterref filter=nova-instance-instance-1cc6-fa163e003b43).
*Note the filter name must be exact!
Generate a new uuid and replace it at the uuid value.
Update filter to match id from instance xml
pre
virsh nwfilter-define /tmp/filter.xml
virsh define libvirt.xml
virsh list --all
/pre
Kill all dnsmasq and restart nova services.
pre
killall dnsmasq; service nova-network restart; service nova-compute restart
/pre
Start the vm
pre
virsh start instance-0
/pre
On the nova DB
pre
UPDATE instances SET vm_state = 'active', power_state = 1 WHERE uuid =
'3f57699a-e773-4650-a443-b4b37eed5a06';
/pre
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
