[Openstack] Server increasing load due increasing processes in D state
Hello guys, at work we've the openstack controller that since some months started to increase its load after some days of uptime. I've seen that the cause is that processes sometimes hangs and remain in D state. I've used some combination of ps args to get these outputs: http://pastebin.com/raw.php?i=LGGzGrWu http://pastie.org/pastes/6332964/text http://pastie.org/pastes/6332979/text The hdd is a soft-raid1 over 2 disks, which SMART values are fine. Commands like lsof, strace on a D process doesn't return. Any idea on what could be the cause? Thanks in advance -- Alessandro Tagliapietra alexfu.it (http://www.alexfu.it) ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Server increasing load due increasing processes in D state
After an strace of lsof I've seen it hangs on
stat(/proc/1227/, {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open(/proc/1227/stat, O_RDONLY) = 4 read(4, 1227 (nova-dhcpbridge) D 1224
25..., 4096) = 242 close(4) = 0 readlink(/proc/1227/cwd, /..., 4096) = 1
stat(/proc/1227/cwd, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
readlink(/proc/1227/root, /, 4096) = 1 stat(/proc/1227/root,
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 readlink(/proc/1227/exe,
/usr/bin/python2.7..., 4096) = 18 stat(/proc/1227/exe,
{st_mode=S_IFREG|0755, st_size=2989480, ...}) = 0 open(/proc/1227/maps,
O_RDONLY) = 4 read(4,
Could it be a memory issue?
Actually I cannot run the memory test, maybe tomorrow. Just to know if someone
else had the same issue.
Thanks in advance
--
Alessandro Tagliapietra
alexfu.it (http://www.alexfu.it)
Il giorno lunedì 25 febbraio 2013, alle ore 12:29, Alessandro Tagliapietra ha
scritto:
Hello guys,
at work we've the openstack controller that since some months started to
increase its load after some days of uptime.
I've seen that the cause is that processes sometimes hangs and remain in D
state.
I've used some combination of ps args to get these outputs:
http://pastebin.com/raw.php?i=LGGzGrWu
http://pastie.org/pastes/6332964/text
http://pastie.org/pastes/6332979/text
The hdd is a soft-raid1 over 2 disks, which SMART values are fine.
Commands like lsof, strace on a D process doesn't return.
Any idea on what could be the cause?
Thanks in advance
--
Alessandro Tagliapietra
alexfu.it (http://www.alexfu.it)
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] DHCP and kernel 3.2
2012/9/5 andi abes andi.a...@gmail.com I've heard of folks having issues with UDP checksum not being generated correctly, and having success by running the command below on the nova compute nodes. iptables -A POSTROUTING -t mangle -p udp --dport 68 -j CHECKSUM --checksum-fill Yeah, I've seen the checksum issue on 3.2 kernels, but it's there also on 3.2 which works fine. Maybe they've added a more strict check. A temp workaround is to put the interface in promiscous mode which some guys reported it to work. Ale I'm not sure if this affects the versions you're working with. On Wed, Sep 5, 2012 at 10:41 AM, Anton Haldin ahal...@griddynamics.com wrote: I have the same issue ( kernel 3.5 , guest vm cannot get response from dnsmasq ) On Fri, Aug 10, 2012 at 7:08 AM, Lorin Hochstein lo...@nimbisservices.com wrote: On Aug 9, 2012, at 3:22 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hello guys, i've just installed kernel 3.4 from Ubuntu kernel PPA archive and after this upgrade VM aren't able to get the DHCP address but with tcpdump i see the request and offer on the network. Someone else experienced this? I've tried also with 3.3, same story. Rolling back to 3.2 and everything works fine. When I had a similar problem the issue turned out to be that I needed to configure the NIC on the compute host to be in promiscuous mode, otherwise the DHCP response wouldn't make it to the VM. Lorin Sent from my iPad ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] HELP: All instances automatically rebooted.
Try to disable • # start_guests_on_host_boot=false in nova.conf, if nova-compute dies it doesn't means that the hypervisor is death. Il giorno 28/ago/2012, alle ore 04:42, Sam Su susltd...@gmail.com ha scritto: Thanks for your guys help! I guess this problem may be caused by nova packages auto upgrade. I just found these two lines in the file /var/log/kern.log: Aug 23 06:34:33 cnode-01 kernel: [4955691.256036] init: nova-network main process (9191) terminated with status 143 Aug 23 06:34:35 cnode-01 kernel: [4955693.402082] init: nova-compute main process (9275) terminated with status 143 Here is the link for detail of /var/log/kern.log http://pastebin.com/GqH1ju1R Also found these info in /var/log/dpkg.log: 2012-08-23 06:34:33 upgrade nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3 2012-08-23 06:34:33 status half-configured nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:33 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:33 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12 2012-08-23 06:34:34 status triggers-pending man-db 2.6.1-2 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:34 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.3 2012-08-23 06:34:34 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.3 2012-08-23 06:34:34 upgrade nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3 2012-08-23 06:34:34 status half-configured nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:35 status half-installed nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:35 status half-installed nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.3 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.3 2012-08-23 06:34:35 upgrade nova-compute 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3 Here is detail: http://pastebin.com/juiSxCue But I am not 100% sure of this. Does anyone knows what it means that a process was terminated with status 143 in Ubuntu 12.04? Thanks ahead, Sam On Mon, Aug 27, 2012 at 5:56 PM, Alejandro Comisario alejandro.comisa...@mercadolibre.com wrote: One of the things i dont like in essex. That the autostart flag in nova.conf with KVM doesnt work with the autostart feature of libvirt/kvm, so if, for some reason you need to restart nova-compute to apply some kind of modification, the instances get soft/hard rebooted because now nova-compute handles the autostart flag from nova.conf Why is that if some one can explain ? On Mon, Aug 27, 2012 at 9:21 PM, Gabe Westmaas gabe.westm...@rackspace.com wrote: Hey Sam, Is it possible your hypervisors restarted? I see this entry in the logs: 2012-08-23 06:35:02 INFO nova.compute.manager [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance: ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute restart. Gabe From: openstack-bounces+gabe.westmaas=rackspace@lists.launchpad.net [mailto:openstack-bounces+gabe.westmaas=rackspace@lists.launchpad.net] On Behalf Of Sam Su Sent: Monday, August 27, 2012 8:10 PM To: openstack Subject: [Openstack] HELP: All instances automatically rebooted. Hi, I have an Essex cluster with 6 compute nodes and one control nodes. All compute nodes are working not any interrupted, for some reason all instances in my cluster automatically rebooted. I am trying to but not figured out why this happened in these couple of days. It's much appreciated if someone can give me some hints about how to deal with this situation. Logs in my /var/log/upstart/nova-compute.log: http://pastebin.com/WYJtS5a5 Let me know if need more info. TIA, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Alejandro ___ Mailing
Re: [Openstack] Cannot create snapshots of instances running not on the controller
Il giorno 27/ago/2012, alle ore 19:12, Vishvananda Ishaya vishvana...@gmail.com ha scritto: a snapshot and an image are the same. The only difference is a piece of metadata saying what instance the snapshot came from. Correct, but if one fails and the other not, the api call done by the dashboard should be different, isn't it? Alessandro Vish On Aug 27, 2012, at 6:06 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Btw, using nova image-create --poll 4dcd5bb6-c65b-47dd-9c87-ba3fed624e22 Instance works fine (nova command is run on 1st node), just it creates a new image and not a snapshot. Best -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it Il giorno 26/ago/2012, alle ore 18:49, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com ha scritto: Il giorno 25/ago/2012, alle ore 01:15, Vishvananda Ishaya vishvana...@gmail.com ha scritto: Actually it looks like a different error. For some reason container format is being sent in as none on the second node. Is it possible the original image that you launched the vm from has been deleted? For some reason it can't determine the container format. Nope, the image from which the instance has been created is still there. If not, can you also make sure that your versions of glance and python-glanceclient are the same on both nodes? you should be able to do `pip freeze` to see the installed versions. I'm using the latest version from ubuntu 12.04 repo, btw, i can see only: glance==2012.1 from pip freeze, no python-glanceclient there. Vish On Aug 24, 2012, at 12:10 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hi Vish, I had already a setting: glance_api_servers=10.0.0.1:9292 i've also tried to add glance_host=10.0.0.1 but i got the same error.. Also, after changing configuration and restarting nova-compute restarts all instances, is that normal? Best Alessandro Il giorno 23/ago/2012, alle ore 20:24, Vishvananda Ishaya vishvana...@gmail.com ha scritto: looks like the compute node has a bad setting for glance_api_servers on the second node. because glance_api_servers defaults to $glance_host:$glance_port, you should be able to fix it by setting: glance_host = ip where glance is running in your nova.conf on the second node. Vish On Aug 23, 2012, at 10:15 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hi all, i've a controller which is running all service and a secondary controller which is un multi_host so it's running compute network and api-metadata. From the dashboard i can successfully create snapshots of instances running on the controller but when i try to create a snapshot of an instance on a compute node i get in its logs: == /var/log/nova/nova-compute.log == 2012-08-23 19:08:14 ERROR nova.rpc.amqp [req-66389a04-b071-4641-949b-3df04da85d08 a63f5293c5454a979bddff1415a216f6 e8c3367ff91d44b1ab1b14eb63f48bf7] Exception during message handling 2012-08-23 19:08:14 TRACE nova.rpc.amqp Traceback (most recent call last): 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py, line 253, in _process_data 2012-08-23 19:08:14 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 183, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp sys.exc_info()) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/contextlib.py, line 24, in __exit__ 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.gen.next() 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 177, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 952, in snapshot_instance 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.driver.snapshot(context, instance_ref, image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py, line 714, in snapshot 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_file) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 306, in update 2012
Re: [Openstack] Cannot create snapshots of instances running not on the controller
Btw, using nova image-create --poll 4dcd5bb6-c65b-47dd-9c87-ba3fed624e22 Instance works fine (nova command is run on 1st node), just it creates a new image and not a snapshot. Best -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it Il giorno 26/ago/2012, alle ore 18:49, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com ha scritto: Il giorno 25/ago/2012, alle ore 01:15, Vishvananda Ishaya vishvana...@gmail.com ha scritto: Actually it looks like a different error. For some reason container format is being sent in as none on the second node. Is it possible the original image that you launched the vm from has been deleted? For some reason it can't determine the container format. Nope, the image from which the instance has been created is still there. If not, can you also make sure that your versions of glance and python-glanceclient are the same on both nodes? you should be able to do `pip freeze` to see the installed versions. I'm using the latest version from ubuntu 12.04 repo, btw, i can see only: glance==2012.1 from pip freeze, no python-glanceclient there. Vish On Aug 24, 2012, at 12:10 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hi Vish, I had already a setting: glance_api_servers=10.0.0.1:9292 i've also tried to add glance_host=10.0.0.1 but i got the same error.. Also, after changing configuration and restarting nova-compute restarts all instances, is that normal? Best Alessandro Il giorno 23/ago/2012, alle ore 20:24, Vishvananda Ishaya vishvana...@gmail.com ha scritto: looks like the compute node has a bad setting for glance_api_servers on the second node. because glance_api_servers defaults to $glance_host:$glance_port, you should be able to fix it by setting: glance_host = ip where glance is running in your nova.conf on the second node. Vish On Aug 23, 2012, at 10:15 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hi all, i've a controller which is running all service and a secondary controller which is un multi_host so it's running compute network and api-metadata. From the dashboard i can successfully create snapshots of instances running on the controller but when i try to create a snapshot of an instance on a compute node i get in its logs: == /var/log/nova/nova-compute.log == 2012-08-23 19:08:14 ERROR nova.rpc.amqp [req-66389a04-b071-4641-949b-3df04da85d08 a63f5293c5454a979bddff1415a216f6 e8c3367ff91d44b1ab1b14eb63f48bf7] Exception during message handling 2012-08-23 19:08:14 TRACE nova.rpc.amqp Traceback (most recent call last): 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py, line 253, in _process_data 2012-08-23 19:08:14 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 183, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp sys.exc_info()) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/contextlib.py, line 24, in __exit__ 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.gen.next() 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 177, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 952, in snapshot_instance 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.driver.snapshot(context, instance_ref, image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py, line 714, in snapshot 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_file) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 306, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp _reraise_translated_image_exception(image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 304, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_meta = client.update_image(image_id, image_meta, data) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/client.py, line 195, in update_image 2012-08-23 19:08:14 TRACE
Re: [Openstack] Cannot create snapshots of instances running not on the controller
Il giorno 25/ago/2012, alle ore 01:15, Vishvananda Ishaya vishvana...@gmail.com ha scritto: Actually it looks like a different error. For some reason container format is being sent in as none on the second node. Is it possible the original image that you launched the vm from has been deleted? For some reason it can't determine the container format. Nope, the image from which the instance has been created is still there. If not, can you also make sure that your versions of glance and python-glanceclient are the same on both nodes? you should be able to do `pip freeze` to see the installed versions. I'm using the latest version from ubuntu 12.04 repo, btw, i can see only: glance==2012.1 from pip freeze, no python-glanceclient there. Vish On Aug 24, 2012, at 12:10 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hi Vish, I had already a setting: glance_api_servers=10.0.0.1:9292 i've also tried to add glance_host=10.0.0.1 but i got the same error.. Also, after changing configuration and restarting nova-compute restarts all instances, is that normal? Best Alessandro Il giorno 23/ago/2012, alle ore 20:24, Vishvananda Ishaya vishvana...@gmail.com ha scritto: looks like the compute node has a bad setting for glance_api_servers on the second node. because glance_api_servers defaults to $glance_host:$glance_port, you should be able to fix it by setting: glance_host = ip where glance is running in your nova.conf on the second node. Vish On Aug 23, 2012, at 10:15 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hi all, i've a controller which is running all service and a secondary controller which is un multi_host so it's running compute network and api-metadata. From the dashboard i can successfully create snapshots of instances running on the controller but when i try to create a snapshot of an instance on a compute node i get in its logs: == /var/log/nova/nova-compute.log == 2012-08-23 19:08:14 ERROR nova.rpc.amqp [req-66389a04-b071-4641-949b-3df04da85d08 a63f5293c5454a979bddff1415a216f6 e8c3367ff91d44b1ab1b14eb63f48bf7] Exception during message handling 2012-08-23 19:08:14 TRACE nova.rpc.amqp Traceback (most recent call last): 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py, line 253, in _process_data 2012-08-23 19:08:14 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 183, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp sys.exc_info()) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/contextlib.py, line 24, in __exit__ 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.gen.next() 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 177, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 952, in snapshot_instance 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.driver.snapshot(context, instance_ref, image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py, line 714, in snapshot 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_file) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 306, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp _reraise_translated_image_exception(image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 304, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_meta = client.update_image(image_id, image_meta, data) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/client.py, line 195, in update_image 2012-08-23 19:08:14 TRACE nova.rpc.amqp res = self.do_request(PUT, /images/%s % image_id, body, headers) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 58, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return func(self, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 420
Re: [Openstack] Cannot create snapshots of instances running not on the controller
Hi Vish, I had already a setting: glance_api_servers=10.0.0.1:9292 i've also tried to add glance_host=10.0.0.1 but i got the same error.. Also, after changing configuration and restarting nova-compute restarts all instances, is that normal? Best Alessandro Il giorno 23/ago/2012, alle ore 20:24, Vishvananda Ishaya vishvana...@gmail.com ha scritto: looks like the compute node has a bad setting for glance_api_servers on the second node. because glance_api_servers defaults to $glance_host:$glance_port, you should be able to fix it by setting: glance_host = ip where glance is running in your nova.conf on the second node. Vish On Aug 23, 2012, at 10:15 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hi all, i've a controller which is running all service and a secondary controller which is un multi_host so it's running compute network and api-metadata. From the dashboard i can successfully create snapshots of instances running on the controller but when i try to create a snapshot of an instance on a compute node i get in its logs: == /var/log/nova/nova-compute.log == 2012-08-23 19:08:14 ERROR nova.rpc.amqp [req-66389a04-b071-4641-949b-3df04da85d08 a63f5293c5454a979bddff1415a216f6 e8c3367ff91d44b1ab1b14eb63f48bf7] Exception during message handling 2012-08-23 19:08:14 TRACE nova.rpc.amqp Traceback (most recent call last): 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py, line 253, in _process_data 2012-08-23 19:08:14 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 183, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp sys.exc_info()) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/contextlib.py, line 24, in __exit__ 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.gen.next() 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 177, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 952, in snapshot_instance 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.driver.snapshot(context, instance_ref, image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py, line 714, in snapshot 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_file) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 306, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp _reraise_translated_image_exception(image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 304, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_meta = client.update_image(image_id, image_meta, data) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/client.py, line 195, in update_image 2012-08-23 19:08:14 TRACE nova.rpc.amqp res = self.do_request(PUT, /images/%s % image_id, body, headers) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 58, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return func(self, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 420, in do_request 2012-08-23 19:08:14 TRACE nova.rpc.amqp headers=headers) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 75, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return func(self, method, url, body, headers) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 547, in _do_request 2012-08-23 19:08:14 TRACE nova.rpc.amqp raise exception.Invalid(res.read()) 2012-08-23 19:08:14 TRACE nova.rpc.amqp Invalid: Data supplied was not valid. 2012-08-23 19:08:14 TRACE nova.rpc.amqp Details: 400 Bad Request 2012-08-23 19:08:14 TRACE nova.rpc.amqp 2012-08-23 19:08:14 TRACE nova.rpc.amqp The server could not comply with the request since it is either malformed or otherwise incorrect. 2012-08-23 19:08:14 TRACE nova.rpc.amqp 2012-08-23 19
[Openstack] Cannot create snapshots of instances running not on the controller
Hi all, i've a controller which is running all service and a secondary controller which is un multi_host so it's running compute network and api-metadata. From the dashboard i can successfully create snapshots of instances running on the controller but when i try to create a snapshot of an instance on a compute node i get in its logs: == /var/log/nova/nova-compute.log == 2012-08-23 19:08:14 ERROR nova.rpc.amqp [req-66389a04-b071-4641-949b-3df04da85d08 a63f5293c5454a979bddff1415a216f6 e8c3367ff91d44b1ab1b14eb63f48bf7] Exception during message handling 2012-08-23 19:08:14 TRACE nova.rpc.amqp Traceback (most recent call last): 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py, line 253, in _process_data 2012-08-23 19:08:14 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 183, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp sys.exc_info()) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/contextlib.py, line 24, in __exit__ 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.gen.next() 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 177, in decorated_function 2012-08-23 19:08:14 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 952, in snapshot_instance 2012-08-23 19:08:14 TRACE nova.rpc.amqp self.driver.snapshot(context, instance_ref, image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return f(*args, **kw) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py, line 714, in snapshot 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_file) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 306, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp _reraise_translated_image_exception(image_id) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/image/glance.py, line 304, in update 2012-08-23 19:08:14 TRACE nova.rpc.amqp image_meta = client.update_image(image_id, image_meta, data) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/client.py, line 195, in update_image 2012-08-23 19:08:14 TRACE nova.rpc.amqp res = self.do_request(PUT, /images/%s % image_id, body, headers) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 58, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return func(self, *args, **kwargs) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 420, in do_request 2012-08-23 19:08:14 TRACE nova.rpc.amqp headers=headers) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 75, in wrapped 2012-08-23 19:08:14 TRACE nova.rpc.amqp return func(self, method, url, body, headers) 2012-08-23 19:08:14 TRACE nova.rpc.amqp File /usr/lib/python2.7/dist-packages/glance/common/client.py, line 547, in _do_request 2012-08-23 19:08:14 TRACE nova.rpc.amqp raise exception.Invalid(res.read()) 2012-08-23 19:08:14 TRACE nova.rpc.amqp Invalid: Data supplied was not valid. 2012-08-23 19:08:14 TRACE nova.rpc.amqp Details: 400 Bad Request 2012-08-23 19:08:14 TRACE nova.rpc.amqp 2012-08-23 19:08:14 TRACE nova.rpc.amqp The server could not comply with the request since it is either malformed or otherwise incorrect. 2012-08-23 19:08:14 TRACE nova.rpc.amqp 2012-08-23 19:08:14 TRACE nova.rpc.amqp Failed to update image metadata. Got error: Data supplied was not valid. Details: 400 Bad Request The server could not comply with the request since it is either malformed or otherwise incorrect. Failed to update image metadata. Got error: Data supplied was not valid. Details: Invalid container format 'None' for image. 2012-08-23 19:08:14 TRACE nova.rpc.amqp any idea? Using nova create-image works but it creates a new image, not a snapshot (as the dashboard shows) Best Regards Alessandro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Closr.it
Hello guys,
I'm having a problem with nova command line. I've added this to .bashrc:
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/;
but when i do nova --debug list i get:
connect: (127.0.0.1, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 127.0.0.1:5000\r\nContent-Length:
115\r\ncontent-type: application/json\r\naccept-encoding: gzip,
deflate\r\naccept: application/json\r\nuser-agent:
python-novaclient\r\n\r\n{auth: {tenantName: admin,
passwordCredentials: {username: admin, password: admin}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Date: Wed, 22 Aug 2012 09:39:20 GMT
header: Transfer-Encoding: chunked
DEBUG (shell:416) n/a (HTTP 400)
Traceback (most recent call last):
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 413, in main
OpenStackComputeShell().main(sys.argv[1:])
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 364, in main
args.func(self.cs, args)
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py, line 520,
in do_list
utils.print_list(cs.servers.list(search_opts=search_opts), columns,
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py, line 279,
in list
return self._list(/servers%s%s % (detail, query_string), servers)
File /usr/lib/python2.7/dist-packages/novaclient/base.py, line 71, in _list
resp, body = self.api.client.get(url)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 136, in get
return self._cs_request(url, 'GET', **kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 124, in
_cs_request
**kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 107, in
request
raise exceptions.from_response(resp, body)
BadRequest: n/a (HTTP 400)
ERROR: n/a (HTTP 400)
I've dumped the traffic with tcpdump and i see a correct response from
keystone, this is the output:
http://pastie.org/private/ms7c5jlpatolsgiosjsva
How can i solve this?
Best Regards
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] uninstall
Il giorno 21/ago/2012, alle ore 23:15, Guillermo Alvarado stallion.a...@gmail.com ha scritto: Hello everyone, Which is the best way to uninstall all the components? I want to re-install it because I have an inconsistent environment. if you're on debian/ubuntu apt-get purge nova.* keystone.* glance.* swift.* Best Thanks in advance!! ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Closr.it
Sorry for the wrong subject, it was another mail that i was sending.
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
Il giorno 22/ago/2012, alle ore 11:40, Alessandro Tagliapietra
tagliapietra.alessan...@gmail.com ha scritto:
Hello guys,
I'm having a problem with nova command line. I've added this to .bashrc:
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/;
but when i do nova --debug list i get:
connect: (127.0.0.1, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 127.0.0.1:5000\r\nContent-Length:
115\r\ncontent-type: application/json\r\naccept-encoding: gzip,
deflate\r\naccept: application/json\r\nuser-agent:
python-novaclient\r\n\r\n{auth: {tenantName: admin,
passwordCredentials: {username: admin, password: admin}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Date: Wed, 22 Aug 2012 09:39:20 GMT
header: Transfer-Encoding: chunked
DEBUG (shell:416) n/a (HTTP 400)
Traceback (most recent call last):
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 413, in
main
OpenStackComputeShell().main(sys.argv[1:])
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 364, in
main
args.func(self.cs, args)
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py, line 520,
in do_list
utils.print_list(cs.servers.list(search_opts=search_opts), columns,
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py, line
279, in list
return self._list(/servers%s%s % (detail, query_string), servers)
File /usr/lib/python2.7/dist-packages/novaclient/base.py, line 71, in
_list
resp, body = self.api.client.get(url)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 136, in
get
return self._cs_request(url, 'GET', **kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 124, in
_cs_request
**kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 107, in
request
raise exceptions.from_response(resp, body)
BadRequest: n/a (HTTP 400)
ERROR: n/a (HTTP 400)
I've dumped the traffic with tcpdump and i see a correct response from
keystone, this is the output:
http://pastie.org/private/ms7c5jlpatolsgiosjsva
How can i solve this?
Best Regards
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Closr.it
Hi Salvatore,
sorry but in nova i don't see any --no_cache option nor a .novaclient folder in
user home. :/
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
Il giorno 22/ago/2012, alle ore 11:52, Salvatore Orlando sorla...@nicira.com
ha scritto:
Hi Alessandro,
When I get 400 errors from nova command line it is usually because of invalid
values in the cache.
Try executing your commands with the --no_cache option or destroying the
cache altogether by removing the .novaclient directory in your home dir.
Salvatore
On 22 August 2012 11:45, Alessandro Tagliapietra
tagliapietra.alessan...@gmail.com wrote:
Sorry for the wrong subject, it was another mail that i was sending.
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
Il giorno 22/ago/2012, alle ore 11:40, Alessandro Tagliapietra
tagliapietra.alessan...@gmail.com ha scritto:
Hello guys,
I'm having a problem with nova command line. I've added this to .bashrc:
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/;
but when i do nova --debug list i get:
connect: (127.0.0.1, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 127.0.0.1:5000\r\nContent-Length:
115\r\ncontent-type: application/json\r\naccept-encoding: gzip,
deflate\r\naccept: application/json\r\nuser-agent:
python-novaclient\r\n\r\n{auth: {tenantName: admin,
passwordCredentials: {username: admin, password: admin}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Date: Wed, 22 Aug 2012 09:39:20 GMT
header: Transfer-Encoding: chunked
DEBUG (shell:416) n/a (HTTP 400)
Traceback (most recent call last):
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 413, in
main
OpenStackComputeShell().main(sys.argv[1:])
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 364, in
main
args.func(self.cs, args)
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py, line
520, in do_list
utils.print_list(cs.servers.list(search_opts=search_opts), columns,
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py, line
279, in list
return self._list(/servers%s%s % (detail, query_string), servers)
File /usr/lib/python2.7/dist-packages/novaclient/base.py, line 71, in
_list
resp, body = self.api.client.get(url)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 136, in
get
return self._cs_request(url, 'GET', **kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 124, in
_cs_request
**kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 107, in
request
raise exceptions.from_response(resp, body)
BadRequest: n/a (HTTP 400)
ERROR: n/a (HTTP 400)
I've dumped the traffic with tcpdump and i see a correct response from
keystone, this is the output:
http://pastie.org/private/ms7c5jlpatolsgiosjsva
How can i solve this?
Best Regards
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Closr.it
Found the problem,
in the endpoint list i had an url with http:/10.x.. so it was missing a /
I'm writing just to keep track of this possible error.
Best
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
Il giorno 22/ago/2012, alle ore 11:55, Alessandro Tagliapietra
tagliapietra.alessan...@gmail.com ha scritto:
Hi Salvatore,
sorry but in nova i don't see any --no_cache option nor a .novaclient folder
in user home. :/
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
Il giorno 22/ago/2012, alle ore 11:52, Salvatore Orlando
sorla...@nicira.com ha scritto:
Hi Alessandro,
When I get 400 errors from nova command line it is usually because of
invalid values in the cache.
Try executing your commands with the --no_cache option or destroying the
cache altogether by removing the .novaclient directory in your home dir.
Salvatore
On 22 August 2012 11:45, Alessandro Tagliapietra
tagliapietra.alessan...@gmail.com wrote:
Sorry for the wrong subject, it was another mail that i was sending.
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
Il giorno 22/ago/2012, alle ore 11:40, Alessandro Tagliapietra
tagliapietra.alessan...@gmail.com ha scritto:
Hello guys,
I'm having a problem with nova command line. I've added this to .bashrc:
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/;
but when i do nova --debug list i get:
connect: (127.0.0.1, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost:
127.0.0.1:5000\r\nContent-Length: 115\r\ncontent-type:
application/json\r\naccept-encoding: gzip, deflate\r\naccept:
application/json\r\nuser-agent: python-novaclient\r\n\r\n{auth:
{tenantName: admin, passwordCredentials: {username: admin,
password: admin}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Date: Wed, 22 Aug 2012 09:39:20 GMT
header: Transfer-Encoding: chunked
DEBUG (shell:416) n/a (HTTP 400)
Traceback (most recent call last):
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 413, in
main
OpenStackComputeShell().main(sys.argv[1:])
File /usr/lib/python2.7/dist-packages/novaclient/shell.py, line 364, in
main
args.func(self.cs, args)
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py, line
520, in do_list
utils.print_list(cs.servers.list(search_opts=search_opts), columns,
File /usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py, line
279, in list
return self._list(/servers%s%s % (detail, query_string), servers)
File /usr/lib/python2.7/dist-packages/novaclient/base.py, line 71, in
_list
resp, body = self.api.client.get(url)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 136,
in get
return self._cs_request(url, 'GET', **kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 124,
in _cs_request
**kwargs)
File /usr/lib/python2.7/dist-packages/novaclient/client.py, line 107,
in request
raise exceptions.from_response(resp, body)
BadRequest: n/a (HTTP 400)
ERROR: n/a (HTTP 400)
I've dumped the traffic with tcpdump and i see a correct response from
keystone, this is the output:
http://pastie.org/private/ms7c5jlpatolsgiosjsva
How can i solve this?
Best Regards
--
Alessandro Tagliapietra | VISup srl
piazza 4 novembre 7
20124 Milano
http://www.visup.it
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] DHCP and kernel 3.2
Hello guys, i've just installed kernel 3.4 from Ubuntu kernel PPA archive and after this upgrade VM aren't able to get the DHCP address but with tcpdump i see the request and offer on the network. Someone else experienced this? I've tried also with 3.3, same story. Rolling back to 3.2 and everything works fine. I've tried with both flatdhcp and vlan mode. Best Alessandro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] DHCP and kernel 3.2
Il giorno 09/ago/2012, alle ore 10:44, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com ha scritto: Il giorno 09/ago/2012, alle ore 10:19, Kiall Mac Innes ki...@managedit.ie ha scritto: That sounds like a kernel, kvm or dnsmasq issue, rather than OpenStack itself. I think Quantal is on the 3.5 kernel, and I assume OpenStack is working there.. Maybe give it's dnsmasq package a go first as it's probably the easiest thing to check… I think that dnsmasq is not the issue, as it's replying to dhcp requests with a dhcp reply. Maybe more a kernel issue with packets, because they don't reach the vm or the vm doesn't receive them. Ubuntu also have some 3.5 packages for Precise, although they are test packages.. http://packages.qa.ubuntu.com/qatracker/milestones/223/builds Going to try with them. I'll reply in a few mins. Got the same issue, this is the tcpdump output: http://pastie.org/4425169 and the vm doesn't get the ip and give up due timeout. Best Alessandro Best Alessandro Thanks, Kiall On Aug 9, 2012 8:14 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hello guys, i've just installed kernel 3.4 from Ubuntu kernel PPA archive and after this upgrade VM aren't able to get the DHCP address but with tcpdump i see the request and offer on the network. Someone else experienced this? I've tried also with 3.3, same story. Rolling back to 3.2 and everything works fine. I've tried with both flatdhcp and vlan mode. Best Alessandro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] No handlers could be found for logger keystoneclient.v2_0.client (HTTP 404)？？？
Have you exported vars as the guide: export SERVICE_ENDPOINT=http://localhost:35357/v2.0; export SERVICE_TOKEN=admin Best Il giorno 03/ago/2012, alle ore 09:37, 王鹏 breakwin...@gmail.com ha scritto: Everyone: Hi! When I install keystone alone, find a error, that’s make me crazy !the situation as follow: This is my conf [DEFAULT] bind_host = 0.0.0.0 public_port = 5000 admin_port = 35357 admin_token = admin compute_port = 8774 verbose = True debug = True log_config = /etc/keystone/logging.conf # = Syslog Options # Send logs to syslog (/dev/log) instead of to file specified # by `log-file` use_syslog = False # Facility to use. If unset defaults to LOG_USER. # syslog_log_facility = LOG_LOCAL0 [sql] #connection = sqlite:var/lib/keystone/keystone.db connection = mysql://keystone:***@172.18.32.7/keystone idle_timeout = 200 [ldap] #url = ldap://localhost #tree_dn = dc=example,dc=com #user_tree_dn = ou=Users,dc=example,dc=com #role_tree_dn = ou=Roles,dc=example,dc=com #tenant_tree_dn = ou=Groups,dc=example,dc=com #user = dc=Manager,dc=example,dc=com #password = freeipa4all #suffix = cn=example,cn=com [identity] driver = keystone.identity.backends.sql.Identity [catalog] driver = keystone.catalog.backends.sql.Catalog [token] driver = keystone.token.backends.sql.Token # Amount of time a token should remain valid (in seconds) expiration = 86400 [policy] driver = keystone.policy.backends.rules.Policy [ec2] driver = keystone.contrib.ec2.backends.sql.Ec2 [filter:debug] paste.filter_factory = keystone.common.wsgi:Debug.factory [filter:token_auth] paste.filter_factory = keystone.middleware:TokenAuthMiddleware.factory [filter:admin_token_auth] paste.filter_factory = keystone.middleware:AdminTokenAuthMiddleware.factory [filter:xml_body] paste.filter_factory = keystone.middleware:XmlBodyMiddleware.factory [filter:json_body] paste.filter_factory = keystone.middleware:JsonBodyMiddleware.factory [filter:crud_extension] paste.filter_factory = keystone.contrib.admin_crud:CrudExtension.factory [filter:ec2_extension] paste.filter_factory = keystone.contrib.ec2:Ec2Extension.factory [app:public_service] paste.app_factory = keystone.service:public_app_factory [app:admin_service] paste.app_factory = keystone.service:admin_app_factory [pipeline:public_api] pipeline = token_auth admin_token_auth xml_body json_body debug ec2_extension public_service [pipeline:admin_api] pipeline = token_auth admin_token_auth xml_body json_body debug ec2_extension crud_extension admin_service [app:public_version_service] paste.app_factory = keystone.service:public_version_app_factory [app:admin_version_service] paste.app_factory = keystone.service:admin_version_app_factory [pipeline:public_version_api] pipeline = xml_body public_version_service [pipeline:admin_version_api] pipeline = xml_body admin_version_service [composite:main] use = egg:Paste#urlmap /v2.0 = public_api / = public_version_api [composite:admin] use = egg:Paste#urlmap /v2.0 = admin_api / = admin_version_api When I restart keystone Add tenant and user : root@keystone:~# keystone user-create --tenant_id e87f632d2c4e4f2b8dfe1fd2691d229d --name admin --pass admin --enabled true No handlers could be found for logger keystoneclient.v2_0.client Unable to communicate with identity service: 404 Not Found The resource could not be found. . (HTTP 404) Why? What can I miss? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Virtual machines are not getting ip
Hello, please help, this thing is getting me crazy. The vm starts fine but i get: cloud-init-nonet waiting 120 seconds for a network device. cloud-init-nonet gave up waiting for a network device. ci-info: lo: 1 127.0.0.1 255.0.0.0 . ci-info: eth0 : 1 . . fa:16:3e:56:2c:9c route_info failed * Stopping Handle applying cloud-config[74G[ OK ] Waiting for network configuration... Waiting up to 60 more seconds for network configuration... Booting system without full network configuration… so no dhcp, but using tcpdump i see this output: http://pastie.org/4369949 and it seems ok to me. What else can be? Best ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-network and corosync
Dear Steven, it seems that nova-network breaks the network also on the interface it doesn't use, I've just done another fresh install using flat_interface eth1, and i get the issue of packets being sent with the public address. I think the effects of nova-network should be restricted to some interfaces, so it doesn't breaks other services. Best Alessandro Il giorno 19/lug/2012, alle ore 11:53, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com ha scritto: Il giorno 19/lug/2012, alle ore 09:55, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 09:40, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 00:52, Steven Dake ha scritto: On 07/18/2012 06:51 AM, Alessandro Tagliapietra wrote: Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Odd - Are you using udpu mode? Which version? Can you subscribe to the corosync list and we can follow-up there? When starting this discussion i was using ubuntu 12.04 repositories so 1.4.2, today i've installed 2.0.1, i've added corosync mailing list in cc. I have to specify that this problem happens only when i've installed nova network on both nodes (i'm using multi-host). I've tried with udpu mode specifying the nodelist and it works fine. I'm going to switch back to 1.4.2 to use regular ubuntu packages and updates and see if that works too. An update, i've removed eth2 (which was just a virtual interface) and set back flat_interface to eth1, now after a reboot i've got the same issue as before, packets are sent using public ip, but if a do a corosync restart after loggin in it works normally. Maybe it needs some delay to work? Regards Regards Let me paste some configs: nova.conf: http://pastie.org/private/c5tcutro6tp0s1te5yq (i've tried with flat_interface eth1 and eth2) ifconfig: http://pastie.org/private/7li8gwodr0ulgbafsi4edw corosync.conf: http://pastie.org/private/cjrtqx9bspgmff7rlye6ew (tried with 10.8.0.0 also as bindnetaddr) ip r: http://pastie.org/private/ckqhy0vqoiwzewuj17v7g iptables-save: http://pastie.org/private/yvypgi5ovs2rtcingrq5iw (all generated by nova-network) If you need any other things just ask. Btw, i'm going to try with udpu now. Regards http://lists.corosync.org/mailman/listinfo/discuss thanks -steve Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr
[Openstack] Virtual machine starting but never finish
Hello, i've create a new ubuntu 12.04 instance, using as image the cloud images they provide. The problem is that on logs i get this http://pastie.org/4335405 and if i check vnc i just the the cursors (not blinking) Any idea? Best -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
That's exactly what I wanted to achieve. Consider me as one of the testers if needed ;) Best Alessandro Il giorno 24/lug/2012, alle ore 20:24, Alex Glikson ha scritto: We are working on a patch that would enable such capability within OpenStack Nova. It is a bit more complex than what you suggested. In a nutshell, one would need to do the following: - re-create the VM on the destination host - re-attach volumes/IPs - update DB - make sure that if/when the original host recovers, it does not cause corruption of the instance file on shared storage (e.g., by removing the corresponding VM from the original host on startup) Regards, Alex Alex Glikson Manager, Cloud Operating System Technologies, IBM Haifa Research Lab http://w3.haifa.ibm.com/dept/stt/cloud_sys.html | https://www.research.ibm.com/haifa/dept/stt/cloud_sys.shtml Email: glik...@il.ibm.com | Phone: +972-4-8281085 | Mobile: +972-54-647 | Fax: +972-4-8296112 From:Alessandro Tagliapietra tagliapietra.alessan...@gmail.com To:Jay Pipes jaypi...@gmail.com, Cc:openstack@lists.launchpad.net Date:24/07/2012 08:52 PM Subject:Re: [Openstack] VM High Availability and Floating IP Sent by:openstack-bounces+glikson=il.ibm@lists.launchpad.net But i don't see any part (except the future plans) talking about HA at instance level, that seems more to an application level Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: Thank you Jay, never read about that. Seems something like scalr/chef? WHich handles application and keeps a minimum number of vm running? Yeah, kinda.. just one more way of doing things... :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [openstack-dev] High Available queues in rabbitmq
Yup, using as resource is a old way as http://www.rabbitmq.com/ha.html Active/active makes sure that you have no downtime and it's simple as you don't need to use DRBD. 2012/7/25 Rosa, Andrea (HP Cloud Services) andrea.r...@hp.com Sorry for my question, I have just seen from the original thread that we are talking about HA with Active/Active solution. -- Andrea Rosa -Original Message- From: Rosa, Andrea (HP Cloud Services) Sent: 25 July 2012 10:45 To: Eugene Kirpichov Cc: openstack-...@lists.openstack.org; Alessandro Tagliapietra; openstack@lists.launchpad.net Subject: Re: [openstack-dev] [Openstack] High Available queues in rabbitmq Hi Your patch doesn't use a Resource manager, so are you working on an Active/Active configuration using mirrored queues? Or are you working on a cluster configuration? I am really interested in that change, thanks for your help. Regards -- Andrea Rosa -Original Message- From: openstack-bounces+andrea.rosa=hp@lists.launchpad.net [mailto:openstack-bounces+andrea.rosa=hp@lists.launchpad.net] On Behalf Of Alessandro Tagliapietra Sent: 24 July 2012 17:58 To: Eugene Kirpichov Cc: openstack-...@lists.openstack.org; openstack@lists.launchpad.net Subject: Re: [Openstack] High Available queues in rabbitmq Oh, so without the need to put an IP floating between hosts. Good job, thanks for helping Best Alessandro Il giorno 24/lug/2012, alle ore 17:49, Eugene Kirpichov ha scritto: Hi Alessandro, My patch is about removing the need for pacemaker (and it's pacemaker that I denoted with the term TCP load balancer). I didn't submit the patch yesterday because I underestimated the effort to write unit tests for it and found a few issues on the way. I hope I'll finish today. On Tue, Jul 24, 2012 at 12:00 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Sorry for the delay, i was out from work. Awesome work Eugene, I don't need the patch instantly as i'm still building the infrastructure. Will it will take alot of time to go in Ubuntu repositories? Why you said you need load balancing? You can use only the master node and in case the rabbitmq-server dies, switch the ip to the new master with pacemaker, that's how I would do. Best Regards Alessadro Il giorno 23/lug/2012, alle ore 21:49, Eugene Kirpichov ha scritto: +openstack-dev@ To openstack-dev: this is a discussion of an upcoming patch about native RabbitMQ H/A support in nova. I'll post the patch for codereview today. On Mon, Jul 23, 2012 at 12:46 PM, Eugene Kirpichov ekirpic...@gmail.com wrote: Yup, that's basically the same thing that Jay suggested :) Obvious in retrospect... On Mon, Jul 23, 2012 at 12:42 PM, Oleg Gelbukh ogelb...@mirantis.com wrote: Eugene, I suggest just add option 'rabbit_servers' that will override 'rabbit_host'/'rabbit_port' pair, if present. This won't break anything, in my understanding. -- Best regards, Oleg Gelbukh Mirantis, Inc. On Mon, Jul 23, 2012 at 10:58 PM, Eugene Kirpichov ekirpic...@gmail.com wrote: Hi, I'm working on a RabbitMQ H/A patch right now. It actually involves more than just using H/A queues (unless you're willing to add a TCP load balancer on top of your RMQ cluster). You also need to add support for multiple RabbitMQ's directly to nova. This is not hard at all, and I have the patch ready and tested in production. Alessandro, if you need this urgently, I can send you the patch right now before the discussion codereview for inclusion in core nova. The only problem is, it breaks backward compatibility a bit: my patch assumes you have a flag rabbit_addresses which should look like rmq-host1:5672,rmq-host2:5672 instead of the prior rabbit_host and rabbit_port flags. Guys, can you advise on a way to do this without being ugly and without breaking compatibility? Maybe have rabbit_host, rabbit_port be ListOpt's? But that sounds weird, as their names are in singular. Maybe have rabbit_host, rabbit_port and also rabbit_host2, rabbit_port2 (assuming we only have clusters of 2 nodes)? Something else? On Mon, Jul 23, 2012 at 11:27 AM, Jay Pipes jaypi...@gmail.com wrote: On 07/23/2012 09:02 AM, Alessandro Tagliapietra wrote: Hi guys, just an idea, i'm deploying Openstack trying to make it HA. The missing thing is rabbitmq, which can be easily started in active/active mode, but it needs to declare the queues adding an x-ha-policy entry. http://www.rabbitmq.com/ha.html It would be nice to add a config entry to be able to declare the queues in that way. If someone know where to edit the openstack code, else i'll try to do that in the next weeks maybe. https://github.com/openstack/openstack- common/blob/master/openstack/common/rpc/impl_kombu.py You'll need to add the config options there and the queue
Re: [Openstack] High Available queues in rabbitmq
Sorry for the delay, i was out from work. Awesome work Eugene, I don't need the patch instantly as i'm still building the infrastructure. Will it will take alot of time to go in Ubuntu repositories? Why you said you need load balancing? You can use only the master node and in case the rabbitmq-server dies, switch the ip to the new master with pacemaker, that's how I would do. Best Regards Alessadro Il giorno 23/lug/2012, alle ore 21:49, Eugene Kirpichov ha scritto: +openstack-dev@ To openstack-dev: this is a discussion of an upcoming patch about native RabbitMQ H/A support in nova. I'll post the patch for codereview today. On Mon, Jul 23, 2012 at 12:46 PM, Eugene Kirpichov ekirpic...@gmail.com wrote: Yup, that's basically the same thing that Jay suggested :) Obvious in retrospect... On Mon, Jul 23, 2012 at 12:42 PM, Oleg Gelbukh ogelb...@mirantis.com wrote: Eugene, I suggest just add option 'rabbit_servers' that will override 'rabbit_host'/'rabbit_port' pair, if present. This won't break anything, in my understanding. -- Best regards, Oleg Gelbukh Mirantis, Inc. On Mon, Jul 23, 2012 at 10:58 PM, Eugene Kirpichov ekirpic...@gmail.com wrote: Hi, I'm working on a RabbitMQ H/A patch right now. It actually involves more than just using H/A queues (unless you're willing to add a TCP load balancer on top of your RMQ cluster). You also need to add support for multiple RabbitMQ's directly to nova. This is not hard at all, and I have the patch ready and tested in production. Alessandro, if you need this urgently, I can send you the patch right now before the discussion codereview for inclusion in core nova. The only problem is, it breaks backward compatibility a bit: my patch assumes you have a flag rabbit_addresses which should look like rmq-host1:5672,rmq-host2:5672 instead of the prior rabbit_host and rabbit_port flags. Guys, can you advise on a way to do this without being ugly and without breaking compatibility? Maybe have rabbit_host, rabbit_port be ListOpt's? But that sounds weird, as their names are in singular. Maybe have rabbit_host, rabbit_port and also rabbit_host2, rabbit_port2 (assuming we only have clusters of 2 nodes)? Something else? On Mon, Jul 23, 2012 at 11:27 AM, Jay Pipes jaypi...@gmail.com wrote: On 07/23/2012 09:02 AM, Alessandro Tagliapietra wrote: Hi guys, just an idea, i'm deploying Openstack trying to make it HA. The missing thing is rabbitmq, which can be easily started in active/active mode, but it needs to declare the queues adding an x-ha-policy entry. http://www.rabbitmq.com/ha.html It would be nice to add a config entry to be able to declare the queues in that way. If someone know where to edit the openstack code, else i'll try to do that in the next weeks maybe. https://github.com/openstack/openstack-common/blob/master/openstack/common/rpc/impl_kombu.py You'll need to add the config options there and the queue is declared here with the options supplied to the ConsumerBase constructor: https://github.com/openstack/openstack-common/blob/master/openstack/common/rpc/impl_kombu.py#L114 Best, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Eugene Kirpichov http://www.linkedin.com/in/eugenekirpichov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Eugene Kirpichov http://www.linkedin.com/in/eugenekirpichov -- Eugene Kirpichov http://www.linkedin.com/in/eugenekirpichov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] VM High Availability and Floating IP
Hi guys, i've 2 missing pieces in my HA openstack install. Actually all openstack services are managed by pacemaker and i can succesfully start/stop vm etc. when the cloud controller is down (i've only 2 servers atm). 1 - how can i make a VM HA? Actually live-migration works fine, but if a host goes down, how can i restart the vm on the other host? Should i edit the 'host' column in the db and issue the restart of the vm? Any other way? 2 - i've the servers hosted at Hetzner, for floating ip we've bought failover ip which are assigned to each host and can be changed via the api. So i have to make sure that if vm is on host1, floating ip associated to the vm is routed to host1. My idea was to run a job that checks the floating ip already associated to any vm, then queries the vm info, checks on which host it's running and if it's different from the other check, calls the hetzner api to switch the ip to the other server. Any other idea? Thanks in advance Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
Thank you Jay, never read about that. Seems something like scalr/chef? WHich handles application and keeps a minimum number of vm running? Best Alessandro Il giorno 24/lug/2012, alle ore 14:34, Jay Pipes ha scritto: On 07/24/2012 04:29 AM, Alessandro Tagliapietra wrote: Hi guys, i've 2 missing pieces in my HA openstack install. Actually all openstack services are managed by pacemaker and i can succesfully start/stop vm etc. when the cloud controller is down (i've only 2 servers atm). 1 - how can i make a VM HA? Actually live-migration works fine, but if a host goes down, how can i restart the vm on the other host? Should i edit the 'host' column in the db and issue the restart of the vm? Any other way? Check out that HEAT API: https://github.com/heat-api/heat/wiki/ 2 - i've the servers hosted at Hetzner, for floating ip we've bought failover ip which are assigned to each host and can be changed via the api. So i have to make sure that if vm is on host1, floating ip associated to the vm is routed to host1. My idea was to run a job that checks the floating ip already associated to any vm, then queries the vm info, checks on which host it's running and if it's different from the other check, calls the hetzner api to switch the ip to the other server. Any other idea? See above :) Best, -jay Thanks in advance Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] High Available queues in rabbitmq
Oh, so without the need to put an IP floating between hosts. Good job, thanks for helping Best Alessandro Il giorno 24/lug/2012, alle ore 17:49, Eugene Kirpichov ha scritto: Hi Alessandro, My patch is about removing the need for pacemaker (and it's pacemaker that I denoted with the term TCP load balancer). I didn't submit the patch yesterday because I underestimated the effort to write unit tests for it and found a few issues on the way. I hope I'll finish today. On Tue, Jul 24, 2012 at 12:00 AM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Sorry for the delay, i was out from work. Awesome work Eugene, I don't need the patch instantly as i'm still building the infrastructure. Will it will take alot of time to go in Ubuntu repositories? Why you said you need load balancing? You can use only the master node and in case the rabbitmq-server dies, switch the ip to the new master with pacemaker, that's how I would do. Best Regards Alessadro Il giorno 23/lug/2012, alle ore 21:49, Eugene Kirpichov ha scritto: +openstack-dev@ To openstack-dev: this is a discussion of an upcoming patch about native RabbitMQ H/A support in nova. I'll post the patch for codereview today. On Mon, Jul 23, 2012 at 12:46 PM, Eugene Kirpichov ekirpic...@gmail.com wrote: Yup, that's basically the same thing that Jay suggested :) Obvious in retrospect... On Mon, Jul 23, 2012 at 12:42 PM, Oleg Gelbukh ogelb...@mirantis.com wrote: Eugene, I suggest just add option 'rabbit_servers' that will override 'rabbit_host'/'rabbit_port' pair, if present. This won't break anything, in my understanding. -- Best regards, Oleg Gelbukh Mirantis, Inc. On Mon, Jul 23, 2012 at 10:58 PM, Eugene Kirpichov ekirpic...@gmail.com wrote: Hi, I'm working on a RabbitMQ H/A patch right now. It actually involves more than just using H/A queues (unless you're willing to add a TCP load balancer on top of your RMQ cluster). You also need to add support for multiple RabbitMQ's directly to nova. This is not hard at all, and I have the patch ready and tested in production. Alessandro, if you need this urgently, I can send you the patch right now before the discussion codereview for inclusion in core nova. The only problem is, it breaks backward compatibility a bit: my patch assumes you have a flag rabbit_addresses which should look like rmq-host1:5672,rmq-host2:5672 instead of the prior rabbit_host and rabbit_port flags. Guys, can you advise on a way to do this without being ugly and without breaking compatibility? Maybe have rabbit_host, rabbit_port be ListOpt's? But that sounds weird, as their names are in singular. Maybe have rabbit_host, rabbit_port and also rabbit_host2, rabbit_port2 (assuming we only have clusters of 2 nodes)? Something else? On Mon, Jul 23, 2012 at 11:27 AM, Jay Pipes jaypi...@gmail.com wrote: On 07/23/2012 09:02 AM, Alessandro Tagliapietra wrote: Hi guys, just an idea, i'm deploying Openstack trying to make it HA. The missing thing is rabbitmq, which can be easily started in active/active mode, but it needs to declare the queues adding an x-ha-policy entry. http://www.rabbitmq.com/ha.html It would be nice to add a config entry to be able to declare the queues in that way. If someone know where to edit the openstack code, else i'll try to do that in the next weeks maybe. https://github.com/openstack/openstack-common/blob/master/openstack/common/rpc/impl_kombu.py You'll need to add the config options there and the queue is declared here with the options supplied to the ConsumerBase constructor: https://github.com/openstack/openstack-common/blob/master/openstack/common/rpc/impl_kombu.py#L114 Best, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Eugene Kirpichov http://www.linkedin.com/in/eugenekirpichov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Eugene Kirpichov http://www.linkedin.com/in/eugenekirpichov -- Eugene Kirpichov http://www.linkedin.com/in/eugenekirpichov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Eugene Kirpichov http://www.linkedin.com/in/eugenekirpichov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack
Re: [Openstack] VM High Availability and Floating IP
But i don't see any part (except the future plans) talking about HA at instance level, that seems more to an application level Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: Thank you Jay, never read about that. Seems something like scalr/chef? WHich handles application and keeps a minimum number of vm running? Yeah, kinda.. just one more way of doing things... :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-network and corosync
Il giorno 19/lug/2012, alle ore 00:52, Steven Dake ha scritto: On 07/18/2012 06:51 AM, Alessandro Tagliapietra wrote: Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Odd - Are you using udpu mode? Which version? Can you subscribe to the corosync list and we can follow-up there? When starting this discussion i was using ubuntu 12.04 repositories so 1.4.2, today i've installed 2.0.1, i've added corosync mailing list in cc. I have to specify that this problem happens only when i've installed nova network on both nodes (i'm using multi-host). Let me paste some configs: nova.conf: http://pastie.org/private/c5tcutro6tp0s1te5yq (i've tried with flat_interface eth1 and eth2) ifconfig: http://pastie.org/private/7li8gwodr0ulgbafsi4edw corosync.conf: http://pastie.org/private/cjrtqx9bspgmff7rlye6ew (tried with 10.8.0.0 also as bindnetaddr) ip r: http://pastie.org/private/ckqhy0vqoiwzewuj17v7g iptables-save: http://pastie.org/private/yvypgi5ovs2rtcingrq5iw (all generated by nova-network) If you need any other things just ask. Btw, i'm going to try with udpu now. Regards http://lists.corosync.org/mailman/listinfo/discuss thanks -steve Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr: 10.8.0.1 Regards -steve Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-network and corosync
Il giorno 19/lug/2012, alle ore 09:55, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 09:40, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 00:52, Steven Dake ha scritto: On 07/18/2012 06:51 AM, Alessandro Tagliapietra wrote: Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Odd - Are you using udpu mode? Which version? Can you subscribe to the corosync list and we can follow-up there? When starting this discussion i was using ubuntu 12.04 repositories so 1.4.2, today i've installed 2.0.1, i've added corosync mailing list in cc. I have to specify that this problem happens only when i've installed nova network on both nodes (i'm using multi-host). I've tried with udpu mode specifying the nodelist and it works fine. I'm going to switch back to 1.4.2 to use regular ubuntu packages and updates and see if that works too. An update, i've removed eth2 (which was just a virtual interface) and set back flat_interface to eth1, now after a reboot i've got the same issue as before, packets are sent using public ip, but if a do a corosync restart after loggin in it works normally. Maybe it needs some delay to work? Regards Regards Let me paste some configs: nova.conf: http://pastie.org/private/c5tcutro6tp0s1te5yq (i've tried with flat_interface eth1 and eth2) ifconfig: http://pastie.org/private/7li8gwodr0ulgbafsi4edw corosync.conf: http://pastie.org/private/cjrtqx9bspgmff7rlye6ew (tried with 10.8.0.0 also as bindnetaddr) ip r: http://pastie.org/private/ckqhy0vqoiwzewuj17v7g iptables-save: http://pastie.org/private/yvypgi5ovs2rtcingrq5iw (all generated by nova-network) If you need any other things just ask. Btw, i'm going to try with udpu now. Regards http://lists.corosync.org/mailman/listinfo/discuss thanks -steve Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr: 10.8.0.1 Regards -steve Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http
[Openstack] nova-network and corosync
Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-network and corosync
Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr: 10.8.0.1 Regards -steve Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Nova-manage service list shows old and new hostname
Hello list, i've this problem, when i run nova-manage service list i get nova-compute server1.site.it nova enabled :-) 2012-07-13 12:15:44 nova-volume server1.site.it nova enabled :-) 2012-07-13 12:15:51 nova-network server1.site.it nova enabled :-) 2012-07-13 12:15:51 nova-consoleauth server1.site.it nova enabled :-) 2012-07-13 12:15:51 nova-scheduler server1.site.it nova enabled :-) 2012-07-13 12:15:51 nova-scheduler Ubuntu-1204-precise-64-minimal nova enabled XXX 2012-07-13 12:06:57 nova-certUbuntu-1204-precise-64-minimal nova enabled XXX 2012-07-13 12:06:57 nova-consoleauth Ubuntu-1204-precise-64-minimal nova enabled XXX 2012-07-13 12:06:57 nova-compute Ubuntu-1204-precise-64-minimal nova enabled XXX None nova-network Ubuntu-1204-precise-64-minimal nova enabled XXX 2012-07-13 12:06:57 nova-certserver1.site.it nova enabled :-) 2012-07-13 12:15:51 that's because i installed openstack changing only the hostname with the hostname command and not in /etc/hostname, laters i've changed that and restarted, now i see services from both the old and new host. Is there a way to remove those services from the hostname Ubuntu-1204-precise-64-minimal? Regards Alessandro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Cannot add floating ip
Hi guys, i've 2 hosts configured in multi_host, i had 2 vm (1 per host) with 1 floating ip each, then nova-compute on second host crashed when deleting the vm2, i've restarted host2 and it deleted the vm successfully, then i've got an error about floating ip cannot be associated to vm2 in logs, so i've removed the floating ip using nova-manage, now i'm not able anymore to add floating ip, when i do sudo nova-manage floating add 5.9.x.x i get no output in both console and /var/log/nova/* logs. Floating list doesn't show the new ip. Any way to debug this? Best regards ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cannot add floating ip
What an idtio, it's create, not add. Sorry for the spam Il giorno 13/lug/2012, alle ore 20:00, Alessandro Tagliapietra ha scritto: Hi guys, i've 2 hosts configured in multi_host, i had 2 vm (1 per host) with 1 floating ip each, then nova-compute on second host crashed when deleting the vm2, i've restarted host2 and it deleted the vm successfully, then i've got an error about floating ip cannot be associated to vm2 in logs, so i've removed the floating ip using nova-manage, now i'm not able anymore to add floating ip, when i do sudo nova-manage floating add 5.9.x.x i get no output in both console and /var/log/nova/* logs. Floating list doesn't show the new ip. Any way to debug this? Best regards ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Per host floating ip
I'm currently building up, i did a test setup with 3 machines and it worked. Now i'm doing it again with secure passwords, firewall etc. Btw, i just bought 3 EX 4 servers with a private lan between them using the switch + additional nic option they provide. I was thinking that another solution would be to use failover ip and when i assign an ip to an instance a custom hook calls the hetzner api to change the routing to point the ip to the server which is running the vm. Next step would be to use a shared fs like gluster for vm HA. I was thinking to make a blog post as soon as i get everything working to share how i did the setup. Regards Il giorno 06/lug/2012, alle ore 23:45, Michael Grosser ha scritto: Sounds like a solution, but a manual one. A more automatic way would be great I reckon. On Fri, Jul 6, 2012 at 7:00 PM, Marnus van Niekerk m...@mjvn.net wrote: I have limited knowledge of OpenStack, but I host many servers with Hetzner and know their infrastructure well. Since nova allows you to assign specific floating IPs to specific instances you could do the following. Create a floating range include all of the IPs Hetzner has given you. Then just associate the IP from the underlying host with a VM running on it that host using euca-associate-address. On 06/07/2012 18:17, Michael Grosser wrote: Hey, I would love to hear more on your setup at hetzner, I was looking to setup openstack there myself, but I didn't see a solution for the per server ip issue. How many machines do you use? How did you set it up? Do you use only nova or swift too? I would love to get more insight. Thanks for taking the time Cheers Michael On Fri, Jul 6, 2012 at 12:24 PM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hello guys, i've some servers at hetzner, i've actually installed essex on those and they're working fine, the step i'm missing is that hetzner gives you additional ip fixed to one host or failover ip that i can move to one server to another with the api. What i want is to assign floating ip to vm, depending on which host the vm is running or use a create hook to move the failover ip to the host which is running the vm. Is it possible in some way? Best Regards ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Per host floating ip
I'm on other tasks actually but as soon as i get everything working i'll send a msg to the list with blog post with my tutorial for sure. Regards Il giorno 07/lug/2012, alle ore 12:19, Marnus van Niekerk ha scritto: I have tried gluster for live migration but could net get it working. Please post back here if get that working. Instances ran fine with /var/lib/nova/instances on gluster, but live migration would change vm state to migrating and then back to running but still have it running where it was without any errors in the logs. M On Saturday, 7 July 2012, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: I'm currently building up, i did a test setup with 3 machines and it worked. Now i'm doing it again with secure passwords, firewall etc. Btw, i just bought 3 EX 4 servers with a private lan between them using the switch + additional nic option they provide. I was thinking that another solution would be to use failover ip and when i assign an ip to an instance a custom hook calls the hetzner api to change the routing to point the ip to the server which is running the vm. Next step would be to use a shared fs like gluster for vm HA. I was thinking to make a blog post as soon as i get everything working to share how i did the setup. Regards Il giorno 06/lug/2012, alle ore 23:45, Michael Grosser ha scritto: Sounds like a solution, but a manual one. A more automatic way would be great I reckon. On Fri, Jul 6, 2012 at 7:00 PM, Marnus van Niekerk m...@mjvn.net wrote: I have limited knowledge of OpenStack, but I host many servers with Hetzner and know their infrastructure well. Since nova allows you to assign specific floating IPs to specific instances you could do the following. Create a floating range include all of the IPs Hetzner has given you. Then just associate the IP from the underlying host with a VM running on it that host using euca-associate-address. On 06/07/2012 18:17, Michael Grosser wrote: Hey, I would love to hear more on your setup at hetzner, I was looking to setup openstack there myself, but I didn't see a solution for the per server ip issue. How many machines do you use? How did you set it up? Do you use only nova or swift too? I would love to get more insight. Thanks for taking the time Cheers Michael On Fri, Jul 6, 2012 at 12:24 PM, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com wrote: Hello guys, i've some servers at hetzner, i've actually installed essex on those and they're working fine, the step i'm missing is that hetzner gives you additional ip fixed to one host or failover ip that i can move to one server to another with the api. What i want is to assign floating ip to vm, depending on which host the vm is running or use a create hook to move the failover ip to the host which is running the vm. Is it possible in some way? Best Regards ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Per host floating ip
Hello guys, i've some servers at hetzner, i've actually installed essex on those and they're working fine, the step i'm missing is that hetzner gives you additional ip fixed to one host or failover ip that i can move to one server to another with the api. What i want is to assign floating ip to vm, depending on which host the vm is running or use a create hook to move the failover ip to the host which is running the vm. Is it possible in some way? Best Regards ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova-compute doesn't start on reboot, only manually
Thank you Russel, i'll wait for the fix Best Regards 2012/5/29 Russell Bryant rbry...@redhat.com On 05/28/2012 01:21 PM, Clint Byrum wrote: Looks to me that you need to make sure the other side of that RPC connection is up before nova-compute. I am not familiar with the specifics of what Nova needs at startup, but I'd guess this is nova-api or keystone. Thats a pretty easy thing to do in a single system (just mess with the upstart jobs or init scripts) but across multiple systems, you'll need some kind of orchestration layer, and even then modeling the dependencies on the network with some other tool seems like something just begging to break. In this case, it's nova-compute expecting nova-network to be up and running when it starts up. This also causes a problem when restarting all of the services at the same time, as seen here: https://bugs.launchpad.net/nova/+bug/999698 Instead, the timeout should just be multiple minutes during startup, and the services should all be able to start in parallel if they are on the same box. I always think of one of those HP EcoPOD that is pre-installed with everything you need for OpenStack, and just shipped and then turned on. You could spend a lot of time trying to get that order just right, or you could just have everything extend their timeouts and get as far as they can without contact with the other services. nova-compute doesn't *know* that the other side is in error, it just knows that it is not responding. This is not a problem with nova-compute, so why should nova-compute fail so quickly? One could even argue that nova-compute should wait *forever* for the other side. From an ops standpoint, they're both down, so why make the operations team take two actions when the actual broken service recovers? The problem is that since nova-network isn't up, the request gets lost. nova-compute is sitting there waiting for a response to a message that was never even received most likely. It's also possible that nova-network received the message but the service stopped before it responded (but that is less likely, I think). The message queues get created by the consumer of messages in nova. So, in this case, nova-network creates the queue. Some possible solutions: 1) We could adjust this code path to just loop around and try again if it hits a timeout. We could make the timeout much shorter than the default, to make recover quicker. The downside would be that we're fixing a single place, when this issue could pop up elsewhere. 2) We could make it so the sender creates the queue if it doesn't exist. This is good because it covers all cases. The bad thing is that we would not be able to set the queue to be auto-deleted in this case, so we could end up with a leak of unwanted message queues. I'm tempted to just write a patch that does #1 for now to address the immediate issue and then do something better later if we come up with something. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova-compute doesn't start on reboot, only manually
Fresh install, it did that multiple times because this is like the 6th install and i've only that problem now.RegardsIl giorno 29/mag/2012, alle ore 10:07, Razique Mahroua ha scritto: Was it a fresh install, or did you have already those components installed before ? Alessandro Tagliapietra 28 mai 2012 23:22The line before isso i think rabbitmq is already started isn't it?Best Regards Razique Mahroua 28 mai 2012 11:00 Hi Stephenlooks like the amqp server doesn't start.make sure it starts along with the nova-scheduler first.Does both services start on boot ?Razique Stephen Gran 28 mai 2012 10:53Hi, On Mon, 2012-05-28 at 11:41 +0300, Alessandro Tagliapietra wrote: Hello, i've installed openstack following the ubuntu 12.04 deploy guide, only problem is that nova-compute has to be started manually, by default it doesn't start on boot, this is the error log: ... 2012-05-27 23:48:14 TRACE nova.rpc.common timeout: timed out ... 2012-05-27 23:48:14 TRACE nova Timeout: Timeout while waiting on RPC response. Then after system boot a start nova-compute make everything working. Any idea? Both of these are MQ timeouts. You'll want to arrange for your MQ server to be running when nova starts up. How you do that depends on your local architecture. Cheers, Alessandro Tagliapietra 28 mai 2012 10:41Hello, i've installed openstack following the ubuntu 12.04 deploy guide, only problem is that nova-compute has to be started manually, by default it doesn't start on boot, this is the error log:2012-05-27 23:47:14 INFO nova.rpc.common [req-46624af9-9d2a-4901-b635-66f557d3b54c None None] Connected to AMQP server on 10.8.0.1:56722012-05-27 23:48:14 ERROR nova.rpc.common [req-46624af9-9d2a-4901-b635-66f557d3b54c None None] Timed out waiting for RPC response: timed out2012-05-27 23:48:14 TRACE nova.rpc.common Traceback (most recent call last):2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure2012-05-27 23:48:14 TRACE nova.rpc.common return method(*args, **kwargs)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume2012-05-27 23:48:14 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events2012-05-27 23:48:14 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events2012-05-27 23:48:14 TRACE nova.rpc.common return connection.drain_events(**kwargs)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events2012-05-27 23:48:14 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi2012-05-27 23:48:14 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple2012-05-27 23:48:14 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout2012-05-27 23:48:14 TRACE nova.rpc.common return self.method_reader.read_method()2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method2012-05-27 23:48:14 TRACE nova.rpc.common raise m2012-05-27 23:48:14 TRACE nova.rpc.common timeout: timed out2012-05-27 23:48:14 TRACE nova.rpc.common 2012-05-27 23:48:14CRITICAL nova [-] Timeout while waiting on RPC response.2012-05-27 23:48:14 TRACE nova Traceback (most recent call last):2012-05-27 23:48:14 TRACE nova File "/usr/bin/nova-compute", line 49, in module 2012-05-27 23:48:14 TRACE nova service.wait()2012-05-27 23:48:14 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait 2012-05-27 23:48:14 TRACE nova _launcher.wait()2012-05-27 23:48:14 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait2012-05-27 23:48:14 TRACE nova service.wait()2012-05-27 23:48:14 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait2012-05-27 23:48:14 TRACE nova
[Openstack] Nova-compute doesn't start on reboot, only manually
Hello, i've installed openstack following the ubuntu 12.04 deploy guide, only problem is that nova-compute has to be started manually, by default it doesn't start on boot, this is the error log: 2012-05-27 23:47:14 INFO nova.rpc.common [req-46624af9-9d2a-4901-b635-66f557d3b54c None None] Connected to AMQP server on 10.8.0.1:5672 2012-05-27 23:48:14 ERROR nova.rpc.common [req-46624af9-9d2a-4901-b635-66f557d3b54c None None] Timed out waiting for RPC response: timed out 2012-05-27 23:48:14 TRACE nova.rpc.common Traceback (most recent call last): 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py, line 490, in ensure 2012-05-27 23:48:14 TRACE nova.rpc.common return method(*args, **kwargs) 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py, line 567, in _consume 2012-05-27 23:48:14 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout) 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/kombu/connection.py, line 175, in drain_events 2012-05-27 23:48:14 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs) 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py, line 238, in drain_events 2012-05-27 23:48:14 TRACE nova.rpc.common return connection.drain_events(**kwargs) 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py, line 57, in drain_events 2012-05-27 23:48:14 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout) 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py, line 63, in wait_multi 2012-05-27 23:48:14 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout) 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py, line 120, in _wait_multiple 2012-05-27 23:48:14 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout) 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py, line 94, in read_timeout 2012-05-27 23:48:14 TRACE nova.rpc.common return self.method_reader.read_method() 2012-05-27 23:48:14 TRACE nova.rpc.common File /usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py, line 221, in read_method 2012-05-27 23:48:14 TRACE nova.rpc.common raise m 2012-05-27 23:48:14 TRACE nova.rpc.common timeout: timed out 2012-05-27 23:48:14 TRACE nova.rpc.common 2012-05-27 23:48:14 CRITICAL nova [-] Timeout while waiting on RPC response. 2012-05-27 23:48:14 TRACE nova Traceback (most recent call last): 2012-05-27 23:48:14 TRACE nova File /usr/bin/nova-compute, line 49, in module 2012-05-27 23:48:14 TRACE nova service.wait() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/nova/service.py, line 413, in wait 2012-05-27 23:48:14 TRACE nova _launcher.wait() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/nova/service.py, line 131, in wait 2012-05-27 23:48:14 TRACE nova service.wait() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/eventlet/greenthread.py, line 166, in wait 2012-05-27 23:48:14 TRACE nova return self._exit_event.wait() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/eventlet/event.py, line 116, in wait 2012-05-27 23:48:14 TRACE nova return hubs.get_hub().switch() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py, line 177, in switch 2012-05-27 23:48:14 TRACE nova return self.greenlet.switch() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/eventlet/greenthread.py, line 192, in main 2012-05-27 23:48:14 TRACE nova result = function(*args, **kwargs) 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/nova/service.py, line 101, in run_server 2012-05-27 23:48:14 TRACE nova server.start() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/nova/service.py, line 162, in start 2012-05-27 23:48:14 TRACE nova self.manager.init_host() 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 247, in init_host 2012-05-27 23:48:14 TRACE nova self.reboot_instance(context, instance['uuid']) 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/nova/exception.py, line 114, in wrapped 2012-05-27 23:48:14 TRACE nova return f(*args, **kw) 2012-05-27 23:48:14 TRACE nova File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 153, in decorated_function 2012-05-27 23:48:14 TRACE nova function(self, context, instance_uuid, *args, **kwargs) 2012-05-27 23:48:14 TRACE nova
Re: [Openstack] Nova-compute doesn't start on reboot, only manually
Hi Stephen, Sorry for the silly question, what you mean with MQ? The rabbit server? Regards Il giorno 28/mag/2012, alle ore 11:53, Stephen Gran ha scritto: Hi, On Mon, 2012-05-28 at 11:41 +0300, Alessandro Tagliapietra wrote: Hello, i've installed openstack following the ubuntu 12.04 deploy guide, only problem is that nova-compute has to be started manually, by default it doesn't start on boot, this is the error log: ... 2012-05-27 23:48:14 TRACE nova.rpc.common timeout: timed out ... 2012-05-27 23:48:14 TRACE nova Timeout: Timeout while waiting on RPC response. Then after system boot a start nova-compute make everything working. Any idea? Both of these are MQ timeouts. You'll want to arrange for your MQ server to be running when nova starts up. How you do that depends on your local architecture. Cheers, -- Stephen Gran Senior Systems Integrator - guardian.co.uk Please consider the environment before printing this email. -- Visit guardian.co.uk - newspaper of the year www.guardian.co.ukwww.observer.co.uk www.guardiannews.com On your mobile, visit m.guardian.co.uk or download the Guardian iPhone app www.guardian.co.uk/iphone To save up to 30% when you subscribe to the Guardian and the Observer visit www.guardian.co.uk/subscriber - This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News Media Limited A member of Guardian Media Group plc Registered Office PO Box 68164 Kings Place 90 York Way London N1P 2AP Registered in England Number 908396 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova-compute doesn't start on reboot, only manually
The line before isConnected to AMQP server on 10.8.0.1:5672so i think rabbitmq is already started isn't it?Best RegardsIl giorno 28/mag/2012, alle ore 12:00, Razique Mahroua ha scritto: Hi Stephenlooks like the amqp server doesn't start.make sure it starts along with the nova-scheduler first.Does both services start on boot ?Razique Stephen Gran 28 mai 2012 10:53Hi, On Mon, 2012-05-28 at 11:41 +0300, Alessandro Tagliapietra wrote: Hello, i've installed openstack following the ubuntu 12.04 deploy guide, only problem is that nova-compute has to be started manually, by default it doesn't start on boot, this is the error log: ... 2012-05-27 23:48:14 TRACE nova.rpc.common timeout: timed out ... 2012-05-27 23:48:14 TRACE nova Timeout: Timeout while waiting on RPC response. Then after system boot a start nova-compute make everything working. Any idea? Both of these are MQ timeouts. You'll want to arrange for your MQ server to be running when nova starts up. How you do that depends on your local architecture. Cheers, Alessandro Tagliapietra 28 mai 2012 10:41Hello, i've installed openstack following the ubuntu 12.04 deploy guide, only problem is that nova-compute has to be started manually, by default it doesn't start on boot, this is the error log:2012-05-27 23:47:14 INFO nova.rpc.common [req-46624af9-9d2a-4901-b635-66f557d3b54c None None] Connected to AMQP server on 10.8.0.1:56722012-05-27 23:48:14 ERROR nova.rpc.common [req-46624af9-9d2a-4901-b635-66f557d3b54c None None] Timed out waiting for RPC response: timed out2012-05-27 23:48:14 TRACE nova.rpc.common Traceback (most recent call last):2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure2012-05-27 23:48:14 TRACE nova.rpc.common return method(*args, **kwargs)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume2012-05-27 23:48:14 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events2012-05-27 23:48:14 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events2012-05-27 23:48:14 TRACE nova.rpc.common return connection.drain_events(**kwargs)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events2012-05-27 23:48:14 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi2012-05-27 23:48:14 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple2012-05-27 23:48:14 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout2012-05-27 23:48:14 TRACE nova.rpc.common return self.method_reader.read_method()2012-05-27 23:48:14 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method2012-05-27 23:48:14 TRACE nova.rpc.common raise m2012-05-27 23:48:14 TRACE nova.rpc.common timeout: timed out2012-05-27 23:48:14 TRACE nova.rpc.common 2012-05-27 23:48:14CRITICAL nova [-] Timeout while waiting on RPC response.2012-05-27 23:48:14 TRACE nova Traceback (most recent call last):2012-05-27 23:48:14 TRACE nova File "/usr/bin/nova-compute", line 49, in module 2012-05-27 23:48:14 TRACE nova service.wait()2012-05-27 23:48:14 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait 2012-05-27 23:48:14 TRACE nova _launcher.wait()2012-05-27 23:48:14 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait2012-05-27 23:48:14 TRACE nova service.wait()2012-05-27 23:48:14 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait2012-05-27 23:48:14 TRACE nova return self._exit_event.wait()2012-05-27 23:48:14 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait2012-05-27 23:48:14 TRACE nova return hubs.get_hub().switch()2012-05-27 23:48:14 TRACE nova File "/usr/lib/pyt
[Openstack] Packets doesn't arrive to iptables
Hello guys, i've associated a floating ip to an instance, this is ip addr result: 2: eth0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:30:48:fc:4c:92 brd ff:ff:ff:ff:ff:ff inet publicip/24 brd 188.165.225.255 scope global eth0 inet floatingip/32 scope global eth0:1 inet6 fe80::230:48ff:fefc:4c92/64 scope link valid_lft forever preferred_lft forever as you can see the floating ip is there, binded to eth0, now i was trying to know why dnat wasn't working, i've tried to add a catch-all log rule into the first table and chain iptables process with this command: iptables -t mangle -D PREROUTING -j LOG --log-prefix=PREROUTING: and i looking at logs i can see all packets except those with destination set to floatingip, also -v shows no hits to the dnat rule but i can see packets on eth0 with tcpdump. Any idea why packets are not reaching the dnat rule? Best ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova network external gateway
You can use this option dhcp-option=6,IP_OF_DNS1,IP_OF_DNS2 that should be appended on the same file. Best Il giorno 21/mag/2012, alle ore 13:34, Sergio Ariel de la Campa Saiz ha scritto: Thanks... it works ok... There is some kind of configuration for DNS too?? I change my DNS server in /etc/resolv.conf but some time later it goes back to the default one set by dsnmasq. :-( :-( Sergio Ariel de la Campa Saiz GMV-SES Infraestructura / GMV-SES Infrastructure GMV Isaac Newton, 11 P.T.M. Tres Cantos E-28760 Madrid Tel. +34 91 807 21 00 Fax +34 91 807 21 99 www.gmv.com De: George Mihaiescu [george.mihaie...@q9.com] Enviado el: viernes, 18 de mayo de 2012 16:11 Para: Sergio Ariel de la Campa Saiz CC: openstack@lists.launchpad.net; openstack-operat...@lists.openstack.org Asunto: RE: [Openstack] nova network external gateway You can try the solution proposed by Vish on February 23, 2012. Put the flag and the config file on the nodes with nova-network as they have dnsmasq running: “ Then you can use --dnsmasq_config_file=/path/to/config in that config file you can use: dhcp_opiton=3,ip of router to force vms to use your router as their gateway. “ From: openstack-bounces+george.mihaiescu=q9@lists.launchpad.net [mailto:openstack-bounces+george.mihaiescu=q9@lists.launchpad.net] On Behalf Of Sergio Ariel de la Campa Saiz Sent: Friday, May 18, 2012 8:49 AM To: openstack@lists.launchpad.net; openstack-operat...@lists.openstack.org Subject: [Openstack] nova network external gateway Hi: I have installed Essex and I´m using vlan networking. All virtual machines use the same vlan, but I want all of them use an external gateway, I mean, a real router of my network instead of a nova node. I have read about it and I have found a parameter named: dhcpoption=3,gateway ip but I don´t know where to put it. I know that dnsmasq loads it but... from where?? I did not find any dnsmasq.conf in my system and, in the other hand, I put it in my nova.conf file and nothing happened. Thanks in advance Good luck... Sergio Ariel de la Campa Saiz GMV-SES Infraestructura / GMV-SES Infrastructure GMV Isaac Newton, 11 P.T.M. Tres Cantos E-28760 Madrid Tel. +34 91 807 21 00 Fax +34 91 807 21 99 www.gmv.com size=2 width=100% align=center This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener información clasificada por su emisor como confidencial en el marco de su Sistema de Gestión de Seguridad de la Información siendo para uso exclusivo del destinatario, quedando prohibida su divulgación copia o distribución a terceros sin la autorización expresa del remitente. Si Vd. ha recibido este mensaje erróneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboración. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informação confidencial, de acordo com nosso Sistema de Gestão de Segurança da Informação, sendo para uso exclusivo do destinatário e estando proibida a sua divulgação, cópia ou distribuição a terceiros sem autorização expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaboração. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener información clasificada por su emisor como confidencial en el marco de su Sistema de Gestión de Seguridad de la Información siendo para uso exclusivo del destinatario, quedando prohibida su divulgación copia o distribución a terceros sin la autorización expresa del remitente. Si Vd. ha recibido este mensaje erróneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboración. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informação confidencial, de acordo com nosso Sistema de Gestão de Segurança da Informação, sendo para uso exclusivo do destinatário e estando proibida a sua divulgação, cópia ou distribuição a terceiros sem autorização expressa do remetente da mesma. Se recebeu
[Openstack] Problems with dnat
Hello everyone, i've this problem, vm can connect to internet but cannot receive traffic when i assign them a floating ip, these are the iptables rules created by nova-network -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N nova-api-OUTPUT -N nova-api-POSTROUTING -N nova-api-PREROUTING -N nova-api-float-snat -N nova-api-snat -N nova-compute-OUTPUT -N nova-compute-POSTROUTING -N nova-compute-PREROUTING -N nova-compute-float-snat -N nova-compute-snat -N nova-network-OUTPUT -N nova-network-POSTROUTING -N nova-network-PREROUTING -N nova-network-float-snat -N nova-network-snat -N nova-postrouting-bottom -A PREROUTING -j nova-network-PREROUTING -A PREROUTING -j nova-compute-PREROUTING -A PREROUTING -j nova-api-PREROUTING -A OUTPUT -j nova-network-OUTPUT -A OUTPUT -j nova-compute-OUTPUT -A OUTPUT -j nova-api-OUTPUT -A POSTROUTING -j nova-network-POSTROUTING -A POSTROUTING -j nova-compute-POSTROUTING -A POSTROUTING -j nova-api-POSTROUTING -A POSTROUTING -j nova-postrouting-bottom -A nova-api-snat -j nova-api-float-snat -A nova-compute-snat -j nova-compute-float-snat -A nova-network-OUTPUT -d MY_FLOATING_IP/32 -j DNAT --to-destination 192.168.4.2 -A nova-network-POSTROUTING -s 192.168.4.0/24 -d MY_PUBLIC_IP/32 -j ACCEPT -A nova-network-POSTROUTING -s 192.168.4.0/24 -d 10.128.0.0/24 -j ACCEPT -A nova-network-POSTROUTING -s 192.168.4.0/24 -d 192.168.4.0/24 -m conntrack ! --ctstate DNAT -j ACCEPT -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination MY_PUBLIC_IP:8775 -A nova-network-PREROUTING -d MY_FLOATING_IP/32 -j DNAT --to-destination 192.168.4.2 -A nova-network-float-snat -s 192.168.4.2/32 -j SNAT --to-source MY_FLOATING_IP -A nova-network-snat -j nova-network-float-snat -A nova-network-snat -s 192.168.4.0/24 -j SNAT --to-source MY_PUBLIC_IP -A nova-postrouting-bottom -j nova-network-snat -A nova-postrouting-bottom -j nova-compute-snat -A nova-postrouting-bottom -j nova-api-snat and this my nova.conf --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/run/lock/nova --allow_admin_api=true --use_deprecated_auth=false --auth_strategy=keystone --scheduler_driver=nova.scheduler.simple.SimpleScheduler --s3_host=MY_PUBLIC_IP --ec2_host=MY_PUBLIC_IP --rabbit_host=MY_PUBLIC_IP --cc_host=MY_PUBLIC_IP --nova_url=http://MY_PUBLIC_IP:8774/v1.1/ --routing_source_ip=MY_PUBLIC_IP --glance_api_servers=MY_PUBLIC_IP:9292 --image_service=nova.image.glance.GlanceImageService --iscsi_ip_prefix=192.168.4 --sql_connection=mysql://novadbadmin:mydbpassword@192.168.3.1/nova --ec2_url=http://MY_PUBLIC_IP:8773/services/Cloud --keystone_ec2_url=http://MY_PUBLIC_IP:5000/v2.0/ec2tokens --api_paste_config=/etc/nova/api-paste.ini --libvirt_type=kvm --libvirt_use_virtio_for_bridges=true --start_guests_on_host_boot=true --resume_guests_state_on_host_boot=true # vnc specific configuration --novnc_enabled=true --novncproxy_base_url=http://MY_PUBLIC_IP:6080/vnc_auto.html --vncserver_proxyclient_address=MY_PUBLIC_IP --vncserver_listen=MY_PUBLIC_IP # network specific settings --network_manager=nova.network.manager.FlatDHCPManager --public_interface=eth0 --flat_interface=eth2 --flat_network_bridge=br100 --fixed_range=192.168.4.0/24 --network_size=254 --flat_network_dhcp_start=192.168.4.1 --flat_injected=False --force_dhcp_release=true --iscsi_helper=tgtadm --connection_type=libvirt --root_helper=sudo nova-rootwrap --verbose=true with tcpdump i correctly see the syn packets on the eth0 (public interface) but can't see them on br100 or eth2 so they're not correctly dnatted. I've tried with both ip_forward enabled and disabled. Best Alessandro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Error in documentation openstack install and deploy manual
I tried to install openstack following this guide, i think these things are wrong. I'm not sure so maybe it's intentional. 1) http://docs.openstack.org/trunk/openstack-compute/install/content/keystone-service-endpoint-create.html Defining the volume service there is this line: $ TENANT=eb7e0c10a99446cfa14c244374549e9d which i think is not needed. Later in image service defining this line is missing the escape char at the end of line --endpoint http://192.168.206.130:35357/v2.0/ 2) http://docs.openstack.org/trunk/openstack-compute/install/content/configure-glance-files.html The step about db version control and sync on ubuntu 12.04 should be done before the services restart else glance-registry will die after a few seconds due no tables found Best Regards Alessandro___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Advice on network configuration
Hello list, at work we've decided to move to openstack, actually we're running proxmox for vm management and this is our actual server configuration: 2 x kimsufi 24g @ http://www.kimsufi.co.uk/ The network is configured this way on the 2 hosts: - eth0 with binded the multiple /32 public ip - eth1 (virtual bridge) with a 10.8.0.0/16 network and all vm's eth0 is bridged there - eth2 (virtual bridge) with a 192.168.0.0/24 dhcp and all vm's eth1 is bridged there The eth1 interfaces are bridged between hosts so a vm from an host can ping a vm on the other host, these are the actual features provided: - host1 host2 uses dnat to do a per-public-ip route to the correct vm, which has a fixed ip on vm's eth0 - the request can come from both host1 and host2, as each host before routing uses connmark + dscp to set from which gateway the requests came, and vm respond using the origin host - each vm directly access internet using the host which is running the vm using the vm's eth1 - i can migrate a vm from host1 to host2 without any change in the network configuration - i don't need to know which host is running the vm, i just need the vm's ip I've set a google doc to show a small example https://docs.google.com/document/d/1oIhUE7eAWDMtKdJlFmCw1jujo4XOGHD9w_ex_6TrGkw/edit How can i implement this using openstack? Is that possible? As the private network's bandwith is shared with the public one because it's incapsulated between servers, this is what i need: - each vm uses its own host as default gateway - if a host receive a request for the public ip of a vm in another host it uses the private network to reach the correct vm in the other host - if a host receive a request for the public ip of a vm on that host it correctly route to the correct vm PS: in addiction of these 2 servers i was thinking to use a vps to run the services except compute and storage to be a always available cloud controller. 2nd PS: later we'll add more servers Thanks in advance, Best Regards Alessandro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
