Re: [Openstack] Can't connect to RabbitMQ
Am Mittwoch, 30. Oktober 2013, 22:35:34 schrieb Dheerendra: Good. Can you share the configuration file-nova.conf on compute ? It will help everybody. Of course: root@compute1:~# cat /etc/nova/nova.conf [DEFAULT] dhcpbridge_flagfile=/etc/nova/nova.conf dhcpbridge=/usr/bin/nova-dhcpbridge logdir=/var/log/nova state_path=/var/lib/nova lock_path=/var/lock/nova force_dhcp_release=True iscsi_helper=tgtadm libvirt_use_virtio_for_bridges=True connection_type=libvirt root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf verbose=True ec2_private_dns_show_ip=True api_paste_config=/etc/nova/api-paste.ini volumes_path=/var/lib/nova/volumes enabled_apis=ec2,osapi_compute,metadata my_ip=192.168.122.11 vncserver_listen=0.0.0.0 vncserver_proxyclient_address=192.168.122.11 glance_host=controller network_manager=nova.network.manager.FlatDHCPManager firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver network_size=254 allow_same_net_traffic=False multi_host=True send_arp_for_ha=True share_dhcp_address=True force_dhcp_release=True flat_network_bridge=br100 flat_interface=eth0 public_interface=eth0 rpc_backend = nova.rpc.impl_kombu rabbit_host = controller rabbit_port = 5672 [database] # The SQLAlchemy connection string used to connect to the database connection = mysql://nova:123@controller/nova Regards, Florian -Dheerendra On Wed, Oct 30, 2013 at 8:25 PM, Florian Lindner mailingli...@xgm.dewrote: Am Mittwoch, 30. Oktober 2013, 19:02:33 schrieb Dheerendra: also look at the logs /var/log/nova/nova-network.log on compute node. It should tell whether it is able to connect to rabbit on controller. I got now! The problem was a missing database entry on the compute node. Sorry guys for this stupid error, though the docs don't mention creating the entry. Thanks for your help! Florian -Dheerendra On Wed, Oct 30, 2013 at 6:41 PM, Dheerendra dheerendra.madhusudh...@gmail.com wrote: Hi Florian I observed this kind of issue when there is race condition. 1. RabbitMQ is NOT running on controller-node 2. You issue nova-network from compute-node. Now this step hangs. 3. Just see what is happening in nova-api.log on controller(tail -f /var/log/nova/nova-api.log) 4. Now you start the RabbitMQ on the controller node. 5. nova-network command compute returns the error. Also you will see 'timeout' error in nova-api.log. From your log I suspect this because initially RabitMQ was not up. Looks like you started it afterwards. Did you try to run the same 'nova-network' command for the second time ? i.e after restarting the rabbitMQ ? -Dheerendra On Wed, Oct 30, 2013 at 3:47 PM, Florian Lindner mailingli...@xgm.dewrote: Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua: You are using Neutron? Looks like the Neutron server is unable to perform the request actually Actually no, I don't think so, I think I use the nova-network. The documentation http://docs.openstack.org/havana/install-guide/install/apt/content/sa ys: Configuring Networking can be one of the most bewildering experiences you will encounter when working with OpenStack. To assist in this we have chosen the simplest production-ready configuration for this guide: the legacy networking in OpenStack Compute, with a flat network, that takes care of DHCP. This setup uses multi-host functionality: the networking is configured to be highly available by splitting networking functionality across multiple hosts. As a result, there is no single network controller that acts as a single point of failure. Because each compute node is configured for networking in this setup, no additional networking configuration is required on the controller. (Chapter 5: Enabling Networking) and I have installed the nova-network package. I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or anyone else wants to have a look! Yes, these 123 are the actual passwords, it's just from a VM on my local computer. ;-) Thanks a lot! Florian On October 29, 2013 at 12:00:18, Florian Lindner ( mailingli...@xgm.de) wrote: Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4= 10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the
Re: [Openstack] Can't connect to RabbitMQ
Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua: You are using Neutron? Looks like the Neutron server is unable to perform the request actually Actually no, I don't think so, I think I use the nova-network. The documentation http://docs.openstack.org/havana/install-guide/install/apt/content/ says: Configuring Networking can be one of the most bewildering experiences you will encounter when working with OpenStack. To assist in this we have chosen the simplest production-ready configuration for this guide: the legacy networking in OpenStack Compute, with a flat network, that takes care of DHCP. This setup uses multi-host functionality: the networking is configured to be highly available by splitting networking functionality across multiple hosts. As a result, there is no single network controller that acts as a single point of failure. Because each compute node is configured for networking in this setup, no additional networking configuration is required on the controller. (Chapter 5: Enabling Networking) and I have installed the nova-network package. I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or anyone else wants to have a look! Yes, these 123 are the actual passwords, it's just from a VM on my local computer. ;-) Thanks a lot! Florian On October 29, 2013 at 12:00:18, Florian Lindner (mailingli...@xgm.de) wrote: Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4=10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3) nova-api.log shows: http://pastebin.com/raw.php?i=13U8a4Na 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-] (1272) accepted ('192.168.122.10', 53116) 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:14:19.218 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238 1272 ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting on RPC response - topic: network, RPC method: create_networks info: unknown 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Traceback (most recent call last): 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py, line 119, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return req.get_response(self.application) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1296, in send 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application, catch_exc_info=False) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1260, in call_application 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py, line 539, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return self.app(env, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/routes/middleware.py, line 131, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack response = self.app(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29
Re: [Openstack] Can't connect to RabbitMQ
Hi Florian I observed this kind of issue when there is race condition. 1. RabbitMQ is NOT running on controller-node 2. You issue nova-network from compute-node. Now this step hangs. 3. Just see what is happening in nova-api.log on controller(tail -f /var/log/nova/nova-api.log) 4. Now you start the RabbitMQ on the controller node. 5. nova-network command compute returns the error. Also you will see 'timeout' error in nova-api.log. From your log I suspect this because initially RabitMQ was not up. Looks like you started it afterwards. Did you try to run the same 'nova-network' command for the second time ? i.e after restarting the rabbitMQ ? -Dheerendra On Wed, Oct 30, 2013 at 3:47 PM, Florian Lindner mailingli...@xgm.dewrote: Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua: You are using Neutron? Looks like the Neutron server is unable to perform the request actually Actually no, I don't think so, I think I use the nova-network. The documentation http://docs.openstack.org/havana/install-guide/install/apt/content/ says: Configuring Networking can be one of the most bewildering experiences you will encounter when working with OpenStack. To assist in this we have chosen the simplest production-ready configuration for this guide: the legacy networking in OpenStack Compute, with a flat network, that takes care of DHCP. This setup uses multi-host functionality: the networking is configured to be highly available by splitting networking functionality across multiple hosts. As a result, there is no single network controller that acts as a single point of failure. Because each compute node is configured for networking in this setup, no additional networking configuration is required on the controller. (Chapter 5: Enabling Networking) and I have installed the nova-network package. I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or anyone else wants to have a look! Yes, these 123 are the actual passwords, it's just from a VM on my local computer. ;-) Thanks a lot! Florian On October 29, 2013 at 12:00:18, Florian Lindner (mailingli...@xgm.de) wrote: Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4= 10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3) nova-api.log shows: http://pastebin.com/raw.php?i=13U8a4Na 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-] (1272) accepted ('192.168.122.10', 53116) 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:14:19.218 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238 1272 ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting on RPC response - topic: network, RPC method: create_networks info: unknown 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Traceback (most recent call last): 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py, line 119, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return req.get_response(self.application) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1296, in send 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application, catch_exc_info=False) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1260, in call_application 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py, line 539, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return self.app(env, start_response)
Re: [Openstack] Can't connect to RabbitMQ
also look at the logs /var/log/nova/nova-network.log on compute node. It should tell whether it is able to connect to rabbit on controller. -Dheerendra On Wed, Oct 30, 2013 at 6:41 PM, Dheerendra dheerendra.madhusudh...@gmail.com wrote: Hi Florian I observed this kind of issue when there is race condition. 1. RabbitMQ is NOT running on controller-node 2. You issue nova-network from compute-node. Now this step hangs. 3. Just see what is happening in nova-api.log on controller(tail -f /var/log/nova/nova-api.log) 4. Now you start the RabbitMQ on the controller node. 5. nova-network command compute returns the error. Also you will see 'timeout' error in nova-api.log. From your log I suspect this because initially RabitMQ was not up. Looks like you started it afterwards. Did you try to run the same 'nova-network' command for the second time ? i.e after restarting the rabbitMQ ? -Dheerendra On Wed, Oct 30, 2013 at 3:47 PM, Florian Lindner mailingli...@xgm.dewrote: Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua: You are using Neutron? Looks like the Neutron server is unable to perform the request actually Actually no, I don't think so, I think I use the nova-network. The documentation http://docs.openstack.org/havana/install-guide/install/apt/content/ says: Configuring Networking can be one of the most bewildering experiences you will encounter when working with OpenStack. To assist in this we have chosen the simplest production-ready configuration for this guide: the legacy networking in OpenStack Compute, with a flat network, that takes care of DHCP. This setup uses multi-host functionality: the networking is configured to be highly available by splitting networking functionality across multiple hosts. As a result, there is no single network controller that acts as a single point of failure. Because each compute node is configured for networking in this setup, no additional networking configuration is required on the controller. (Chapter 5: Enabling Networking) and I have installed the nova-network package. I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or anyone else wants to have a look! Yes, these 123 are the actual passwords, it's just from a VM on my local computer. ;-) Thanks a lot! Florian On October 29, 2013 at 12:00:18, Florian Lindner (mailingli...@xgm.de) wrote: Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4= 10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3) nova-api.log shows: http://pastebin.com/raw.php?i=13U8a4Na 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-] (1272) accepted ('192.168.122.10', 53116) 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:14:19.218 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238 1272 ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting on RPC response - topic: network, RPC method: create_networks info: unknown 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Traceback (most recent call last): 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py, line 119, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return req.get_response(self.application) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1296, in send 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application, catch_exc_info=False) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1260, in call_application 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response)
Re: [Openstack] Can't connect to RabbitMQ
Am Mittwoch, 30. Oktober 2013, 19:02:33 schrieb Dheerendra: also look at the logs /var/log/nova/nova-network.log on compute node. It should tell whether it is able to connect to rabbit on controller. I got now! The problem was a missing database entry on the compute node. Sorry guys for this stupid error, though the docs don't mention creating the entry. Thanks for your help! Florian -Dheerendra On Wed, Oct 30, 2013 at 6:41 PM, Dheerendra dheerendra.madhusudh...@gmail.com wrote: Hi Florian I observed this kind of issue when there is race condition. 1. RabbitMQ is NOT running on controller-node 2. You issue nova-network from compute-node. Now this step hangs. 3. Just see what is happening in nova-api.log on controller(tail -f /var/log/nova/nova-api.log) 4. Now you start the RabbitMQ on the controller node. 5. nova-network command compute returns the error. Also you will see 'timeout' error in nova-api.log. From your log I suspect this because initially RabitMQ was not up. Looks like you started it afterwards. Did you try to run the same 'nova-network' command for the second time ? i.e after restarting the rabbitMQ ? -Dheerendra On Wed, Oct 30, 2013 at 3:47 PM, Florian Lindner mailingli...@xgm.dewrote: Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua: You are using Neutron? Looks like the Neutron server is unable to perform the request actually Actually no, I don't think so, I think I use the nova-network. The documentation http://docs.openstack.org/havana/install-guide/install/apt/content/ says: Configuring Networking can be one of the most bewildering experiences you will encounter when working with OpenStack. To assist in this we have chosen the simplest production-ready configuration for this guide: the legacy networking in OpenStack Compute, with a flat network, that takes care of DHCP. This setup uses multi-host functionality: the networking is configured to be highly available by splitting networking functionality across multiple hosts. As a result, there is no single network controller that acts as a single point of failure. Because each compute node is configured for networking in this setup, no additional networking configuration is required on the controller. (Chapter 5: Enabling Networking) and I have installed the nova-network package. I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or anyone else wants to have a look! Yes, these 123 are the actual passwords, it's just from a VM on my local computer. ;-) Thanks a lot! Florian On October 29, 2013 at 12:00:18, Florian Lindner (mailingli...@xgm.de) wrote: Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4= 10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3) nova-api.log shows: http://pastebin.com/raw.php?i=13U8a4Na 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-] (1272) accepted ('192.168.122.10', 53116) 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:14:19.218 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238 1272 ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting on RPC response - topic: network, RPC method: create_networks info: unknown 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Traceback (most recent call last): 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py, line 119, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return req.get_response(self.application) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1296, in send 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application, catch_exc_info=False) 2013-10-29 19:15:19.238 1272 TRACE
Re: [Openstack] Can't connect to RabbitMQ
Good. Can you share the configuration file-nova.conf on compute ? It will help everybody. -Dheerendra On Wed, Oct 30, 2013 at 8:25 PM, Florian Lindner mailingli...@xgm.dewrote: Am Mittwoch, 30. Oktober 2013, 19:02:33 schrieb Dheerendra: also look at the logs /var/log/nova/nova-network.log on compute node. It should tell whether it is able to connect to rabbit on controller. I got now! The problem was a missing database entry on the compute node. Sorry guys for this stupid error, though the docs don't mention creating the entry. Thanks for your help! Florian -Dheerendra On Wed, Oct 30, 2013 at 6:41 PM, Dheerendra dheerendra.madhusudh...@gmail.com wrote: Hi Florian I observed this kind of issue when there is race condition. 1. RabbitMQ is NOT running on controller-node 2. You issue nova-network from compute-node. Now this step hangs. 3. Just see what is happening in nova-api.log on controller(tail -f /var/log/nova/nova-api.log) 4. Now you start the RabbitMQ on the controller node. 5. nova-network command compute returns the error. Also you will see 'timeout' error in nova-api.log. From your log I suspect this because initially RabitMQ was not up. Looks like you started it afterwards. Did you try to run the same 'nova-network' command for the second time ? i.e after restarting the rabbitMQ ? -Dheerendra On Wed, Oct 30, 2013 at 3:47 PM, Florian Lindner mailingli...@xgm.dewrote: Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua: You are using Neutron? Looks like the Neutron server is unable to perform the request actually Actually no, I don't think so, I think I use the nova-network. The documentation http://docs.openstack.org/havana/install-guide/install/apt/content/says: Configuring Networking can be one of the most bewildering experiences you will encounter when working with OpenStack. To assist in this we have chosen the simplest production-ready configuration for this guide: the legacy networking in OpenStack Compute, with a flat network, that takes care of DHCP. This setup uses multi-host functionality: the networking is configured to be highly available by splitting networking functionality across multiple hosts. As a result, there is no single network controller that acts as a single point of failure. Because each compute node is configured for networking in this setup, no additional networking configuration is required on the controller. (Chapter 5: Enabling Networking) and I have installed the nova-network package. I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or anyone else wants to have a look! Yes, these 123 are the actual passwords, it's just from a VM on my local computer. ;-) Thanks a lot! Florian On October 29, 2013 at 12:00:18, Florian Lindner ( mailingli...@xgm.de) wrote: Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4= 10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3) nova-api.log shows: http://pastebin.com/raw.php?i=13U8a4Na 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-] (1272) accepted ('192.168.122.10', 53116) 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:14:19.218 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238 1272 ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting on RPC response - topic: network, RPC method: create_networks info: unknown 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Traceback (most recent call last): 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py, line 119, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
Re: [Openstack] Can't connect to RabbitMQ
Interesting, never know about that thanks! On Oct 28, 2013, at 13:44, Craig E. Ward cw...@isi.edu wrote: On the RabbitMQ server, are vhost and user id setup correctly? % rabbitmqctl list_users % rabbitmqctl list_vhosts The user and vhost should match what is in the OpenStack configurations. On 10/28/13 1:14 PM, Florian Lindner wrote: Am Montag, 28. Oktober 2013, 12:04:02 schrieb Razique Mahroua: hey is «controller» a hostame all ur servers can resolve? Yes, set in /etc/hosts. Regards, Florian - Razique On Oct 28, 2013, at 11:40, Florian Lindner mailingli...@xgm.de wrote: Hello, I've installed OpenStack on a Ubuntu 12.04 VM according to the install guide. It worked fine until I created the compute node on a second VM and tried to set up nova networking: http://docs.openstack.org/havana/install-guide/install/apt/content/nova-ne twork.html # nova network-create vmnet --fixed-range-v4=10.0.0.0/24 --bridge- interface=br100 --multi-host=T just does nothing until canceled. I tried on the controller and compute1 node. nova-api.log shows: ERROR nova.openstack.common.rpc.common AMQP server on controller:5672 is unreachable: Socket closed. Trying again in 1 seconds. INFO nova.openstack.common.rpc.common Reconnecting to AMQP server on controller:5672 controller:~# netstat -lpn | grep beam tcp0 0 0.0.0.0:36595 0.0.0.0:* LISTEN 1045/beam tcp6 0 0 :::5672 :::*LISTEN 1045/beam It listens on this port, but IPv6. IMO on linux listing on an IPv6 port also means listening on the same v4 port. What's wrong with the installation? Thanks a lot! Florian ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack -- Craig E. Ward University of Southern California Information Sciences Institute cw...@isi.edu ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack signature.asc Description: Message signed with OpenPGP using GPGMail ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Can't connect to RabbitMQ
Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4=10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3) nova-api.log shows: http://pastebin.com/raw.php?i=13U8a4Na 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-] (1272) accepted ('192.168.122.10', 53116) 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:14:19.218 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238 1272 ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting on RPC response - topic: network, RPC method: create_networks info: unknown 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Traceback (most recent call last): 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py, line 119, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return req.get_response(self.application) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1296, in send 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application, catch_exc_info=False) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1260, in call_application 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py, line 539, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return self.app(env, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/routes/middleware.py, line 131, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack response = self.app(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 130, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 195, in call_func 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return self.func(req, *args, **kwargs) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py, line 917, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack content_type, body, accept) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py, line 976, in _process_stack 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py, line 1057, in dispatch 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return method(req=request, **action_args) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-
Re: [Openstack] Can't connect to RabbitMQ
You are using Neutron? Looks like the Neutron server is unable to perform the request actually On October 29, 2013 at 12:00:18, Florian Lindner (mailingli...@xgm.de) wrote: Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward: On the RabbitMQ server, are vhost and user id setup correctly? Ok, part of the problem seemed to be that the documentation assumed I set the password guest for the rabbit guest account. Did that now and the problem changed: root@controller:~# nova network-create vmnet --fixed-range-v4=10.0.0.0/24 -- bridge-interface=br100 --multi-host=T ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3) nova-api.log shows: http://pastebin.com/raw.php?i=13U8a4Na 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-] (1272) accepted ('192.168.122.10', 53116) 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:14:19.218 1272 INFO nova.openstack.common.rpc.common [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238 1272 ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting on RPC response - topic: network, RPC method: create_networks info: unknown 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Traceback (most recent call last): 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py, line 119, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return req.get_response(self.application) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1296, in send 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application, catch_exc_info=False) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/request.py, line 1260, in call_application 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py, line 539, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return self.app(env, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/routes/middleware.py, line 131, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack response = self.app(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 144, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return resp(environ, start_response) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 130, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/webob/dec.py, line 195, in call_func 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return self.func(req, *args, **kwargs) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py, line 917, in __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack content_type, body, accept) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py, line 976, in _process_stack 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args) 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
Re: [Openstack] Can't connect to RabbitMQ
Am Montag, 28. Oktober 2013, 12:04:02 schrieb Razique Mahroua: hey is «controller» a hostame all ur servers can resolve? Yes, set in /etc/hosts. Regards, Florian - Razique On Oct 28, 2013, at 11:40, Florian Lindner mailingli...@xgm.de wrote: Hello, I've installed OpenStack on a Ubuntu 12.04 VM according to the install guide. It worked fine until I created the compute node on a second VM and tried to set up nova networking: http://docs.openstack.org/havana/install-guide/install/apt/content/nova-ne twork.html # nova network-create vmnet --fixed-range-v4=10.0.0.0/24 --bridge- interface=br100 --multi-host=T just does nothing until canceled. I tried on the controller and compute1 node. nova-api.log shows: ERROR nova.openstack.common.rpc.common AMQP server on controller:5672 is unreachable: Socket closed. Trying again in 1 seconds. INFO nova.openstack.common.rpc.common Reconnecting to AMQP server on controller:5672 controller:~# netstat -lpn | grep beam tcp0 0 0.0.0.0:36595 0.0.0.0:* LISTEN 1045/beam tcp6 0 0 :::5672 :::*LISTEN 1045/beam It listens on this port, but IPv6. IMO on linux listing on an IPv6 port also means listening on the same v4 port. What's wrong with the installation? Thanks a lot! Florian ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Can't connect to RabbitMQ
On the RabbitMQ server, are vhost and user id setup correctly? % rabbitmqctl list_users % rabbitmqctl list_vhosts The user and vhost should match what is in the OpenStack configurations. On 10/28/13 1:14 PM, Florian Lindner wrote: Am Montag, 28. Oktober 2013, 12:04:02 schrieb Razique Mahroua: hey is «controller» a hostame all ur servers can resolve? Yes, set in /etc/hosts. Regards, Florian - Razique On Oct 28, 2013, at 11:40, Florian Lindner mailingli...@xgm.de wrote: Hello, I've installed OpenStack on a Ubuntu 12.04 VM according to the install guide. It worked fine until I created the compute node on a second VM and tried to set up nova networking: http://docs.openstack.org/havana/install-guide/install/apt/content/nova-ne twork.html # nova network-create vmnet --fixed-range-v4=10.0.0.0/24 --bridge- interface=br100 --multi-host=T just does nothing until canceled. I tried on the controller and compute1 node. nova-api.log shows: ERROR nova.openstack.common.rpc.common AMQP server on controller:5672 is unreachable: Socket closed. Trying again in 1 seconds. INFO nova.openstack.common.rpc.common Reconnecting to AMQP server on controller:5672 controller:~# netstat -lpn | grep beam tcp0 0 0.0.0.0:36595 0.0.0.0:* LISTEN 1045/beam tcp6 0 0 :::5672 :::*LISTEN 1045/beam It listens on this port, but IPv6. IMO on linux listing on an IPv6 port also means listening on the same v4 port. What's wrong with the installation? Thanks a lot! Florian ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack -- Craig E. Ward University of Southern California Information Sciences Institute cw...@isi.edu ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack