Re: [Openstack] PLEASE HELP multi_host not working
On 04/07/2012 15:53, Mark Russell wrote: There is likely a better way to fix this than what I did (drop and recreate nova db) but it was the simplest fix given that this was a brand new installation. There is almost certainly a better way to fix it by manually hacking nova db. Thank you Mark! Creating a new nova database called novaMulti and creating the pvt and floating networks from scratch has taken me leaps forward. Still struggling a bit with some of the routing, but at least I can now launce VMs on all of the compute nodes. Tx M ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] PLEASE HELP multi_host not working
Hi Marnus, I experienced the exact same symptom (no fixed IP given to instance). Is it possible that you previously deleted and recreated your network with networks that overlap or match the old network? In my case the problem was related to this bug: https://bugs.launchpad.net/nova/+bug/754900 In this bug, instances are getting fixed IPs but from a previously deleted pool. Another bug (marked as a dupe of the above) describes this situation a little better (no IP given to instance): https://bugs.launchpad.net/nova/+bug/812573 One way to confirm if deleting and recreating the network is the problem is to tail -f /var/log/nova/nova-network.log while starting and stopping an instance on that node. When I was seeing this problem, nothing would show up in nova-network.log during launch, but when I terminated the instance, nova-network.log would show an error saying essentially that network 2 could not be found. However we were on network #5 by then. So clearly something was off in nova db. There is likely a better way to fix this than what I did (drop and recreate nova db) but it was the simplest fix given that this was a brand new installation. There is almost certainly a better way to fix it by manually hacking nova db. Hope this helps, Mark On 07/03/2012 11:22 PM, Marnus van Niekerk wrote: I installed a minimal cloud image (cirros) and found that it had no network configures after booting. Even running udhcpc manually does not work - it never gets an answer from dhcp. I checked and dnsmasq is running on all of the the compute nodes: /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=10.10.11.135 --except-interface=lo --dhcp-range=10.10.11.130,static,120s --dhcp-lease-max=64 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro Why would the DHCP not be working? I am running out of ideas here. Have tried everything I can read from searching google and still multi_host does not come close to working. I am now even considering abandoning OpenStack and going back to Eucalyptus. I have even tried changing to FlatNetwork with flat_injected=True and then IPs are set in the VMs but the metadata service still fails (no route to host). I have checked and each host has the 169.254.269.254 addresses natted to itself. I have also deleted and recreated the network with the --multi_host=T flag. I have this on the controller: #--routing_source_ip=10.10.20.114 --ec2_dmz_host=10.10.20.114 --multi_host=True --enabled_apis=ec2,osapi_compute,osapi_volume,metadata --dmz_cidr=10.10.20.114/32 And this on the compute nodes: #--routing_source_ip=10.10.20.114 --ec2_dmz_host=10.10.20.114 --multi_host=True --enabled_apis=metadata --dmz_cidr=10.10.20.114/32 On 03/07/2012 10:03, Marnus van Niekerk wrote: On 03/07/2012 09:53, Marnus van Niekerk wrote: I can now see the bridge created and assigned an address on each compute node, but all of the VMs get stuck after the bootloader - they never boot any further. Sorry, they do actually boot after a while but without any networking.. cloud-init-nonet waiting 120 seconds for a network device. cloud-init-nonet gave up waiting for a network device. ci-info: lo: 1 127.0.0.1 255.0.0.0 . ci-info: eth0 : 1 . . fa:16:3e:39:5f:02 route_info failed Waiting for network configuration... Waiting up to 60 more seconds for network configuration... Booting system without full network configuration... ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Mark Russell Premium Service Architect | Canonical, Ltd. mark.russ...@canonical.com | GPG: 4096R/B3BBA7D1 www.ubuntu.com | www.canonical.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] PLEASE HELP multi_host not working
On 04/07/2012 21:43, Vishvananda Ishaya wrote: It looks like the ip for your hosts is in the 10./8 range which is probably messing up routing and snatting. You will need to use a smaller range when you create your vm network, say 10.75.0.0/16 http://10.75.0.0/16 and make sure you set fixed_range to the same value in nova.conf I already have it sub-netted to 10.10.11.128/26 and have --fixed_range=10.10.11.128/26 in nova.conf. For metadata, you may have to set metadata_host (it sets this automatically, but if you have multiple interfaces it could be getting the wrong value) to the ip of the compute node on each compute host (and run nova-api-metadata or nova-api on each compute host as well. On each host I see an iptables rule like this natting the 169.254.169.254 address to the br100 address of the host like this: -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.10.20.114:8775 But wget http://169.254.169.254/; on the controller node gives me the horizon index page! On the other compute nodes it gives connection refused. Should metadata_host be set to the public (eth0) ip of the host instead? Tx for the advice so far. M ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] PLEASE HELP multi_host not working
I installed a minimal cloud image (cirros) and found that it had no network configures after booting. Even running udhcpc manually does not work - it never gets an answer from dhcp. I checked and dnsmasq is running on all of the the compute nodes: /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=10.10.11.135 --except-interface=lo --dhcp-range=10.10.11.130,static,120s --dhcp-lease-max=64 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro Why would the DHCP not be working? I am running out of ideas here. Have tried everything I can read from searching google and still multi_host does not come close to working. I am now even considering abandoning OpenStack and going back to Eucalyptus. I have even tried changing to FlatNetwork with flat_injected=True and then IPs are set in the VMs but the metadata service still fails (no route to host). I have checked and each host has the 169.254.269.254 addresses natted to itself. I have also deleted and recreated the network with the --multi_host=T flag. I have this on the controller: #--routing_source_ip=10.10.20.114 --ec2_dmz_host=10.10.20.114 --multi_host=True --enabled_apis=ec2,osapi_compute,osapi_volume,metadata --dmz_cidr=10.10.20.114/32 And this on the compute nodes: #--routing_source_ip=10.10.20.114 --ec2_dmz_host=10.10.20.114 --multi_host=True --enabled_apis=metadata --dmz_cidr=10.10.20.114/32 On 03/07/2012 10:03, Marnus van Niekerk wrote: On 03/07/2012 09:53, Marnus van Niekerk wrote: I can now see the bridge created and assigned an address on each compute node, but all of the VMs get stuck after the bootloader - they never boot any further. Sorry, they do actually boot after a while but without any networking.. cloud-init-nonet waiting 120 seconds for a network device. cloud-init-nonet gave up waiting for a network device. ci-info: lo: 1 127.0.0.1 255.0.0.0 . ci-info: eth0 : 1 . . fa:16:3e:39:5f:02 route_info failed Waiting for network configuration... Waiting up to 60 more seconds for network configuration... Booting system without full network configuration... ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
