Re: [Openstack] volumeId is not in proper format
Well that's embarrassing. Thanks. On Wed, Jun 5, 2013 at 11:53 AM, John Griffith john.griff...@solidfire.com wrote: Not exactly sure what it's tripping up on when using the volume name there, but I would suggest using the volume-id rather than the name. On Wed, Jun 5, 2013 at 9:46 AM, Greg Chavez greg.cha...@gmail.com wrote: So I'm happily spinning virts on my new Grizzly installation (Ubuntu 13.04, per tenant routers, vm config network via GRE, and Ceph block storage on the compute nodes). Suddenly, we have encountered the following problem attaching the following volume: root@kcon-gen-01i:~l# nova volume-attach maps-cs-vm-01u maps-cs-vm-01u_vol0 /dev/vdc ERROR: Bad volumeId format: volumeId is not in proper format (maps-cs-vm-01u_vol0) (HTTP 400) (Request-ID: req-1205e592-8e96-4874-a7a0-1f7117859f84) I've several other volumes with similar names and have had no trouble attaching them. Any idea what's wrong here? Thanks! -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Understanding the metadata service
Even after reading all the posts and IRC chatter regarding how to configure VM metadata services, I'm no closer to understanding how this should be set up in *my* environment. As it stands, my VMs are fully functional (available from external network via floaters) but cannot connect to 169.254.169.254. I am running Grizzly on Raring, with per-tenant routers, ip-namespaces/overlapping-ips, GRE tunneling, and a separate network node with three NICs for public, management, and vm config nets. Also: I have nova-api installed on the compute node. It is listening for 8775 on the controller's management network. The relavent configurations: controller nova.conf: -- service_quantum_metadata_proxy = True quantum_metadata_proxy_shared_secret = PvUafKreIImX3OePynlc metadata_host = 192.168.241.100 metadata_listen = 192.168.241.100 metadata_listen_port = 8775 net-node quantum.conf / dhcp_agent.ini / l3_agent.ini -- allow_overlapping_ips = True net-node metadata_agent.ini -- nova_metadata_ip = 192.168.241.100 nova_metadata_port = 8775 metadata_proxy_shared_secret = PvUafKreIImX3OePynlc compute nodes nova.conf -- service_quantum_metadata_proxy = True quantum_metadata_proxy_shared_secret = PvUafKreIImX3OePynlc What am I missing here? Thanks in advance. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A Grizzly GRE failure [SOLVED]
I've had a terrible time getting the community to help me with this
problem. So special thanks to Darragh O'Reilly and rkeene on
#openstack who was mean and a bit of a wisenheimer (I'd use different
words elsewhere), but at least he talked to me and got me to think
twice about my GRE setup.
But enough of that, problem solved and a bug report has been
submitted: https://bugs.launchpad.net/quantum/+bug/1179223.. I added
an s to the front of persists in the subject, but whatever. I
always leave one thing in the hotel room, and I always leave one
embarrassing typo.
Here's the part explaining how it was fixed:
SOLUTION:
mysql delete from ovs_tunnel_endpoints where id = 1;
Query OK, 1 row affected (0.00 sec)
mysql select * from ovs_tunnel_endpoints;
+-++
| ip_address | id |
+-++
| 192.168.239.110 | 3 |
| 192.168.239.114 | 4 |
| 192.168.239.115 | 5 |
| 192.168.239.99 | 2 |
+-++
4 rows in set (0.00 sec)
* After doing that, I simply restarted the quantum ovs agents on the
network and compute nodes. The old GRE tunnel is not re-created.
Thereafter, VM network traffic to and from the external network
proceeds without incident.
* Should these tables be cleaned up as well, I wonder:
mysql select * from ovs_network_bindings;
+--+--+--+-+
| network_id | network_type | physical_network | segmentation_id |
+--+--+--+-+
| 4e8aacca-8b38-40ac-a628-18cac3168fe6 | gre | NULL | 2 |
| af224f3f-8de6-4e0d-b043-6bcd5cb014c5 | gre | NULL | 1 |
+--+--+--+-+
2 rows in set (0.00 sec)
mysql select * from ovs_tunnel_allocations where allocated != 0;
+---+---+
| tunnel_id | allocated |
+---+---+
| 1 | 1 |
| 2 | 1 |
+---+---+
2 rows in set (0.00 sec)
Cheers, and happy openstacking. Even you, rkeene!
--Greg Chavez
On Sat, May 11, 2013 at 2:28 PM, Greg Chavez greg.cha...@gmail.com wrote:
So to be clear:
* I have a three nics on my network node. The VM traffic goes out the
1st nic on 192.168.239.99/24 to the other compute nodes, while
management traffic goes out the 2nd nic on 192.168.241.99. The 3rd nic
is external and has no IP.
* I have four GRE endpoints on the VM network, one at the network node
(192.168.239.99) and three on compute nodes
(192.168.239.{110,114,115}), all with IDs 2-5.
* I have a fifth GRE endpoint with id 1 to 192.168.241.99, the network
node's management interface. This was the first tunnel created when I
deployed the network node because that is how I set the remote_ip in
the ovs plugin ini. I corrected the setting later, but the
192.168.241.99 endpoint persists and, as your response implies, *this
extraneous endpoint is the cause of my troubles*.
My next question then is what is happening? My guess:
* I ping a guest from the external network using its floater (10.21.166.4).
* It gets NAT'd at the tenant router on the network node to
192.168.252.3, at which point an arp request is sent over the unified
GRE broadcast domain.
* On a compute node, the arp request is received by the VM, which then
sends a reply to the tenant router's MAC (which I verified with
tcpdumps).
* There are four endpoints for the packet to go down:
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port gre-1
Interface gre-1
type: gre
options: {in_key=flow, out_key=flow,
remote_ip=192.168.241.99}
Port gre-4
Interface gre-4
type: gre
options: {in_key=flow, out_key=flow,
remote_ip=192.168.239.114}
Port gre-3
Interface gre-3
type: gre
options: {in_key=flow, out_key=flow,
remote_ip=192.168.239.110}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port gre-2
Interface gre-2
type: gre
options: {in_key=flow, out_key=flow,
remote_ip=192.168.239.99}
Here's where I get confused. Does it know that gre-1 is a different
broadcast domain than the others, or does is see all endpoints as the
same domain?
What happens here? Is this the cause of my network timeouts on
external connections to the VMs? Does this also explain the sporadic
nature of the timeouts, why they aren't consistent in frequency or
duration?
Finally, what happens when I remove the oddball endpoint from the DB?
Sounds risky!
Thanks for your help
--Greg Chavez
On Fri, May 10, 2013 at 7:17 PM, Darragh O'Reilly
dara2002-openst...@yahoo.com wrote:
I'm not sure how to rectify that. You may have to delete the bad row from
the DB
Re: [Openstack] A Grizzly GRE failure
So to be clear:
* I have a three nics on my network node. The VM traffic goes out the
1st nic on 192.168.239.99/24 to the other compute nodes, while
management traffic goes out the 2nd nic on 192.168.241.99. The 3rd nic
is external and has no IP.
* I have four GRE endpoints on the VM network, one at the network node
(192.168.239.99) and three on compute nodes
(192.168.239.{110,114,115}), all with IDs 2-5.
* I have a fifth GRE endpoint with id 1 to 192.168.241.99, the network
node's management interface. This was the first tunnel created when I
deployed the network node because that is how I set the remote_ip in
the ovs plugin ini. I corrected the setting later, but the
192.168.241.99 endpoint persists and, as your response implies, *this
extraneous endpoint is the cause of my troubles*.
My next question then is what is happening? My guess:
* I ping a guest from the external network using its floater (10.21.166.4).
* It gets NAT'd at the tenant router on the network node to
192.168.252.3, at which point an arp request is sent over the unified
GRE broadcast domain.
* On a compute node, the arp request is received by the VM, which then
sends a reply to the tenant router's MAC (which I verified with
tcpdumps).
* There are four endpoints for the packet to go down:
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port gre-1
Interface gre-1
type: gre
options: {in_key=flow, out_key=flow, remote_ip=192.168.241.99}
Port gre-4
Interface gre-4
type: gre
options: {in_key=flow, out_key=flow,
remote_ip=192.168.239.114}
Port gre-3
Interface gre-3
type: gre
options: {in_key=flow, out_key=flow,
remote_ip=192.168.239.110}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port gre-2
Interface gre-2
type: gre
options: {in_key=flow, out_key=flow, remote_ip=192.168.239.99}
Here's where I get confused. Does it know that gre-1 is a different
broadcast domain than the others, or does is see all endpoints as the
same domain?
What happens here? Is this the cause of my network timeouts on
external connections to the VMs? Does this also explain the sporadic
nature of the timeouts, why they aren't consistent in frequency or
duration?
Finally, what happens when I remove the oddball endpoint from the DB?
Sounds risky!
Thanks for your help
--Greg Chavez
On Fri, May 10, 2013 at 7:17 PM, Darragh O'Reilly
dara2002-openst...@yahoo.com wrote:
I'm not sure how to rectify that. You may have to delete the bad row from the
DB and restart the agents:
mysql use quantum;
mysql select * from ovs_tunnel_endpoints;
...
On Fri, May 10, 2013 at 6:43 PM, Greg Chavez greg.cha...@gmail.com wrote:
I'm refactoring my question once again (see A Grizzly arping
failure and Failure to arp by quantum router).
Quickly, the problem is in a multi-node Grizzly+Raring setup with a
separate network node and a dedicated VLAN for VM traffic. External
connections time out within a minute and dont' resume until traffic is
initiated from the VM.
I got some rather annoying and hostile assistance just now on IRC and
while it didn't result in a fix, it got me to realize that the problem
is possibly with my GRE setup.
I made a mistake when I originally set this up, assigning the mgmt
interface of the network node (192.168.241.99) as its GRE remote_ip
instead if the vm_config network interface (192.168.239.99). I
realized my mistake and reconfigured the OVS plugin on the network
node and moved one. But now, taking a look at my OVS bridges on the
network node, I see that the old remote IP is still there!
Bridge br-tun
snip
Port gre-1
Interface gre-1
type: gre
options: {in_key=flow, out_key=flow,
remote_ip=192.168.241.99}
snip
This is also on all the compute nodes.
( Full ovs-vsctl show output here: http://pastebin.com/xbre1fNV)
What's more, I have this error every time I restart OVS:
2013-05-10 18:21:24ERROR [quantum.agent.linux.ovs_lib] Unable to
execute ['ovs-vsctl', '--timeout=2', 'add-port', 'br-tun', 'gre-5'].
Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
'ovs-vsctl', '--timeout=2', 'add-port', 'br-tun', 'gre-5']
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: cannot create a port named gre-5 because a port
named gre-5 already exists on bridge br-tun\n'
Could that be because grep-1 is vestigial and possibly fouling up the
works by creating two possible paths for VM traffic?
Is it as simple as removing it with ovs-vsctl or is something else required?
Or is this actually needed for some reason? Argh... help
Re: [Openstack] Grizzly fails on ubuntu server 13.04
No, but maybe that's not a requirement for him. Nikhil, I have Grizzly running on 13.04, but I'm also having network problems (see subject A Grizzly arping failure). I'm investigating these issues today, hopefully with plenty of help from the community. Maybe the fix I need is similar to the fix that you need. We shall see. --Greg Chavez On Fri, May 10, 2013 at 12:21 PM, Pranav pps.pra...@gmail.com wrote: but NAT wont make the VM accessible from outside world ... will it? Regards, Pranav On Fri, May 10, 2013 at 5:16 PM, Nikhil Mittal mittalnik...@gmail.com wrote: Nope, assigning a floating IP is not required. The VM can be accessed with whatever IP that is assigned as the NAT'ng will take care of that. On Fri, May 10, 2013 at 4:50 PM, Pranav pps.pra...@gmail.com wrote: you need to expose the VM's internal network to an external network by allocating it floating IP address. Regards, Pranav On Thu, May 9, 2013 at 10:47 PM, Nikhil Mittal mittalnik...@gmail.com wrote: I am seeing this email after a long time. Quite recently I tried installing Grizzly on Ubuntu 13.04 on x86 64 machine. It installed fine except that i am facing some network configuration issues after the VMs start running. On Fri, Apr 26, 2013 at 12:05 AM, skible.openst...@gmail.com skible.openst...@gmail.com wrote: Hi, So the ubuntu 13.04 has just been released. Everything is fine except for the virtualization using KVM. After installing nova-compute-kvm, nova-compute does not start and this is what i found in my log file: Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory which is obvious because there is no more KVM in ubuntu 13.04, instead a qemu-kvm package. Anyone got Grizzly on ubuntu 13.04 ? regards, Bilel ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A Grizzly arping failure
More information: Even thought it appears that the dhcp namespace has the ip/MAC mappings in arp, is still doesn't respond to arps: # ip netns exec qdhcp-af224f3f-8de6-4e0d-b043-6bcd5cb014c5 arp Address HWtype HWaddress Flags MaskIface 192.168.252.3ether fa:16:3e:8a:02:78 C tap14301d16-5d 192.168.252.5ether fa:16:3e:ca:64:56 C tap14301d16-5d # ip netns exec qrouter-5ade8a36-4bcf-443a-bf96-89168ec34a13 tcpdump -ni qr-9ae4cc80-c7 host 192.168.252.5 13:30:36.243354 ARP, Request who-has 192.168.252.5 tell 192.168.252.1, length 28 13:30:36.243649 ARP, Request who-has 192.168.252.5 tell 192.168.252.1, length 28 13:30:37.241445 ARP, Request who-has 192.168.252.5 tell 192.168.252.1, length 28 13:30:37.241475 ARP, Request who-has 192.168.252.5 tell 192.168.252.1, length 28 13:30:38.241260 ARP, Request who-has 192.168.252.5 tell 192.168.252.1, length 28 Very annoying! On Fri, May 10, 2013 at 12:36 PM, Greg Chavez greg.cha...@gmail.com wrote: I sent a message about this 3 days ago, but I didn't get a response. So if at first you don't succeed... The symptom is that my VMs only survive on the external network for about a minute. The behavior is not consistent, however. I've observed the following different types of failures: * ssh session to VM is idle for a minute, then the connection dies. It can only be re-established by initiating network activity from the VM console. * ssh session to VM is idle for many minutes, then I start typing and after a few seconds of latency, the connection resumes. * initiating a new ssh or icmp to a VM fails 95% of the time. It seems that the VMs occasionally request new DHCP leases, at which time the VM is briefly available on the external network. I've troubleshot this to the dnmasq instance in the tenant's network namespace. When a VM is unavailable I can trace traffic all the way to the qrouter. Tenant IP: 192.168.252.6/23 Floating IP: 10.21.166.5/22 SSHing IP: 10.21.164.10/22 # ip netns exec qrouter-5ade8a36-4bcf-443a-bf96-89168ec34a13 tcpdump -ni qg-498f9986-74 host 10.21.166.5 12:29:00.279454 IP 10.21.164.10 10.21.166.5: ICMP echo request, id 17219, seq 1, length 64 12:29:01.278800 IP 10.21.164.10 10.21.166.5: ICMP echo request, id 17219, seq 2, length 64 12:29:02.278793 IP 10.21.164.10 10.21.166.5: ICMP echo request, id 17219, seq 3, length 64 12:29:03.277277 IP 10.21.166.5 10.21.164.10: ICMP host 10.21.166.5 unreachable, length 92 12:29:03.277305 IP 10.21.166.5 10.21.164.10: ICMP host 10.21.166.5 unreachable, length 92 12:29:03.277315 IP 10.21.166.5 10.21.164.10: ICMP host 10.21.166.5 unreachable, length 92 12:29:03.277758 IP 10.21.164.10 10.21.166.5: ICMP echo request, id 17219, seq 4, length 64 12:29:04.277755 IP 10.21.164.10 10.21.166.5: ICMP echo request, id 17219, seq 5, length 64 12:29:05.278769 ARP, Request who-has 10.21.166.5 tell 10.21.164.10, length 46 12:29:05.278781 ARP, Reply 10.21.166.5 is-at fa:16:3e:4d:e6:93, length 28 12:29:06.277277 IP 10.21.166.5 10.21.164.10: ICMP host 10.21.166.5 unreachable, length 92 12:29:06.277301 IP 10.21.166.5 10.21.164.10: ICMP host 10.21.166.5 unreachable, length 92 The arp reply points to the MAC for interface qg-498f9986-74. So that's good. But on the other side of the router, arp fails: # ip netns exec qrouter-5ade8a36-4bcf-443a-bf96-89168ec34a13 tcpdump -ni qr-9ae4cc80-c7 host 192.168.252.6 12:31:34.798835 ARP, Request who-has 192.168.252.6 tell 192.168.252.1, length 28 12:31:34.799136 ARP, Request who-has 192.168.252.6 tell 192.168.252.1, length 28 12:31:35.797250 ARP, Request who-has 192.168.252.6 tell 192.168.252.1, length 28 12:31:35.797310 ARP, Request who-has 192.168.252.6 tell 192.168.252.1, length 28 12:31:36.797262 ARP, Request who-has 192.168.252.6 tell 192.168.252.1, length 28 12:31:36.797296 ARP, Request who-has 192.168.252.6 tell 192.168.252.1, length 28 12:31:37.797858 ARP, Request who-has 192.168.252.6 tell 192.168.252.1, length 28 No reply. Nothing. But look what happens when I ping from my VM console: # ip netns exec qrouter-5ade8a36-4bcf-443a-bf96-89168ec34a13 tcpdump -ni qr-9ae4cc80-c7 host 192.168.252.6 12:33:44.190214 ARP, Request who-has 192.168.252.1 tell 192.168.252.6, length 28 12:33:44.190245 ARP, Reply 192.168.252.1 is-at fa:16:3e:f8:a4:54, length 28 12:33:44.190981 IP 192.168.252.6 10.21.164.1: ICMP echo request, id 46648, seq 1, length 64 12:33:44.192025 IP 10.21.164.1 192.168.252.6: ICMP echo reply, id 46648, seq 1, length 64 12:33:45.188743 IP 192.168.252.6 10.21.164.1: ICMP echo request, id 46648, seq 2, length 64 12:33:45.189242 IP 10.21.164.1 192.168.252.6: ICMP echo reply, id 46648, seq 2, length 64 Works. Does anybody have any idea what's going on here? My next move is to look into how dnmsasq is configured. I assume that's the source of the problem, but I feel a little in over my head
Re: [Openstack] A Grizzly arping failure
Well, the subject is A Grizzly apring failure. So I'm using Grizzly :) Sorry, bad joke. I have a multi-node setup on Raring, with a 3-nic network node (public, management, vm config nets) using OVS and GRE encapsulation. It looks like this: http://chavezy.files.wordpress.com/2013/03/ostack-log-net_iscsi.png It's clear to me that the arp fails on the tenant-network-side of the quantum router. If you look at my original message, you see that the arp requests for the tenant IP go unanswered by dnsmasq (I assume?). So whatever's wrong, it *seems* like it's in the tenant namespace on the network node. Now is an interesting time, though, because I can ping all the VMs I have up from the external network. But notice the latency of the first ping packet: PING 10.21.166.2 (10.21.166.2) 56(84) bytes of data. 64 bytes from 10.21.166.2: icmp_seq=1 ttl=63 time=1001 ms 64 bytes from 10.21.166.2: icmp_seq=2 ttl=63 time=1.95 ms 64 bytes from 10.21.166.2: icmp_seq=3 ttl=63 time=0.646 ms 64 bytes from 10.21.166.2: icmp_seq=4 ttl=63 time=0.901 ms That's a full second! My developers won't stand for that. My suspicion is that something is amiss with the tenant network's dnsmasq instance. But I don't know for sure. PLEASE can somebody help! Darragh, thank you for replying. On Fri, May 10, 2013 at 3:07 PM, Darragh O'Reilly dara2002-openst...@yahoo.com wrote: can you try using -e with tcpdump to see the ethernet headers - it may be arps from the router to ff:ff:ff:ff:ff:ff that are not getting across in that direction. You should continue tcpdumping on the devices along the path to the instance to see where the arp request (or reply) stops. You do not say which plugin you are using? Is it multinode? (if OVS plugin, GRE or VLANs?) Grizzly? -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A Grizzly arping failure
Arg, of course the arp replies come from the instance. See? I'm easily confused. Here's more, using tcpdump -e. So if I ping the floater (10.21.166.2 - 192.168.252.3) ) from the outside (10.21.164.10), I see this on the tenant router's external port: 16:16:30.771776 00:22:19:65:ae:42 fa:16:3e:4d:e6:93, ethertype IPv4 (0x0800), length 98: 10.21.164.10 10.21.166.2: ICMP echo request, id 18720, seq 1, length 64 16:16:31.772293 00:22:19:65:ae:42 fa:16:3e:4d:e6:93, ethertype IPv4 (0x0800), length 98: 10.21.164.10 10.21.166.2: ICMP echo request, id 18720, seq 2, length 64 16:16:32.773259 00:22:19:65:ae:42 fa:16:3e:4d:e6:93, ethertype IPv4 (0x0800), length 98: 10.21.164.10 10.21.166.2: ICMP echo request, id 18720, seq 3, length 64 16:16:33.769279 fa:16:3e:4d:e6:93 00:22:19:65:ae:42, ethertype IPv4 (0x0800), length 126: 10.21.166.2 10.21.164.10: ICMP host 10.21.166.2 unreachable, length 92 16:16:33.769307 fa:16:3e:4d:e6:93 00:22:19:65:ae:42, ethertype IPv4 (0x0800), length 126: 10.21.166.2 10.21.164.10: ICMP host 10.21.166.2 unreachable, length 92 16:16:33.769317 fa:16:3e:4d:e6:93 00:22:19:65:ae:42, ethertype IPv4 (0x0800), length 126: 10.21.166.2 10.21.164.10: ICMP host 10.21.166.2 unreachable, length 92 I see this on the tenant router's internal port (unrequited arps): 16:16:30.771820 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:16:30.772124 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:16:31.769261 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:16:31.769297 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:16:32.769235 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:16:32.769269 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 I see this on the dmsasq's port (same): 16:20:31.524529 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:20:31.524710 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:20:32.521264 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:20:32.521289 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:20:33.521268 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:20:33.521308 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 And here's the instance's ovs interface: 16:35:41.227497 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:35:41.228075 fa:16:3e:8a:02:78 fa:16:3e:f8:a4:54, ethertype ARP (0x0806), length 42: Reply 192.168.252.3 is-at fa:16:3e:8a:02:78, length 28 16:35:42.225296 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:35:42.225620 fa:16:3e:8a:02:78 fa:16:3e:f8:a4:54, ethertype ARP (0x0806), length 42: Reply 192.168.252.3 is-at fa:16:3e:8a:02:78, length 28 16:35:43.225268 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:35:43.225558 fa:16:3e:8a:02:78 fa:16:3e:f8:a4:54, ethertype ARP (0x0806), length 42: Reply 192.168.252.3 is-at fa:16:3e:8a:02:78, length 28 16:35:44.225707 fa:16:3e:f8:a4:54 ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.252.3 tell 192.168.252.1, length 28 16:35:44.226013 fa:16:3e:8a:02:78 fa:16:3e:f8:a4:54, ethertype ARP (0x0806), length 42: Reply 192.168.252.3 is-at fa:16:3e:8a:02:78, length 28 That MAC, is the correct MAC for 192.168.252.1. So the instance gets the arp requests and replies to them, but they don't get back to the tenant router. Something is squelching the arp reply between the compute node and the network node. GRE issue? Ugh... On Fri, May 10, 2013 at 4:28 PM, Darragh O'Reilly dara2002-openst...@yahoo.com wrote: In my setup I see the arp replies being answered by the instance - not dnsmasq. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net
[Openstack] A Grizzly GRE failure
I'm refactoring my question once again (see A Grizzly arping
failure and Failure to arp by quantum router).
Quickly, the problem is in a multi-node Grizzly+Raring setup with a
separate network node and a dedicated VLAN for VM traffic. External
connections time out within a minute and dont' resume until traffic is
initiated from the VM.
I got some rather annoying and hostile assistance just now on IRC and
while it didn't result in a fix, it got me to realize that the problem
is possibly with my GRE setup.
I made a mistake when I originally set this up, assigning the mgmt
interface of the network node (192.168.241.99) as its GRE remote_ip
instead if the vm_config network interface (192.168.239.99). I
realized my mistake and reconfigured the OVS plugin on the network
node and moved one. But now, taking a look at my OVS bridges on the
network node, I see that the old remote IP is still there!
Bridge br-tun
snip
Port gre-1
Interface gre-1
type: gre
options: {in_key=flow, out_key=flow, remote_ip=192.168.241.99}
snip
This is also on all the compute nodes.
( Full ovs-vsctl show output here: http://pastebin.com/xbre1fNV)
What's more, I have this error every time I restart OVS:
2013-05-10 18:21:24ERROR [quantum.agent.linux.ovs_lib] Unable to
execute ['ovs-vsctl', '--timeout=2', 'add-port', 'br-tun', 'gre-5'].
Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
'ovs-vsctl', '--timeout=2', 'add-port', 'br-tun', 'gre-5']
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: cannot create a port named gre-5 because a port
named gre-5 already exists on bridge br-tun\n'
Could that be because grep-1 is vestigial and possibly fouling up the
works by creating two possible paths for VM traffic?
Is it as simple as removing it with ovs-vsctl or is something else required?
Or is this actually needed for some reason? Argh... help!
--
\*..+.-
--Greg Chavez
+//..;};
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] Failure to arp by quantum router
Problem: VMs launch, get local IPs from DHCP, get floating ips, and are available from the external network, but seem to time out after a minute of inactivity. After that time, the VM cannot be reached. I can sometimes network access back by pinging the external gateway from the VM console, but eventually times out as well. Background: I have Grizzly up and running on Ubuntu 13.04 with gre-tunneling and per-tenant routers. I have a network node with three interfaces, public, management, and vm config. The compute nodes and quantum node interact on the vm config network. Everything else runs from the controller node. Here's a diagram of my setup: http://chavezy.files.wordpress.com/2013/03/ostack-log-net_iscsi.png I can trace the packet to the external port of the tenant router on the network node. So the network node arps for the floating IP with its interface, no problem. But when I sniff the tenant network side of the router, I see unanswered arp requests for the VM's local IP. What's failing here? How would you troubleshoot this? Thanks. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Grizzly] Inbound VM traffic fails at compute node
A fellow name George clued me into my problem. I had my secgroup rules set for source 0.0.0.0/24 which is stupid. This is how it should look: root@kcon-cs-gen-01i:~# nova secgroup-list-rules default +-+---+-+---+--+ | IP Protocol | From Port | To Port | IP Range | Source Group | +-+---+-+---+--+ | icmp| -1| -1 | 0.0.0.0/0 | | | tcp | 22| 22 | 0.0.0.0/0 | | +-+---+-+---+--+ Thanks again, George. On Thu, May 2, 2013 at 6:27 PM, Greg Chavez greg.cha...@gmail.com wrote: I have Grizzly up and running on Ubuntu 13.04, following the excellent instructions by Msekni Bilel. I'm using gre tunneling and per-tenant routers. It looks something like this: http://chavezy.files.wordpress.com/2013/03/ostack-log-net_iscsi.png I was able to get a cirros m1.tiny VM launched easily. But although I've associated a floating IP and configured secgroup rules, I am unable to get any inbound traffic past the VM bridge. The internal network is 192.168.252.0/23. The floating IP range is 10.21.166.1-254. The guest has IP 192.168.252.3 and is associate to 10.21.166.2. So if I ping 10.21.166.2 from my external network, I can sniff the icmp packets all the way to the VM linux bridge on the compute node. I can see packets on qvb* but not tap*. From the VM console I am able to reach the external network. Packet dumps show that traffic originates from 10.21.166.2. Finally, I see no hits on my secgroup rules. Any advice? I have interesting command output here: http://pastebin.com/Cs514mkN Thanks in advance. -- \*..+.- --Greg Chavez +//..;}; -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [Grizzly] Inbound VM traffic fails at compute node
I have Grizzly up and running on Ubuntu 13.04, following the excellent instructions by Msekni Bilel. I'm using gre tunneling and per-tenant routers. It looks something like this: http://chavezy.files.wordpress.com/2013/03/ostack-log-net_iscsi.png I was able to get a cirros m1.tiny VM launched easily. But although I've associated a floating IP and configured secgroup rules, I am unable to get any inbound traffic past the VM bridge. The internal network is 192.168.252.0/23. The floating IP range is 10.21.166.1-254. The guest has IP 192.168.252.3 and is associate to 10.21.166.2. So if I ping 10.21.166.2 from my external network, I can sniff the icmp packets all the way to the VM linux bridge on the compute node. I can see packets on qvb* but not tap*. From the VM console I am able to reach the external network. Packet dumps show that traffic originates from 10.21.166.2. Finally, I see no hits on my secgroup rules. Any advice? I have interesting command output here: http://pastebin.com/Cs514mkN Thanks in advance. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] How to decommission a compute node
I'm just getting up and running with Folsom on Ubuntu 12.10. Of two compute nodes I have running, one of them died (some kind of grub corruption). So my next step is to re-kick it, but I don't know how to remove the compute node from Nova. I can do this: root@kcon-gen-01i:~# nova-manage service disable --host=kvm-sn-10i --service=nova-compute root@kcon-gen-01i:~# nova-manage service list Binary Host Zone Status State Updated_At nova-certkcon-gen-01i nova enabled :-) 2013-04-30 18:32:56 nova-consoleauth kcon-gen-01i nova enabled :-) 2013-04-30 18:32:56 nova-scheduler kcon-gen-01i nova enabled :-) 2013-04-30 18:32:52 nova-compute kvm-sn-10inova disabled :-) 2013-04-30 18:32:54 nova-compute kvm-sn-14inova enabled :-) 2013-04-30 18:32:55 But the host itself is still there! root@kcon-gen-01i:~# nova-manage host list | sed 's/cs-//' host zone kcon-gen-01i nova kvm-sn-10i nova kvm-sn-14i nova My concern is that when I bring kvm-sn-10i back to life, my controller node won't be able to authorize it. So what is the proper way to delete/remove/decommission a compute node? Thanks. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] br-int not recreated after compute node reboot
I have a fairly fresh Folsom install on Quantal with just a couple of test virts. In preparation for setting up Ceph, I upgraded the kernel to linux-3.6.11 on my compute node, then rebooted. But, as the subject suggests, this went very badly as br-int wasn't recreated. BEFORE: http://pastebin.com/eZegR62A (scroll to line 118 to see the compute node's vitals) AFTER: http://pastebin.com/0MvTRKkV The bridge shows up with ovs-ctrl and I have br-int setup in my /etc/network/interfaces file like this: # Integration Bridge auto br-int iface br-int inet manual up ifconfig $IFACE 0.0.0.0 up down ifconfig $IFACE down So when my test vm's come up, they obviously don't get an IP. The bridge-bridge is missing. Thanks in advance for your suggestions. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Agent out of sync with plugin!
Logan, thanks for your reply. I've been very conscientious of NTP, so I'm very confident that that is not an issue. Gary: so in this case the agent = quantum-plugin-openvswitch-agent, and plugin = quantum-server. That's confusing. And what you're saying is that the ovs plugin/agent - whatever it is - is simply stating that it assumes it's out of sync since it's starting up, and it's going to phone home. Is that right? Thanks! On Tue, Mar 12, 2013 at 3:18 AM, Gary Kotton gkot...@redhat.com wrote: On 03/12/2013 12:13 AM, Greg Chavez wrote: So I'm setting up Folsom on Ubuntu 12.10, using the Github Folsom Install Guide: https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst After following the steps to instantiate the network node, I'm left with 3 new but downed OVS bridges, and three error-filled logs for ovs-plugin, dhcp-agent, and l3-agent. I rectify it with this: http://pastebin.com/L43d9q8a When I restart the networking and then restart the plugin and agents, everything seems to work, but I'm getting this strange INFO message in /var/log/quantum/openvswitch-agent.log: 2013-03-11 17:48:02 INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent out of sync with plugin! I traced it to this .py file: https://github.com/openstack/quantum/blob/master/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py Now I realize that this is and INFO message, not an error. But I would still like to know what this means. Thanks! The Quantum agents need to retrieve the port data from the Quantum service. When the agents start this is done automatically (hence the message that you have seen). This can also happen if there is an exception in the agent or if the agent is unable to communicate with the service - for example there is a problem with the connection between the agent and the service (link down, etc). -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Agent out of sync with plugin!
So I'm setting up Folsom on Ubuntu 12.10, using the Github Folsom Install Guide: https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst After following the steps to instantiate the network node, I'm left with 3 new but downed OVS bridges, and three error-filled logs for ovs-plugin, dhcp-agent, and l3-agent. I rectify it with this: http://pastebin.com/L43d9q8a When I restart the networking and then restart the plugin and agents, everything seems to work, but I'm getting this strange INFO message in /var/log/quantum/openvswitch-agent.log: 2013-03-11 17:48:02 INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent out of sync with plugin! I traced it to this .py file: https://github.com/openstack/quantum/blob/master/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py Now I realize that this is and INFO message, not an error. But I would still like to know what this means. Thanks! -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Fwd: Fwd: Initial quantum network state broken
Hey Anil, thanks for responding. Here's the output: root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-vsctl show 9d9f7949-2b80-40c8-a9e0-6a116200ed96 Bridge br-int Port br-int Interface br-int type: internal Port int-br-eth1 Interface int-br-eth1 Bridge br-eth1 Port phy-br-eth1 Interface phy-br-eth1 Port br-eth1 Interface br-eth1 type: internal Port eth1 Interface eth1 ovs_version: 1.4.3 root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-dpctl show system@br-eth1: lookups: hit:5227 missed:24022 lost:0 flows: 1 port 0: br-eth1 (internal) port 1: eth1 port 6: phy-br-eth1 system@br-int: lookups: hit:2994 missed:13754 lost:0 flows: 1 port 0: br-int (internal) port 2: int-br-eth1 root@kvm-cs-sn-10i:/var/lib/nova/instances# brctl show bridge name bridge id STP enabled interfaces br-eth1 .bc305befedd1 no eth1 phy-br-eth1 br-int .8ae31e5f7941 no int-br-eth1 qbr5334a0cb-64 8000.76fb293fe9cf no qvb5334a0cb-64 vnet0 root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=75251.156s, table=0, n_packets=16581, n_bytes=3186436, priority=2,in_port=2 actions=drop cookie=0x0, duration=75251.527s, table=0, n_packets=0, n_bytes=0, priority=1 actions=NORMAL Thanks! I have until tomorrow to get this working, then my boss has mandating that I try Cloudstack. Argh, but I'm so close! On Wed, Feb 20, 2013 at 10:29 AM, Anil Vishnoi vishnoia...@gmail.comwrote: Hi Greg, Can you paste the output of following command from your compute node. ovs-vsctl show ovs-dpctl show brctl show ovs-ofctl dump-flows br-int ovs-ofctl dump-flows br-eth1 Because i think first issue we need to resolve here is why DHCP packet is not leaving your compute host. Anil On Wed, Feb 20, 2013 at 6:48 AM, Greg Chavez greg.cha...@gmail.comwrote: From my perspective, it seems that the OVS bridges are not being brought up correctly. As you can see in my earlier post, the integration bridge (int) and the physical interface bridges (br-ex and br-eth1) are downed. I've tried to bring them up in promiscuous mode in the case of br-int, and with the physical interfaces ported to the bridge in the case of br-ex and br-eth1. I've had no luck unfortunately. It seems that nothing is getting past br-int. I can see BOOTP packets on the VM side of br-int, and I can see VTP packets on the physical side of br-int. But that's where it ends. For example, when I reboot my VM, I see this: root@kvm-cs-sn-10i:/var/lib/nova/instances# tcpdump -i qvo5334a0cb-64 tcpdump: WARNING: qvo5334a0cb-64: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on qvo5334a0cb-64, link-type EN10MB (Ethernet), capture size 65535 bytes 13:42:08.099061 IP 0.0.0.0.bootpc 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280 13:42:08.101675 IP6 :: ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 13:42:08.161728 IP6 :: ff02::1:ff06:4809: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe06:4809, length 24 13:42:08.373745 IP6 :: ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 13:42:11.102528 IP 0.0.0.0.bootpc 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280 13:42:14.105850 IP 0.0.0.0.bootpc 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280 But that's as far as it goes. The dhcp agent never get this. Ive tried deleting and recreating the bridges, rebooting the systems, but nothing seems to work. Maybe it's just the right combination of things. I don't know. Help! On Tue, Feb 19, 2013 at 5:23 AM, Sylvain Bauza sylvain.ba...@digimind.com wrote: Hi Greg, I did have trouble with DHCP assignation (see my previous post in this list), which was being fixed by deleting ovs bridges on network node, recreating them and restarting OVS plugin and L3/DHCP agents (which were all on the same physical node). Maybe it helps. Anyway, when DHCP'ing from your VM (asking for an IP), could you please tcpdump : 1. your virtual network interface on compute node 2. your physical network interface on compute node 3. your physical network interface on network node and see BOOTP/DHCP packets ? On the physical layer, you should see GRE packets (provided you correctly followed the mentioned guide) encapsulating your BOOTP/DHCP packets. If that's OK, could you please issue the below commands (on the network node) : - brctl show - ip a - ovs-vsctl show - route -n Thanks, -Sylvain Le 19/02/2013 00:55, Greg Chavez a écrit : Third time I'm replying to my own message. It seems like the initial network state is a problem for many first time openstackers
Re: [Openstack] Fwd: Initial quantum network state broken
Here's the last command output you asked for: root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-eth1 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=78793.694s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=6 actions=drop cookie=0x0, duration=78794.033s, table=0, n_packets=17355, n_bytes=3335788, priority=1 actions=NORMAL On Wed, Feb 20, 2013 at 10:57 AM, Greg Chavez greg.cha...@gmail.com wrote: Hey Anil, thanks for responding. Here's the output: root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-vsctl show 9d9f7949-2b80-40c8-a9e0-6a116200ed96 Bridge br-int Port br-int Interface br-int type: internal Port int-br-eth1 Interface int-br-eth1 Bridge br-eth1 Port phy-br-eth1 Interface phy-br-eth1 Port br-eth1 Interface br-eth1 type: internal Port eth1 Interface eth1 ovs_version: 1.4.3 root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-dpctl show system@br-eth1: lookups: hit:5227 missed:24022 lost:0 flows: 1 port 0: br-eth1 (internal) port 1: eth1 port 6: phy-br-eth1 system@br-int: lookups: hit:2994 missed:13754 lost:0 flows: 1 port 0: br-int (internal) port 2: int-br-eth1 root@kvm-cs-sn-10i:/var/lib/nova/instances# brctl show bridge name bridge id STP enabled interfaces br-eth1 .bc305befedd1 no eth1 phy-br-eth1 br-int .8ae31e5f7941 no int-br-eth1 qbr5334a0cb-64 8000.76fb293fe9cf no qvb5334a0cb-64 vnet0 root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=75251.156s, table=0, n_packets=16581, n_bytes=3186436, priority=2,in_port=2 actions=drop cookie=0x0, duration=75251.527s, table=0, n_packets=0, n_bytes=0, priority=1 actions=NORMAL Thanks! I have until tomorrow to get this working, then my boss has mandating that I try Cloudstack. Argh, but I'm so close! On Wed, Feb 20, 2013 at 10:29 AM, Anil Vishnoi vishnoia...@gmail.comwrote: Hi Greg, Can you paste the output of following command from your compute node. ovs-vsctl show ovs-dpctl show brctl show ovs-ofctl dump-flows br-int ovs-ofctl dump-flows br-eth1 Because i think first issue we need to resolve here is why DHCP packet is not leaving your compute host. Anil On Wed, Feb 20, 2013 at 6:48 AM, Greg Chavez greg.cha...@gmail.comwrote: From my perspective, it seems that the OVS bridges are not being brought up correctly. As you can see in my earlier post, the integration bridge (int) and the physical interface bridges (br-ex and br-eth1) are downed. I've tried to bring them up in promiscuous mode in the case of br-int, and with the physical interfaces ported to the bridge in the case of br-ex and br-eth1. I've had no luck unfortunately. It seems that nothing is getting past br-int. I can see BOOTP packets on the VM side of br-int, and I can see VTP packets on the physical side of br-int. But that's where it ends. For example, when I reboot my VM, I see this: root@kvm-cs-sn-10i:/var/lib/nova/instances# tcpdump -i qvo5334a0cb-64 tcpdump: WARNING: qvo5334a0cb-64: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on qvo5334a0cb-64, link-type EN10MB (Ethernet), capture size 65535 bytes 13:42:08.099061 IP 0.0.0.0.bootpc 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280 13:42:08.101675 IP6 :: ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 13:42:08.161728 IP6 :: ff02::1:ff06:4809: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe06:4809, length 24 13:42:08.373745 IP6 :: ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 13:42:11.102528 IP 0.0.0.0.bootpc 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280 13:42:14.105850 IP 0.0.0.0.bootpc 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280 But that's as far as it goes. The dhcp agent never get this. Ive tried deleting and recreating the bridges, rebooting the systems, but nothing seems to work. Maybe it's just the right combination of things. I don't know. Help! On Tue, Feb 19, 2013 at 5:23 AM, Sylvain Bauza sylvain.ba...@digimind.com wrote: Hi Greg, I did have trouble with DHCP assignation (see my previous post in this list), which was being fixed by deleting ovs bridges on network node, recreating them and restarting OVS plugin and L3/DHCP agents (which were all on the same physical node). Maybe it helps. Anyway, when DHCP'ing from your VM (asking for an IP), could you please tcpdump : 1. your virtual network interface on compute node 2. your physical network interface on compute node 3. your physical network interface on network node and see BOOTP/DHCP packets
Re: [Openstack] Openstack] Suggestions for shared-storage cluster file system
So it sounds like what we're talking about here is running a Ceph or GlusterFS node alongside the compute node (ie, on the same physical system). I assume then that VMs access their volumes via NFS or iSCSI from Cinder on the controller node, and in turn, Cinder reads and writes to the cluster FS, which is spread out among the physical data volumes on the compute nodes. Is this correct? On Fri, Feb 15, 2013 at 2:29 PM, Sébastien Han han.sebast...@gmail.comwrote: Hi, Important: Mount the CephFS filesystem on the client machine, not the cluster machine. It's just like NFS, if you mount an NFS export on the NFS server, you get kernel locks. Unfortunately even if love Ceph far more than the other, I won't go with CephFS, at least not know. But if are in a hurry and looking for a DFS then GlusterFS seems to be a good candidate. NFS works pretty well too. Cheers. -- Regards, Sébastien Han. On Fri, Feb 15, 2013 at 4:49 PM, JuanFra Rodriguez Cardoso juanfra.rodriguez.card...@gmail.com wrote: Another one: - MooseFS ( http://docs.openstack.org/trunk/openstack-compute/admin/content/installing-moosefs-as-backend.html ) - GlusterFS - Ceph - Lustre Regards, JuanFra 2013/2/15 Samuel Winchenbach swinc...@gmail.com Hi All, Can anyone give me a recommendation for a good shared-storage cluster filesystem? I am running kvm-libvirt and would like to enable live migration. I have a number of hosts (up to 16) each with 2xTB drives. These hosts are also my compute/network/controller nodes. The three I am considering are: GlusterFS - I have the most experience with this, and it seems the easiest. CephFS/RADOS - Interesting because glance supports the rbd backend. Slightly worried because of this though Important: Mount the CephFS filesystem on the client machine, not the cluster machine. (I wish it said why...) and CephFS is not quite as stable as the block device and the object storage gateway. Lustre - A little hesitant now that Oracle is involved with it. If anyone has any advice, or can point out another that I should consider it would be greatly appreciated. Thanks! Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- \*..+.- --Greg Chavez +//..;}; -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Initial quantum network state broken
Third time I'm replying to my own message. It seems like the initial network state is a problem for many first time openstackers. Surely somewhere would be well to assist me. I'm running out of time to make this work. Thanks. On Sun, Feb 17, 2013 at 3:08 AM, Greg Chavez greg.cha...@gmail.com wrote: I'm replying to my own message because I'm desperate. My network situation is a mess. I need to add this as well: my bridge interfaces are all down. On my compute node: root@kvm-cs-sn-10i:/var/lib/nova/instances/instance-0005# ip addr show | grep ^[0-9] 1: lo: LOOPBACK,UP,LOWER_UP mtu 16436 qdisc noqueue state UNKNOWN 2: eth0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 3: eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 4: eth2: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN qlen 1000 5: eth3: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN qlen 1000 9: br-int: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 10: br-eth1: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 13: phy-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 14: int-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 15: qbre56c5d9e-b6: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP 16: qvoe56c5d9e-b6: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 17: qvbe56c5d9e-b6: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbre56c5d9e-b6 state UP qlen 1000 19: qbrb805a9c9-11: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP 20: qvob805a9c9-11: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 21: qvbb805a9c9-11: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbrb805a9c9-11 state UP qlen 1000 34: qbr2b23c51f-02: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP 35: qvo2b23c51f-02: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 36: qvb2b23c51f-02: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbr2b23c51f-02 state UP qlen 1000 37: vnet0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbr2b23c51f-02 state UNKNOWN qlen 500 And on my network node: root@knet-cs-gen-01i:~# ip addr show | grep ^[0-9] 1: lo: LOOPBACK,UP,LOWER_UP mtu 16436 qdisc noqueue state UNKNOWN 2: eth0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 3: eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 4: eth2: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 5: eth3: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN qlen 1000 6: br-int: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 7: br-eth1: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 8: br-ex: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UNKNOWN 22: phy-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 23: int-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 I gave br-ex an IP and UP'ed it manually. I assume this is correct. By I honestly don't know. Thanks. On Fri, Feb 15, 2013 at 6:54 PM, Greg Chavez greg.cha...@gmail.comwrote: Sigh. So I abandoned RHEL 6.3, rekicked my systems and set up the scale-ready installation described in these instructions: https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst Basically: (o) controller node on a mgmt and public net (o) network node (quantum and openvs) on a mgmt, net-config, and public net (o) compute node is on a mgmt and net-config net Took me just over an hour and ran into only a few easily-fixed speed bumps. But the VM networks are totally non-functioning. VMs launch but no network traffic can go in or out. I'm particularly befuddled by these problems: ( 1 ) This error in nova-compute: ERROR nova.network.quantumv2 [-] _get_auth_token() failed ( 2 ) No NAT rules on the compute node, which probably explains why the VMs complain about not finding a network or being able to get metadata from 169.254.169.254. root@kvm-cs-sn-10i:~# iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N nova-api-metadat-OUTPUT -N nova-api-metadat-POSTROUTING -N nova-api-metadat-PREROUTING -N nova-api-metadat-float-snat -N nova-api-metadat-snat -N nova-compute-OUTPUT -N nova-compute-POSTROUTING -N nova-compute-PREROUTING -N nova-compute-float-snat -N nova-compute-snat -N nova-postrouting-bottom -A PREROUTING -j nova-api-metadat-PREROUTING -A PREROUTING -j nova-compute-PREROUTING -A OUTPUT -j nova-api-metadat-OUTPUT -A OUTPUT -j nova-compute-OUTPUT -A POSTROUTING -j nova-api-metadat-POSTROUTING -A POSTROUTING -j nova
Re: [Openstack] Initial quantum network state broken
I'm replying to my own message because I'm desperate. My network situation is a mess. I need to add this as well: my bridge interfaces are all down. On my compute node: root@kvm-cs-sn-10i:/var/lib/nova/instances/instance-0005# ip addr show | grep ^[0-9] 1: lo: LOOPBACK,UP,LOWER_UP mtu 16436 qdisc noqueue state UNKNOWN 2: eth0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 3: eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 4: eth2: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN qlen 1000 5: eth3: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN qlen 1000 9: br-int: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 10: br-eth1: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 13: phy-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 14: int-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 15: qbre56c5d9e-b6: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP 16: qvoe56c5d9e-b6: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 17: qvbe56c5d9e-b6: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbre56c5d9e-b6 state UP qlen 1000 19: qbrb805a9c9-11: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP 20: qvob805a9c9-11: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 21: qvbb805a9c9-11: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbrb805a9c9-11 state UP qlen 1000 34: qbr2b23c51f-02: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP 35: qvo2b23c51f-02: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 36: qvb2b23c51f-02: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbr2b23c51f-02 state UP qlen 1000 37: vnet0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master qbr2b23c51f-02 state UNKNOWN qlen 500 And on my network node: root@knet-cs-gen-01i:~# ip addr show | grep ^[0-9] 1: lo: LOOPBACK,UP,LOWER_UP mtu 16436 qdisc noqueue state UNKNOWN 2: eth0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 3: eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 4: eth2: BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 5: eth3: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN qlen 1000 6: br-int: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 7: br-eth1: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN 8: br-ex: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UNKNOWN 22: phy-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 23: int-br-eth1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000 I gave br-ex an IP and UP'ed it manually. I assume this is correct. By I honestly don't know. Thanks. On Fri, Feb 15, 2013 at 6:54 PM, Greg Chavez greg.cha...@gmail.com wrote: Sigh. So I abandoned RHEL 6.3, rekicked my systems and set up the scale-ready installation described in these instructions: https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst Basically: (o) controller node on a mgmt and public net (o) network node (quantum and openvs) on a mgmt, net-config, and public net (o) compute node is on a mgmt and net-config net Took me just over an hour and ran into only a few easily-fixed speed bumps. But the VM networks are totally non-functioning. VMs launch but no network traffic can go in or out. I'm particularly befuddled by these problems: ( 1 ) This error in nova-compute: ERROR nova.network.quantumv2 [-] _get_auth_token() failed ( 2 ) No NAT rules on the compute node, which probably explains why the VMs complain about not finding a network or being able to get metadata from 169.254.169.254. root@kvm-cs-sn-10i:~# iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N nova-api-metadat-OUTPUT -N nova-api-metadat-POSTROUTING -N nova-api-metadat-PREROUTING -N nova-api-metadat-float-snat -N nova-api-metadat-snat -N nova-compute-OUTPUT -N nova-compute-POSTROUTING -N nova-compute-PREROUTING -N nova-compute-float-snat -N nova-compute-snat -N nova-postrouting-bottom -A PREROUTING -j nova-api-metadat-PREROUTING -A PREROUTING -j nova-compute-PREROUTING -A OUTPUT -j nova-api-metadat-OUTPUT -A OUTPUT -j nova-compute-OUTPUT -A POSTROUTING -j nova-api-metadat-POSTROUTING -A POSTROUTING -j nova-compute-POSTROUTING -A POSTROUTING -j nova-postrouting-bottom -A nova-api-metadat-snat -j nova-api-metadat-float-snat -A nova-compute-snat -j nova-compute-float-snat -A nova-postrouting-bottom -j nova-api-metadat-snat -A nova-postrouting-bottom -j nova-compute-snat (3) A lastly, no default secgroup rules, whose function governs... what exactly? Connections
[Openstack] Horizon Keystone Nova miscommunication
It seems that nova and horizon are not communicating on my controller node.
Acces and security objects created with nova are not seen by Horizon and
vice versa. This includes key pairs and secgroup rules. For example, if I
create a keypair with the nova client, it isn't visible in horizon, and if
I create one in horizon it is not visible via the nova client.
Possibly related: VMs that I create, whether via the nova client or
Horizon, are not shown with I run nova list. The nova-api.log shows a
successful servers-detail query, but it comes back empty.
Also possibly related: Although I have all my services and endpoints
configured correctly, I can't get individual endpoint detail with
endpoint-get. What's more, I see this error in Horizon's error log:
[Sun Feb 17 07:02:50 2013] [error] EndpointNotFound: Endpoint not found.
[Sun Feb 17 07:06:55 2013] [error] unable to retrieve service catalog with
token
This matches what I get when I run:
$ keystone endpoint-get --service nova
Endpoint not found.
But that can't be because endpoint-list shows all six endpoints I created
and all the information seems correct in the database:
mysql select * from endpoint where service_id
='9e40d355b49342f8ac6947c497df76d2'\G
*** 1. row ***
id: 922baafde75f4cffa7dbe7f57cddb951
region: RegionOne
service_id: 9e40d355b49342f8ac6947c497df76d2
extra: {adminurl: http://192.168.241.100:35357/v2.0;,
internalurl: http://192.168.241.100:5000/v2.0;, publicurl:
http://10.21.164.75:5000/v2.0}
1 row in set (0.00 sec)
mysql select * from service where id ='9e40d355b49342f8ac6947c497df76d2'\G
*** 1. row ***
id: 9e40d355b49342f8ac6947c497df76d2
type: identity
extra: {description: OpenStack Identity, name: keystone}
1 row in set (0.00 sec)
Please please please help me. My boss is giving my project the ax on
Monday if I can't get this to work.
--
\*..+.-
--Greg Chavez
+//..;};
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] Initial quantum network state broken
Sigh. So I abandoned RHEL 6.3, rekicked my systems and set up the scale-ready installation described in these instructions: https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst Basically: (o) controller node on a mgmt and public net (o) network node (quantum and openvs) on a mgmt, net-config, and public net (o) compute node is on a mgmt and net-config net Took me just over an hour and ran into only a few easily-fixed speed bumps. But the VM networks are totally non-functioning. VMs launch but no network traffic can go in or out. I'm particularly befuddled by these problems: ( 1 ) This error in nova-compute: ERROR nova.network.quantumv2 [-] _get_auth_token() failed ( 2 ) No NAT rules on the compute node, which probably explains why the VMs complain about not finding a network or being able to get metadata from 169.254.169.254. root@kvm-cs-sn-10i:~# iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N nova-api-metadat-OUTPUT -N nova-api-metadat-POSTROUTING -N nova-api-metadat-PREROUTING -N nova-api-metadat-float-snat -N nova-api-metadat-snat -N nova-compute-OUTPUT -N nova-compute-POSTROUTING -N nova-compute-PREROUTING -N nova-compute-float-snat -N nova-compute-snat -N nova-postrouting-bottom -A PREROUTING -j nova-api-metadat-PREROUTING -A PREROUTING -j nova-compute-PREROUTING -A OUTPUT -j nova-api-metadat-OUTPUT -A OUTPUT -j nova-compute-OUTPUT -A POSTROUTING -j nova-api-metadat-POSTROUTING -A POSTROUTING -j nova-compute-POSTROUTING -A POSTROUTING -j nova-postrouting-bottom -A nova-api-metadat-snat -j nova-api-metadat-float-snat -A nova-compute-snat -j nova-compute-float-snat -A nova-postrouting-bottom -j nova-api-metadat-snat -A nova-postrouting-bottom -j nova-compute-snat (3) A lastly, no default secgroup rules, whose function governs... what exactly? Connections to the VM's public or private IPs? I guess I'm just not sure if this is relevant to my overall problem of ZERO VM network connectivity. I seek guidance please. Thanks. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Network cannot be found
Brilliant. Ran it like this:
nova --os-username demo --os-password demo --os-tenant-name demoProject
boot --image aefa581f-47b0-4d46-8dbc-1a1f7f02dfa0 --flavor 2 --nic
net-id=3de1e780-07d1-42af-89cc-0feaf1ece6e9 server-01
And it spawned! Into an ERROR state, perhaps, but it spawned! I'm telling
you, it's like whack-a-mole with thing. I'll kick off a new email with the
new problem if I can't fix it soon.
Thanks for your help though.
On Mon, Feb 11, 2013 at 12:40 AM, Aaron Rosen aro...@nicira.com wrote:
Hi Greg,
I checked out why this can happen and I think the reason you are getting
this is because you are trying to create a vm as an admin user on a network
that you are not the owner.
https://github.com/openstack/nova/blob/master/nova/network/quantumv2/api.py#L95
Thanks,
Aaron
On Sun, Feb 10, 2013 at 4:28 PM, Greg Chavez greg.cha...@gmail.comwrote:
This has me utterly befuddled. All I'm trying to do is bring up a Cirros
VM on my 3-node controller+network+compute Folsom setup. So nova is
configured on the controller to use quantum:
network_api_class = nova.network.quantumv2.api.API
quantum_admin_username = quantum
quantum_admin_password = quantum
quantum_admin_auth_url = http://192.168.241.100:35357/v2.0
quantum_auth_strategy = keystone
quantum_admin_tenant_name = service
quantum_url = http://192.168.241.100:9696
192.168.241.100 is the controller management port IP.
I have the following network available:
# quantum net-list | grep demo-net
| 3de1e780-07d1-42af-89cc-0feaf1ece6e9 | demo-net |
7f54d28f-b87a-43f7-a15f-1c0d20569e2a |
# quantum subnet-list | grep 7f54d28f
| 7f54d28f-b87a-43f7-a15f-1c0d20569e2a | | 192.168.1.0/24 |
{start: 192.168.1.2, end: 192.168.1.254} |
# quantum net-show 3de1e780-07d1-42af-89cc-0feaf1ece6e9
+---+--+
| Field | Value|
+---+--+
| admin_state_up| True |
| id| 3de1e780-07d1-42af-89cc-0feaf1ece6e9 |
| name | demo-net |
| provider:network_type | vlan |
| provider:physical_network | physnet1 |
| provider:segmentation_id | 2001 |
| router:external | False|
| shared| False|
| status| ACTIVE |
| subnets | 7f54d28f-b87a-43f7-a15f-1c0d20569e2a |
| tenant_id | 6ea263cf12ce44e29d4d7baefacb9689 |
+---+--+
# quantum router-port-list 37d3b2dd-ad19-435b-b4b9-3edbcd812f58
..snip snip..
{subnet_id: 7f54d28f-b87a-43f7-a15f-1c0d20569e2a, ip_address:
192.168.1.1} |
{subnet_id: e45987aa-2f3a-45e7-b9c8-7499d8eb93e6, ip_address:
10.21.166.2}
So far so good. From my perspective everything looks good. But when I
try to boot a vm it fails:
# nova boot --image aefa581f-47b0-4d46-8dbc-1a1f7f02dfa0 --flavor 2 --nic
net-id=3de1e780-07d1-42af-89cc-0feaf1ece6e9 --key-name demo-key server-01
ERROR: The resource could not be found. (HTTP 404) (Request-ID:
req-b0badcb3-244a-4e76-92b2-8bbabc645f3b)
Looking in api.log I find this error:
Caught error: Network 3de1e780-07d1-42af-89cc-0feaf1ece6e9 could not be
found.
WHAT! How is it not found? It clearly exists, unless net-id is expecting
something else as its value. Argh.
Thanks in advance.
--
\*..+.-
--Greg Chavez
+//..;};
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] brctl meltdown on RHEL 6.3
Running latest EPEL Folsom packages on RHEL 6.3. Three nodes right now, one controller, one network node, one compute node. The network node has three NICs, one for external net, one for management net, one for VM network traffic. It has been a miserable journey so far. The lastest calamity began with a failed spawn of the Cirros test image. I booted it like this: # nova --os-username demo --os-password demo --os-tenant-name demoProject boot --image aefa581f-47b0-4d46-8dbc-1a1f7f02dfa0 --flavor 2 --nic net-id=3de1e780-07d1-42af-89cc-0feaf1ece6e9 server-01 This succeeded but went directly into an ERROR state. The compute node's /var/log/nova/compute.log showed this: ProcessExecutionError: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addbr qbr2218b8c4-7d Exit code: 1 Stdout: '' Stderr: 'add bridge failed: Package not installed\n' Hrm. So then I ran this: # brctl show bridge name bridge id STP enabled interfaces br-eth1 /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory .bc305befedd1 no br-int /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory .7e1636f42c4b no GAH! What!!! First of all, bridge capability is set by default in the RHEL 6.3 kernel. Secondly, nova knows that it's supposed to be using openvswitch. The ProcessExecutionError's trace showed that the offending code came from /usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py line 216 which has this comment: def plug(self, instance, vif): Plug using hybrid strategy Create a per-VIF linux bridge, then link that bridge to the OVS integration bridge via a veth device, setting up the other end of the veth device just like a normal OVS port. Then boot the VIF on the linux bridge using standard libvirt mechanisms Thirdly, ovs-vsctrl is happy: # ovs-vsctl show 44435595-8cc8-469c-ace4-ded76a7b864d Bridge br-eth1 Port br-eth1 Interface br-eth1 type: internal Port phy-br-eth1 Interface phy-br-eth1 Port eth1 Interface eth1 Bridge br-int Port int-br-eth1 Interface int-br-eth1 Port br-int Interface br-int type: internal ovs_version: 1.7.3 Final note, my network node fails the same way, but the controller does not. I hope so much that somebody knows what is going on here. This is very terrible for me as I am struggling to achieve minimal functionality. Thanks. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] brctl meltdown on RHEL 6.3
Solution: [root@kvm-cs-sn-10i nova]# modprobe -r brcompat [root@kvm-cs-sn-10i nova]# modprobe bridge [root@kvm-cs-sn-10i nova]# brctl show bridge name bridge id STP enabled interfaces Still can't boot a VM... looking into the reasons now. On Mon, Feb 11, 2013 at 11:33 AM, Greg Chavez greg.cha...@gmail.com wrote: Running latest EPEL Folsom packages on RHEL 6.3. Three nodes right now, one controller, one network node, one compute node. The network node has three NICs, one for external net, one for management net, one for VM network traffic. It has been a miserable journey so far. The lastest calamity began with a failed spawn of the Cirros test image. I booted it like this: # nova --os-username demo --os-password demo --os-tenant-name demoProject boot --image aefa581f-47b0-4d46-8dbc-1a1f7f02dfa0 --flavor 2 --nic net-id=3de1e780-07d1-42af-89cc-0feaf1ece6e9 server-01 This succeeded but went directly into an ERROR state. The compute node's /var/log/nova/compute.log showed this: ProcessExecutionError: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addbr qbr2218b8c4-7d Exit code: 1 Stdout: '' Stderr: 'add bridge failed: Package not installed\n' Hrm. So then I ran this: # brctl show bridge name bridge id STP enabled interfaces br-eth1 /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory /sys/class/net/br-eth1/bridge: No such file or directory .bc305befedd1 no br-int /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory /sys/class/net/br-int/bridge: No such file or directory .7e1636f42c4b no GAH! What!!! First of all, bridge capability is set by default in the RHEL 6.3 kernel. Secondly, nova knows that it's supposed to be using openvswitch. The ProcessExecutionError's trace showed that the offending code came from /usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py line 216 which has this comment: def plug(self, instance, vif): Plug using hybrid strategy Create a per-VIF linux bridge, then link that bridge to the OVS integration bridge via a veth device, setting up the other end of the veth device just like a normal OVS port. Then boot the VIF on the linux bridge using standard libvirt mechanisms Thirdly, ovs-vsctrl is happy: # ovs-vsctl show 44435595-8cc8-469c-ace4-ded76a7b864d Bridge br-eth1 Port br-eth1 Interface br-eth1 type: internal Port phy-br-eth1 Interface phy-br-eth1 Port eth1 Interface eth1 Bridge br-int Port int-br-eth1 Interface int-br-eth1 Port br-int Interface br-int type: internal ovs_version: 1.7.3 Final note, my network node fails the same way, but the controller does not. I hope so much that somebody knows what is going on here. This is very terrible for me as I am struggling to achieve minimal functionality. Thanks. -- \*..+.- --Greg Chavez +//..;}; -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Network cannot be found
This has me utterly befuddled. All I'm trying to do is bring up a Cirros
VM on my 3-node controller+network+compute Folsom setup. So nova is
configured on the controller to use quantum:
network_api_class = nova.network.quantumv2.api.API
quantum_admin_username = quantum
quantum_admin_password = quantum
quantum_admin_auth_url = http://192.168.241.100:35357/v2.0
quantum_auth_strategy = keystone
quantum_admin_tenant_name = service
quantum_url = http://192.168.241.100:9696
192.168.241.100 is the controller management port IP.
I have the following network available:
# quantum net-list | grep demo-net
| 3de1e780-07d1-42af-89cc-0feaf1ece6e9 | demo-net |
7f54d28f-b87a-43f7-a15f-1c0d20569e2a |
# quantum subnet-list | grep 7f54d28f
| 7f54d28f-b87a-43f7-a15f-1c0d20569e2a | | 192.168.1.0/24 | {start:
192.168.1.2, end: 192.168.1.254} |
# quantum net-show 3de1e780-07d1-42af-89cc-0feaf1ece6e9
+---+--+
| Field | Value|
+---+--+
| admin_state_up| True |
| id| 3de1e780-07d1-42af-89cc-0feaf1ece6e9 |
| name | demo-net |
| provider:network_type | vlan |
| provider:physical_network | physnet1 |
| provider:segmentation_id | 2001 |
| router:external | False|
| shared| False|
| status| ACTIVE |
| subnets | 7f54d28f-b87a-43f7-a15f-1c0d20569e2a |
| tenant_id | 6ea263cf12ce44e29d4d7baefacb9689 |
+---+--+
# quantum router-port-list 37d3b2dd-ad19-435b-b4b9-3edbcd812f58
..snip snip..
{subnet_id: 7f54d28f-b87a-43f7-a15f-1c0d20569e2a, ip_address:
192.168.1.1} |
{subnet_id: e45987aa-2f3a-45e7-b9c8-7499d8eb93e6, ip_address:
10.21.166.2}
So far so good. From my perspective everything looks good. But when I try
to boot a vm it fails:
# nova boot --image aefa581f-47b0-4d46-8dbc-1a1f7f02dfa0 --flavor 2 --nic
net-id=3de1e780-07d1-42af-89cc-0feaf1ece6e9 --key-name demo-key server-01
ERROR: The resource could not be found. (HTTP 404) (Request-ID:
req-b0badcb3-244a-4e76-92b2-8bbabc645f3b)
Looking in api.log I find this error:
Caught error: Network 3de1e780-07d1-42af-89cc-0feaf1ece6e9 could not be
found.
WHAT! How is it not found? It clearly exists, unless net-id is expecting
something else as its value. Argh.
Thanks in advance.
--
\*..+.-
--Greg Chavez
+//..;};
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova.Scheduler TypeError: can't compare datetime.datetime to NoneType
Thanks, Unmesh. My nova.conf on my compute node had some conflicting settings. E.g. I am using quantum, but I still had network_manager = nova.network.manager.FlatDHCPManager. That error is gone. But naturally I have new ones now. I'll start a new thread for that :) On Fri, Feb 8, 2013 at 12:49 AM, Unmesh Gurjar unmesh.gur...@gmail.comwrote: Greg, IMO, this qualifies as a bug. To debug what's causing this scenario is your case, I would suggest checking the database connection setting in nova.conf on your Compute Controller host. AFAIK, it either means Compute's status is not being updated (Compute service down) or its being updated in a different database server. Hope it helps. Regards, Unmesh G. On Fri, Feb 8, 2013 at 5:55 AM, Greg Chavez greg.cha...@gmail.com wrote: So I'm getting this when I try to boot a Cirros image in scheduler.log. I have a controller node, network node, and one compute node setup with Folsom on RHEL 6. Any ideas? Thanks! 2013-02-07 19:21:45 WARNING nova.scheduler.manager [req-358f5c96-e236-4167-b6a9-ccc02c853d41 65fdf1a86cfa4e50aafb6c132106e00f cec6647113c74b66b308de14032e591a] Failed to schedule_run_instance: can't compare datetime.datetime to NoneType 2013-02-07 19:21:45 WARNING nova.scheduler.manager [req-358f5c96-e236-4167-b6a9-ccc02c853d41 65fdf1a86cfa4e50aafb6c132106e00f cec6647113c74b66b308de14032e591a] [instance: 1a318b16-7d98-40ef-acb9-63d705542200] Setting instance to ERROR state. 2013-02-07 19:21:45 1688 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py, line 276, in _process_data 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py, line 145, in dispatch 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/manager.py, line 118, in run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp context, ex, request_spec) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib64/python2.6/contextlib.py, line 23, in __exit__ 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp self.gen.next() 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/manager.py, line 106, in run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp requested_networks, is_first_time, filter_properties) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/multi.py, line 69, in schedule_run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp return self.drivers['compute'].schedule_run_instance(*args, **kwargs) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/filter_scheduler.py, line 73, in schedule_run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp filter_properties, instance_uuids) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/filter_scheduler.py, line 251, in _schedule 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp elevated, topic) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/host_manager.py, line 284, in get_all_host_states 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp host_state.update_from_compute_node(compute) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/host_manager.py, line 125, in update_from_compute_node 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp if self.updated and self.updated compute['updated_at']: 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp TypeError: can't compare datetime.datetime to NoneType -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Thanks Regards, Unmesh G. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https
[Openstack] Compute Node OSError: [Errno 13] Permission denied: '/run'\n
-0654-45f0-8c81-8aaa3acdcd65] -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Compute Node OSError: [Errno 13] Permission denied: '/run'\n
Figured it out myself. The sample nova.conf at https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rsthas this setting: lock_path=/run/lock/nova Gah. Must be an Ubuntu thing? I'm running on RHEL 6.3 I changed this to: lock_path=/var/lock/nova And then I had to run this: usermod -G lock nova After that: [root@kvm-cs-sn-10i nova]# virsh list IdName State 1 instance-000a running On Fri, Feb 8, 2013 at 12:05 PM, Greg Chavez greg.cha...@gmail.com wrote: It's a slog, but I appear to be on the cusp on launching my first VM with using the 3-NIC network node setup outline by the OpenStack_Folsom_Install_Guide_WebVersion.rst guide at github. Right now, I'm getting this error on my compute node when I try to launch the Cirros test image. I am so close. Can anyone help? Thanks. 2013-02-08 11:42:09 ERROR nova.compute.manager [req-2aa04352-6e7f-4c54-abcd-4586ce522266 65fdf1a86cfa4e50aafb6c132106e00f cec6647113c74b66b308de14032e591a] [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] Instance failed to spawn 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] Traceback (most recent call last): 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/compute/manager.py, line 747, in _spawn 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] block_device_info) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/exception.py, line 117, in wrapped 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] temp_level, payload) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib64/python2.6/contextlib.py, line 23, in __exit__ 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self.gen.next() 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/exception.py, line 92, in wrapped 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] return f(*args, **kw) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py, line 1092, in spawn 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] block_device_info) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py, line 1929, in _create_domain_and_network 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self.firewall_driver.setup_basic_filtering(instance, network_info) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/libvirt/firewall.py, line 215, in setup_basic_filtering 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self.refresh_provider_fw_rules() 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/firewall.py, line 453, in refresh_provider_fw_rules 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self._do_refresh_provider_fw_rules() 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/utils.py, line 739, in inner 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] ensure_tree(local_lock_path) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/utils.py, line 1324, in ensure_tree 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] os.makedirs(path) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib64/python2.6/os.py, line 150, in makedirs 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] makedirs(head, mode) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654
Re: [Openstack] Compute Node OSError: [Errno 13] Permission denied: '/run'\n
The github install document is the only one I've found so far (including the docs.openstack.org docs) that provides a quick way to establish a robust cloud infrastructure using all the non-deprecated services and methods. I know it's for Ubuntu systems, so I cross-reference Fedora/RHEL guides to fill in the gaps. Very frustrating... but it's gotten me closer than following the Fedora/RHEL guides alone. I'll post other problems as I come across them, believe me! Thanks On Fri, Feb 8, 2013 at 12:44 PM, Anne Gentle a...@openstack.org wrote: Good to know, thanks for the follow-up Greg. Yes, that guide clearly states it's for Ubuntu 12.10, glad that's the only gotcha (is it?). In the official docs we maintain two versions of nova.conf, now I'm going to double-check that in our guides. Thanks for posting. Anne On Fri, Feb 8, 2013 at 11:21 AM, Greg Chavez greg.cha...@gmail.comwrote: Figured it out myself. The sample nova.conf at https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rsthas this setting: lock_path=/run/lock/nova Gah. Must be an Ubuntu thing? I'm running on RHEL 6.3 I changed this to: lock_path=/var/lock/nova And then I had to run this: usermod -G lock nova After that: [root@kvm-cs-sn-10i nova]# virsh list IdName State 1 instance-000a running On Fri, Feb 8, 2013 at 12:05 PM, Greg Chavez greg.cha...@gmail.comwrote: It's a slog, but I appear to be on the cusp on launching my first VM with using the 3-NIC network node setup outline by the OpenStack_Folsom_Install_Guide_WebVersion.rst guide at github. Right now, I'm getting this error on my compute node when I try to launch the Cirros test image. I am so close. Can anyone help? Thanks. 2013-02-08 11:42:09 ERROR nova.compute.manager [req-2aa04352-6e7f-4c54-abcd-4586ce522266 65fdf1a86cfa4e50aafb6c132106e00f cec6647113c74b66b308de14032e591a] [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] Instance failed to spawn 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] Traceback (most recent call last): 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/compute/manager.py, line 747, in _spawn 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] block_device_info) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/exception.py, line 117, in wrapped 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] temp_level, payload) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib64/python2.6/contextlib.py, line 23, in __exit__ 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self.gen.next() 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/exception.py, line 92, in wrapped 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] return f(*args, **kw) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py, line 1092, in spawn 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] block_device_info) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py, line 1929, in _create_domain_and_network 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self.firewall_driver.setup_basic_filtering(instance, network_info) 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/libvirt/firewall.py, line 215, in setup_basic_filtering 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self.refresh_provider_fw_rules() 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] File /usr/lib/python2.6/site-packages/nova/virt/firewall.py, line 453, in refresh_provider_fw_rules 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7-0654-45f0-8c81-8aaa3acdcd65] self._do_refresh_provider_fw_rules() 2013-02-08 11:42:09 40898 TRACE nova.compute.manager [instance: 563385d7
[Openstack] Nova.Scheduler TypeError: can't compare datetime.datetime to NoneType
So I'm getting this when I try to boot a Cirros image in scheduler.log. I have a controller node, network node, and one compute node setup with Folsom on RHEL 6. Any ideas? Thanks! 2013-02-07 19:21:45 WARNING nova.scheduler.manager [req-358f5c96-e236-4167-b6a9-ccc02c853d41 65fdf1a86cfa4e50aafb6c132106e00f cec6647113c74b66b308de14032e591a] Failed to schedule_run_instance: can't compare datetime.datetime to NoneType 2013-02-07 19:21:45 WARNING nova.scheduler.manager [req-358f5c96-e236-4167-b6a9-ccc02c853d41 65fdf1a86cfa4e50aafb6c132106e00f cec6647113c74b66b308de14032e591a] [instance: 1a318b16-7d98-40ef-acb9-63d705542200] Setting instance to ERROR state. 2013-02-07 19:21:45 1688 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py, line 276, in _process_data 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py, line 145, in dispatch 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/manager.py, line 118, in run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp context, ex, request_spec) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib64/python2.6/contextlib.py, line 23, in __exit__ 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp self.gen.next() 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/manager.py, line 106, in run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp requested_networks, is_first_time, filter_properties) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/multi.py, line 69, in schedule_run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp return self.drivers['compute'].schedule_run_instance(*args, **kwargs) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/filter_scheduler.py, line 73, in schedule_run_instance 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp filter_properties, instance_uuids) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/filter_scheduler.py, line 251, in _schedule 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp elevated, topic) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/host_manager.py, line 284, in get_all_host_states 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp host_state.update_from_compute_node(compute) 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/scheduler/host_manager.py, line 125, in update_from_compute_node 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp if self.updated and self.updated compute['updated_at']: 2013-02-07 19:21:45 1688 TRACE nova.openstack.common.rpc.amqp TypeError: can't compare datetime.datetime to NoneType -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] openvswitch rpm on RHEL 6.3 frustration
Ken'ichi, thank you again. You actually had the answer to my problem in
your first reply, but I did not listen.
If you install openvswitch first, it fails dependency checking looking for
openvswitch-kmod. However, if you install kmod-openvswitch *first*,
the dependency checks for the openvswitch rpm passes somehow.
The vagaries of rpms. If I had time, I'd crack open these rpms and see how
this was possible. Clearly it's not as simple as checking the name of the
rpm.
On Fri, Feb 1, 2013 at 2:53 AM, Ken'ichi Ohmichi
oomi...@mxs.nes.nec.co.jpwrote:
Hi Greg,
On Fri, 1 Feb 2013 02:25:00 -0500
Greg Chavez greg.cha...@gmail.com wrote:
Arigato gozaimasu. Where did you get that rpm?
Dou itashimashite,
I built the rpm files from source file by the following way:
# wget http://openvswitch.org/releases/openvswitch-1.7.3.tar.gz
# tar -zxvf openvswitch-1.7.3.tar.gz
# vi ./openvswitch-1.7.3/datapath/linux/compat/include/linux/skbuff.h
-- To avoid compile error, I remove the following lines. --
-#if LINUX_VERSION_CODE KERNEL_VERSION(3,2,0)
-static inline struct page *skb_frag_page(const skb_frag_t *frag)
-{
- return frag-page;
-}
-#endif
-
# rm openvswitch-1.7.3.tar.gz
# tar -zcvf openvswitch-1.7.3.tar.gz openvswitch-1.7.3/
# mv openvswitch-1.7.3.tar.gz rpmbuild/SOURCES/
# rpmbuild -bb -D `uname -r` openvswitch-1.7.3/rhel/openvswitch.spec
# rpmbuild -bb -D `uname -r`
openvswitch-1.7.3/rhel/openvswitch-kmod-rhel6.spec
# ls ~/rpmbuild/RPMS/x86_64/
kmod-openvswitch-1.7.3-1.el6.x86_64.rpm openvswitch-1.7.3-1.x86_64.rpm
openvswitch-debuginfo-1.7.3-1.x86_64.rpm
#
Thanks
Ken'ichi Ohmichi
---
On Fri, Feb 1, 2013 at 1:19 AM, Ken'ichi Ohmichi
oomi...@mxs.nes.nec.co.jpwrote:
Hi Greg,
I can install openvswitch-1.7.3-1.x86_64.rpm on CentOS6.3 if specifying
with kmod-openkmod-openvswitch-1.7.3-1.el6.x86_64.rpm:
# rpm -ivh openvswitch-1.7.3-1.x86_64.rpm
kmod-openvswitch-1.7.3-1.el6.x86_64.rpm
Preparing...###
[100%]
1:kmod-openvswitch
### [
50%]
2:openvswitch###
[100%]
#
Thanks
Ken'ichi Ohmichi
---
On Thu, 31 Jan 2013 23:52:39 -0500
Greg Chavez greg.cha...@gmail.com wrote:
On RHEL 6.3, with EPEL repos, we have
openstack-quantum-openvswitch.noarch.
It requires openvswitch.x86_64, which isn't provided by either RHEL
channels or EPEL!
So I tracked down the openvswitch source (not that hard) , and after
a
few
hours of battling various compilation erros, ended up with three rpms
that
I add to my mrepo yum repo:
[root@cs-repo-02 SysEng6-x86_64]# ls *openv*
kmod-openvswitch-1.7.3-1.el6.x86_64.rpm
openvswitch-1.7.3-1.x86_64.rpm
openvswitch-debuginfo-1.7.3-1.x86_64.rpm
So I re-run my attempt to install the OpenVswitch plugin and guess
what!
Error: Package: openvswitch-1.7.3-1.x86_64 (syseng6)
Requires: openvswitch-kmod
You say openvswitch-kmod, I say kmod-openvswitch, you say go F
yourself.
It's late, I've been working this for weeks. This is just the latest
obstacle. Please help.
The OpenVswitch people are few and unresponsive, by the way. Useless
mailing lists.
--
\*..+.-
--Greg Chavez
+//..;};
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
--
\*..+.-
--Greg Chavez
+//..;};
--
\*..+.-
--Greg Chavez
+//..;};
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Centos : yum install openstack-quantum-openvswitch
Hey guys, I'm having the same problem on RHEL 6.3. Did a search on openvswitch at RHN and it came up with nothing. Where is it? Fedora's core repos? On Sun, Nov 18, 2012 at 4:15 PM, George Lekatsas glekats...@gmail.comwrote: Hello, following the installation instruction and yum install openstack-quantum-openvswitch in centos 6.3 i have: Loaded plugins: fastestmirror, security Loading mirror speeds from cached hostfile * base: ftp. * epel: ftp.cc. * extras: ftp. * updates: ftp. Setting up Install Process Resolving Dependencies -- Running transaction check --- Package openstack-quantum-openvswitch.noarch 0:2012.2-2.el6 will be installed -- Processing Dependency: openvswitch for package: openstack-quantum-openvswitch-2012.2-2.el6.noarch -- Finished Dependency Resolution Error: Package: openstack-quantum-openvswitch-2012.2-2.el6.noarch (epel) Requires: openvswitch You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Do i have to install openvswitch from sources??? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Centos : yum install openstack-quantum-openvswitch
Disco: https://github.com/homework/openvswitch/blob/master/INSTALL.RHEL. And then, in my case, I'm just adding it to a local mrepo configuration. That really sucks, though. C'mon EPEL. Seriously, this is one of the hardest application deployments I've ever done. It's actually worse than OpenNMS! On Thu, Jan 31, 2013 at 6:19 PM, Greg Chavez greg.cha...@gmail.com wrote: Hey guys, I'm having the same problem on RHEL 6.3. Did a search on openvswitch at RHN and it came up with nothing. Where is it? Fedora's core repos? On Sun, Nov 18, 2012 at 4:15 PM, George Lekatsas glekats...@gmail.comwrote: Hello, following the installation instruction and yum install openstack-quantum-openvswitch in centos 6.3 i have: Loaded plugins: fastestmirror, security Loading mirror speeds from cached hostfile * base: ftp. * epel: ftp.cc. * extras: ftp. * updates: ftp. Setting up Install Process Resolving Dependencies -- Running transaction check --- Package openstack-quantum-openvswitch.noarch 0:2012.2-2.el6 will be installed -- Processing Dependency: openvswitch for package: openstack-quantum-openvswitch-2012.2-2.el6.noarch -- Finished Dependency Resolution Error: Package: openstack-quantum-openvswitch-2012.2-2.el6.noarch (epel) Requires: openvswitch You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Do i have to install openvswitch from sources??? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- \*..+.- --Greg Chavez +//..;}; -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] openvswitch rpm on RHEL 6.3 frustration
On RHEL 6.3, with EPEL repos, we have openstack-quantum-openvswitch.noarch. It requires openvswitch.x86_64, which isn't provided by either RHEL channels or EPEL! So I tracked down the openvswitch source (not that hard) , and after a few hours of battling various compilation erros, ended up with three rpms that I add to my mrepo yum repo: [root@cs-repo-02 SysEng6-x86_64]# ls *openv* kmod-openvswitch-1.7.3-1.el6.x86_64.rpm openvswitch-1.7.3-1.x86_64.rpm openvswitch-debuginfo-1.7.3-1.x86_64.rpm So I re-run my attempt to install the OpenVswitch plugin and guess what! Error: Package: openvswitch-1.7.3-1.x86_64 (syseng6) Requires: openvswitch-kmod You say openvswitch-kmod, I say kmod-openvswitch, you say go F yourself. It's late, I've been working this for weeks. This is just the latest obstacle. Please help. The OpenVswitch people are few and unresponsive, by the way. Useless mailing lists. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] openvswitch rpm on RHEL 6.3 frustration
Ken'ich, Arigato gozaimasu. Where did you get that rpm? On Fri, Feb 1, 2013 at 1:19 AM, Ken'ichi Ohmichi oomi...@mxs.nes.nec.co.jpwrote: Hi Greg, I can install openvswitch-1.7.3-1.x86_64.rpm on CentOS6.3 if specifying with kmod-openkmod-openvswitch-1.7.3-1.el6.x86_64.rpm: # rpm -ivh openvswitch-1.7.3-1.x86_64.rpm kmod-openvswitch-1.7.3-1.el6.x86_64.rpm Preparing...### [100%] 1:kmod-openvswitch ### [ 50%] 2:openvswitch### [100%] # Thanks Ken'ichi Ohmichi --- On Thu, 31 Jan 2013 23:52:39 -0500 Greg Chavez greg.cha...@gmail.com wrote: On RHEL 6.3, with EPEL repos, we have openstack-quantum-openvswitch.noarch. It requires openvswitch.x86_64, which isn't provided by either RHEL channels or EPEL! So I tracked down the openvswitch source (not that hard) , and after a few hours of battling various compilation erros, ended up with three rpms that I add to my mrepo yum repo: [root@cs-repo-02 SysEng6-x86_64]# ls *openv* kmod-openvswitch-1.7.3-1.el6.x86_64.rpm openvswitch-1.7.3-1.x86_64.rpm openvswitch-debuginfo-1.7.3-1.x86_64.rpm So I re-run my attempt to install the OpenVswitch plugin and guess what! Error: Package: openvswitch-1.7.3-1.x86_64 (syseng6) Requires: openvswitch-kmod You say openvswitch-kmod, I say kmod-openvswitch, you say go F yourself. It's late, I've been working this for weeks. This is just the latest obstacle. Please help. The OpenVswitch people are few and unresponsive, by the way. Useless mailing lists. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] FixedIpNotFoundForAddress errors in network.log
I'm in the process of setting up a multi-node Folsom installation. Everything is going well so far, but I can't figure out why I keep getting these errors in network.log. There are literally hundreds of them, 2 or more per hour. The IP in this error, 10.21.165.121, is on our production network and is in now way affiliated with Openstack. In fact, all of our production IPs make it into this log eventually. Does anyone know what's causing this? My guess is that it has something to do with avahi, but I have no idea how to track down the actual problem. Thanks in advance. 2013-01-26 19:26:31 15905 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py, line 276, in _process_data 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args) 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py, line 145, in dispatch 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/network/manager.py, line 266, in wrapped 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp return func(self, context, *args, **kwargs) 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/network/manager.py, line 1801, in get_fixed_ip_by_address 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp fixed = self.db.fixed_ip_get_by_address(context, address) 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/db/api.py, line 477, in fixed_ip_get_by_address 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp return IMPL.fixed_ip_get_by_address(context, address) 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py, line 129, in wrapper 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs) 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py, line 1201, in fixed_ip_get_by_address 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp raise exception.FixedIpNotFoundForAddress(address=address) 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp FixedIpNotFoundForAddress: Fixed ip not found for address 10.21.165.121. 2013-01-26 19:26:31 15905 TRACE nova.openstack.common.rpc.amqp 2013-01-26 19:26:31 15905 ERROR nova.openstack.common.rpc.common [-] Returning exception Fixed ip not found for address 10.21.165.121. to caller 2013-01-26 19:26:31 15905 ERROR nova.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', ' File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py, line 276, in _process_data\nrval = self.proxy.dispatch(ctxt, version, method, **args)\n', ' File /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py, line 145, in dispatch\nreturn getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File /usr/lib/python2.6/site-packages/nova/network/manager.py, line 266, in wrapped\nreturn func(self, context, *args, **kwargs)\n', ' File /usr/lib/python2.6/site-packages/nova/network/manager.py, line 1801, in get_fixed_ip_by_address\nfixed = self.db.fixed_ip_get_by_address(context, address)\n', ' File /usr/lib/python2.6/site-packages/nova/db/api.py, line 477, in fixed_ip_get_by_address\nreturn IMPL.fixed_ip_get_by_address(context, address)\n', ' File /usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py, line 129, in wrapper\nreturn f(*args, **kwargs)\n', ' File /usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py, line 1201, in fixed_ip_get_by_address\nraise exception.FixedIpNotFoundForAddress(address=address)\n', 'FixedIpNotFoundForAddress: Fixed ip not found for address 10.21.165.121.\n'] -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Wasted Compute node disk space
We are in the process of replacing our Diablo KVM infrastructure with Folsom. Up until now, our virts have been using the local Compute node's disk space for their images which, obviously, defeats much of the purpose of a virtualizing. We are ready to implement an iSCSI SAN, but we're a bit bummed that we're going to essentially be wasting the copious disk space on all the systems we ordered to serve as Compute nodes. SO... we were' thinking about doubling up our Compute nodes and Swift storage nodes, fully aware that this might require us to reserve more cores for the KVM host. Has any one else tried this? Is this clearly a bad, bad, bad idea? Thanks. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Wasted Compute node disk space
Razique, Yes. We have the OS on a RAID 1 and the rest of the disks are in a RAID 10. However, should we go with the Compute+Swift Node architecture, we'll be using this: [root@kvm-cs-gen-09i ~]# df -h FilesystemSize Used Avail Use% Mounted on /dev/mapper/vg00-root 130G 1.9G 122G 2% / tmpfs 379G 0 379G 0% /dev/shm /dev/sda1 194M 33M 152M 18% /boot /dev/sdb1 559G 33M 559G 1% /swift1 /dev/sdc1 559G 33M 559G 1% /swift2 /dev/sdd1 559G 33M 559G 1% /swift3 /dev/sde1 559G 33M 559G 1% /swift4 /dev/sdf1 559G 33M 559G 1% /swift5 /dev/sdg1 559G 33M 559G 1% /swift6 /dev/sdh1 559G 33M 559G 1% /swift7 /dev/sdi1 559G 33M 559G 1% /swift8 What do you think? --Greg On Tue, Jan 8, 2013 at 11:04 AM, Razique Mahroua razique.mahr...@gmail.comwrote: Hey Greg, so if I understand well, you want to have the disk on the hypervisors be used as Swift nodes right? is there any underlying RAID? Regards, *Razique Mahroua** - **Nuage Co* razique.mahr...@gmail.com Tel : +33 9 72 37 94 15 Le 8 janv. 2013 à 16:28, Greg Chavez greg.cha...@gmail.com a écrit : We are in the process of replacing our Diablo KVM infrastructure with Folsom. Up until now, our virts have been using the local Compute node's disk space for their images which, obviously, defeats much of the purpose of a virtualizing. We are ready to implement an iSCSI SAN, but we're a bit bummed that we're going to essentially be wasting the copious disk space on all the systems we ordered to serve as Compute nodes. SO... we were' thinking about doubling up our Compute nodes and Swift storage nodes, fully aware that this might require us to reserve more cores for the KVM host. Has any one else tried this? Is this clearly a bad, bad, bad idea? Thanks. -- \*..+.- --Greg Chavez +//..;}; ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp NUAGECO-LOGO-Fblan_petit.jpg___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
