Re: [Openstack] Rebooted, now can't ping my guest
So...is this the root of my problem? The MAC addresses in my quantum port-list do not match my real MAC addresses. Any time I restart openvswitch/quantum-plugin-openvswitch-agent, my br-tun blocks my traffic with incorrect flows. What went wrong here...should these interfaces actually have fa:16:3e MAC addresses, or is the database wrong? root@os-network:~# quantum port-list +--+--+---+-+ | id | name | mac_address | fixed_ips | +--+--+---+-+ | 9f9041ce-654d-4706-a208-60cf5fca5d90 | | fa:16:3e:e2:38:da | {subnet_id: 0617c874-9a95-40dc-ae4f-bb44eec806f6, ip_address: 10.5.5.1} | | 28108125-119c-4ce4-a3a3-537639589791 | | fa:16:3e:a9:a3:4c | {subnet_id: cde12bce-eeed-4041-87e3-5a1b905b3c98, ip_address: 10.42.36.130} | | 45ffdc5f-dad9-444a-aff4-3d39b607f828 | | fa:16:3e:36:2e:54 | {subnet_id: 0617c874-9a95-40dc-ae4f-bb44eec806f6, ip_address: 10.5.5.2} | : +--+--+---+-+ root@os-network:~# ifconfig qr-9f9041ce-65 qr-9f9041ce-65 Link encap:Ethernet HWaddr 4e:bf:4a:70:a7:9e inet addr:10.5.5.1 Bcast:10.5.5.255 Mask:255.255.255.0 : root@os-network:~# ifconfig tap45ffdc5f-da tap45ffdc5f-da Link encap:Ethernet HWaddr 16:83:e6:27:a4:b8 inet addr:10.5.5.2 Bcast:10.5.5.255 Mask:255.255.255.0 : root@os-network:~# service openvswitch-switch restart : root@os-network:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=35.154s, table=0, n_packets=123, n_bytes=8526, priority=0 actions=NORMAL root@os-network:~# service quantum-plugin-openvswitch-agent restart quantum-plugin-openvswitch-agent stop/waiting quantum-plugin-openvswitch-agent start/running, process 1419 root@os-network:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=4.022s, table=0, n_packets=3, n_bytes=418, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=4.056s, table=0, n_packets=5, n_bytes=548, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=3.816s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=fa:16:3e:36:2e:54 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=3.987s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=fa:16:3e:e2:38:da actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=4.573s, table=0, n_packets=9, n_bytes=1828, priority=1 actions=drop root@os-network:~# ovs-ofctl add-flow br-tun priority=3,tun_id=0x1,dl_dst=16:83:e6:27:a4:b8 actions=mod_vlan_vid:1,NORMAL root@os-network:~# ovs-ofctl add-flow br-tun priority=3,tun_id=0x1,dl_dst=4e:bf:4a:70:a7:9e actions=mod_vlan_vid:1,NORMAL root@os-network:~# ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Rebooted, now can't ping my guest
Perhaps somebody could give me the contents of their quantum node's ovs-ofctl dump-flows br-tun and I could figure out what mine *should* look like? On Tue, Mar 12, 2013 at 4:24 PM, The King in Yellow yellowk...@gmail.comwrote: Okay, I have worked around my problem-- but I don't quite understand it, and hope somebody can help me. It appears to be a problem with the Open vSwitch flows in br-tun on both compute and network. Here are the flows as they are now, working. I have manually added the priority=5 lines. Without those added manually on both sides, traffic from the guest's doesn't work properly. root@os-network:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=5331.934s, table=0, n_packets=, n_bytes=171598, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=3.119s, table=0, n_packets=6, n_bytes=496, priority=5,dl_vlan=1 actions=NORMAL cookie=0x0, duration=10.759s, table=0, n_packets=0, n_bytes=0, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=5331.725s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=fa:16:3e:36:2e:54 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=5331.898s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=fa:16:3e:e2:38:da actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=5332.499s, table=0, n_packets=3502, n_bytes=286312, priority=1 actions=drop root@os-network:~# root@os-compute-01:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=22348.618s, table=0, n_packets=20165, n_bytes=991767, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=177.949s, table=0, n_packets=151, n_bytes=21830, priority=5,dl_vlan=1 actions=NORMAL cookie=0x0, duration=411.826s, table=0, n_packets=80, n_bytes=9566, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=22348.567s, table=0, n_packets=, n_bytes=252718, priority=3,tun_id=0x1,dl_dst=fa:16:3e:ee:9e:b2 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=22348.128s, table=0, n_packets=1107, n_bytes=123234, priority=3,tun_id=0x1,dl_dst=fa:16:3e:8d:6d:13 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=22348.353s, table=0, n_packets=1494, n_bytes=124036, priority=3,tun_id=0x1,dl_dst=fa:16:3e:95:94:9c actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=22347.912s, table=0, n_packets=3334, n_bytes=425776, priority=3,tun_id=0x1,dl_dst=fa:16:3e:7b:e3:ee actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=22349.47s, table=0, n_packets=879, n_bytes=75279, priority=1 actions=drop root@os-compute-01:~# Here is a sample packet that would have been blocked, sniffed in GRE. The yellow background (if you can see the color) is the GRE header. Inside the GRE payload, MAC (red) is1272.590f.cf56, and MAC (orange) is fa16.3e7b.e3ee are exchanging ping packets. 00 50 56 81 44 e7 00 50 56 81 25 73 08 00 45 00 .PV.D..PV.%s..E. 0010 00 82 64 93 40 00 40 2f ad a3 0a 0a 0a 02 0a 0a ..d.@.@/ 0020 0a 01 20 00 65 58 00 00 00 00 12 72 59 0f cf 56 .. .eX.rY..V 0030 fa 16 3e 95 94 9c 81 00 00 01 08 00 45 00 00 54 ...E..T 0040 00 00 40 00 40 01 1d 00 0a 05 05 04 0a 2a 04 77 ..@.@*.w 0050 08 00 00 9f 0e 17 00 08 17 87 3f 51 00 00 00 00 ..?Q 0060 d3 96 00 00 00 00 00 00 10 11 12 13 14 15 16 17 0070 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 !#$%' 0080 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 ()*+,-./01234567 00 50 56 81 25 73 00 50 56 81 44 e7 08 00 45 00 .PV.%s.PV.D...E. 0010 00 82 95 e3 40 00 40 2f 7c 53 0a 0a 0a 01 0a 0a @.@/|S.. 0020 0a 02 20 00 65 58 00 00 00 00 fa 16 3e 95 94 9c .. .eX. 0030 12 72 59 0f cf 56 81 00 00 01 08 00 45 00 00 54 .rY..V..E..T 0040 1e 24 00 00 3e 01 40 dc 0a 2a 04 77 0a 05 05 04 .$...@..*.w 0050 00 00 08 9f 0e 17 00 08 17 87 3f 51 00 00 00 00 ..?Q 0060 d3 96 00 00 00 00 00 00 10 11 12 13 14 15 16 17 0070 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 !#$%' 0080 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 ()*+,-./01234567 The source MAC does match the MAC for his gateway, 10.5.5.1: root@os-network:~# ovs-ofctl show br-int OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:862cf391d546 n_tables:255, n_buffers:256 features: capabilities:0xc7, actions:0xfff 1(qr-9f9041ce-65): addr:12:72:59:0f:cf:56 config: 0 state: 0 : ...which is the understandable problem. That MAC address is not specifically entered in the OVS bridge br-tun. Any clue why? This is across reboots, service restarts, etc... I guess it is the /etc/openswitch/conf.db that is corrupted? What I don't understand at this point is why removing the priority 5 flow on the compute
Re: [Openstack] question on the GRE Communication
That's the best way I found to see if GRE is up, just watching for two way proto gre traffic. Here's how you can match the IP addresses *inside* the GRE packet, which you probably will want. Note that 0x0a050505 is hexadecimal for my desired IP address of 10.5.5.5: root@os-network:~# tcpdump -i eth1 'proto gre and ( ip[58:4] = 0x0a050505 or ip[62:4] = 0x0a050505 )' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 16:18:17.434378 IP os-network-d os-compute-01-d: GREv0, key=0x0, length 110: IP opskzlp119.snops.net 10.5.5.5: ICMP echo request, id 21321, seq 488, length 64 16:18:17.436190 IP os-compute-01-d os-network-d: GREv0, key=0x0, length 110: IP 10.5.5.5 opskzlp119.snops.net: ICMP echo reply, id 21321, seq 488, length 64 16:18:18.435750 IP os-network-d os-compute-01-d: GREv0, key=0x0, length 110: IP opskzlp119.snops.net 10.5.5.5: ICMP echo request, id 21321, seq 489, length 64 16:18:18.437798 IP os-compute-01-d os-network-d: GREv0, key=0x0, length 110: IP 10.5.5.5 opskzlp119.snops.net: ICMP echo reply, id 21321, seq 489, length 64 : On Thu, Mar 14, 2013 at 9:55 AM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Thanks for the reply. I have one more question. How we will check whether the tunneling is established or not? tcpdump can show you the GRE traffic: tcpdump -i ethX proto gre Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Can't register for forums.openstack.org
If anybody from the forums is on here, or can forward this message to the forum admins, I have been trying for weeks to register. This is from multiple IP address ranges-- work, home, and cellular. I keep getting the following error: Your IP 96.60.255.159 has been blocked because it is blacklisted. For details please see http://search.atlbl.com/search.php?q=96.60.255.159. { IP_BLACKLISTED_INFO } search.atlbl.com goes to a parked website, so I'm guessing the registration blacklist is actually broken. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Rebooted, now can't ping my guest
A little more information, or a completely different problem...I just realized my network node can't ping its public gateway. (Instead of 7.7.7.0/24, I am using 10.42.36.0/23. My gateway is .1 (and has non-redundant IP of .10), and my Network Node IP is .130). This did work at one time. root@os-network:~# ping 10.42.36.1 PING 10.42.36.1 (10.42.36.1) 56(84) bytes of data. From 10.42.36.130 icmp_seq=1 Destination Host Unreachable From 10.42.36.130 icmp_seq=2 Destination Host Unreachable From 10.42.36.130 icmp_seq=3 Destination Host Unreachable ^C --- 10.42.36.1 ping statistics --- 5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4023ms pipe 3 root@os-network:~# arp -an ? (10.42.36.1) at incomplete on qg-28108125-11 ? (10.10.10.2) at 00:50:56:81:25:73 [ether] on eth1 ? (10.42.38.11) at 40:55:39:25:a5:c1 [ether] on eth3 ? (192.168.0.1) at 00:50:56:81:48:02 [ether] on eth0 ? (10.5.5.4) at fa:16:3e:95:94:9c [ether] on tap45ffdc5f-da ? (10.42.36.10) at incomplete on qg-28108125-11 ? (10.5.5.3) at fa:16:3e:8d:6d:13 [ether] on tap45ffdc5f-da ? (10.42.38.1) at 00:07:b4:01:b5:01 [ether] on eth3 root@os-network:~# Nor does it work the other way: From my gateway: sniktc-nx10A-7010-KXDC-E12# sho run int vlan402 !Command: show running-config interface Vlan402 !Time: Fri Mar 8 14:11:45 2013 version 5.1(3) interface Vlan402 vrf member Enterprise no ip redirects ip address 10.42.4.10/22 ip router eigrp 1 ip pim sparse-mode glbp 402 ip 10.42.4.1 priority 200 authentication text KXDCglbp402 no shutdown mtu 9216 sniktc-nx10A-7010-KXDC-E12# ping 10.42.36.130 vrf Enterprise source 10.42.36.10 PING 10.42.36.130 (10.42.36.130) from 10.42.36.10: 56 data bytes Request 0 timed out Request 1 timed out Request 2 timed out Request 3 timed out Request 4 timed out --- 10.42.36.130 ping statistics --- 5 packets transmitted, 0 packets received, 100.00% packet loss sniktc-nx10A-7010-KXDC-E12# root@os-network:~# ifconfig qg-28108125-11 qg-28108125-11 Link encap:Ethernet HWaddr fa:16:3e:a9:a3:4c inet addr:10.42.36.130 Bcast:10.42.37.255 Mask:255.255.254.0 inet6 addr: fe80::f816:3eff:fea9:a34c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:34 errors:0 dropped:0 overruns:0 frame:0 TX packets:354 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4840 (4.8 KB) TX bytes:22619 (22.6 KB) root@os-network:~# ovs-vsctl show e232f8c8-1cb8-4cf5-9de5-49f41e59fd38 Bridge br-tun Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Port gre-2 Interface gre-2 type: gre options: {in_key=flow, out_key=flow, remote_ip=10.10.10.2} Bridge br-ex Port qg-28108125-11 Interface qg-28108125-11 type: internal Port br-ex Interface br-ex type: internal Port eth2 Interface eth2 Bridge br-int Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port tap45ffdc5f-da tag: 1 Interface tap45ffdc5f-da type: internal Port qr-9f9041ce-65 tag: 1 Interface qr-9f9041ce-65 type: internal ovs_version: 1.4.0+build0 root@os-network:~# I suppose something has gone wrong inside quantum on the network node? Is there a good way to blow everything away and rebuild? I tried the following, but the quantum-networking script doesn't really work this way, as it assumes it is creating items (and getting the uuids in return), which doesn't work the second time through. ovs-vsctl del-br br-int ovs-vsctl del-br br-ex service openvswitch-switch restart ovs-vsctl add-br br-int ovs-vsctl add-br br-ex ovs-vsctl add-port br-ex eth2 ip link set up br-ex service quantum-plugin-openvswitch-agent restart service quantum-dhcp-agent restart service quantum-l3-agent restart sh ./quantum-networking-filled.sh service quantum-l3-agent restart Here is what quantum looks like right now-- note the additional provider-routers, I'm guessing from failed runs of the quantum-networking script. Note also ...771's external gateway of ...3de0, which isn't showing up anywhere...could that be the problem? Should that be subnet ...06f6? root@os-network:~# quantum subnet-list +--+--+---+--+ | id | name | cidr | allocation_pools |
Re: [Openstack] Rebooted, now can't ping my guest
On Wed, Mar 6, 2013 at 10:15 AM, Sylvain Bauza sylvain.ba...@digimind.comwrote: Le 05/03/2013 18:14, The King in Yellow a écrit : I'm not clear on what the interfaces are, but q-9f9041ce-65 is 10.5.5.1 on the network node, so he seems to be seeing the traffic. tap45ffdc5f-da is listed as 10.5.5.2, and I have no idea what function that is performing. qr-X is the internal router IP interface (as you correctly guessed, ie. 10.5.5.1) and bound to br-int. tap- is the DHCP server IP interface (ie. 10.5.5.2), also bound to br-int. 'brctl show' gives you the output. Could you please provide us route -n on the network node ? os@os-network:~$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 0.0.0.0 10.42.36.1 0.0.0.0 UG0 00 qg-28108125-11 0.0.0.0 10.42.38.1 0.0.0.0 UG0 00 eth3 10.5.5.00.0.0.0 255.255.255.0 U 0 00 tap45ffdc5f-da 10.5.5.00.0.0.0 255.255.255.0 U 0 00 qr-9f9041ce-65 10.10.10.0 0.0.0.0 255.255.255.0 U 0 00 eth1 10.42.36.0 0.0.0.0 255.255.254.0 U 0 00 qg-28108125-11 10.42.38.0 0.0.0.0 255.255.254.0 U 1 00 eth3 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 00 eth0 192.168.0.0 0.0.0.0 255.255.255.0 U 0 00 eth0 os@os-network:~$ Could you also make sure (with 'ovs-vsctl show') that port 'qr-' and 'tap' do have the same tag number as for VMs ? (should be tag:1) They are: root@os-network:~# ovs-vsctl show e232f8c8-1cb8-4cf5-9de5-49f41e59fd38 Bridge br-tun Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port gre-2 Interface gre-2 type: gre options: {in_key=flow, out_key=flow, remote_ip=10.10.10.2} Bridge br-int Port br-int Interface br-int type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port qr-9f9041ce-65 tag: 1 Interface qr-9f9041ce-65 type: internal Port tap45ffdc5f-da tag: 1 Interface tap45ffdc5f-da type: internal Bridge br-ex Port qg-28108125-11 Interface qg-28108125-11 type: internal Port eth2 Interface eth2 Port br-ex Interface br-ex type: internal ovs_version: 1.4.0+build0 root@os-network:~# ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Rebooted, now can't ping my guest
That didn't quite do it. Rebooted 10.5.5.5/6 and they did not get IPs. Brought one up manually and could not ping anything else. I note that I'm missing the tag statement on those recreated interfaces in ovs-vsctl show, so I deleted the interfaces and reran the statements you gave with tag=1 appended. Now, my manually configured 10.5.5.5 COULD ping my working 10.5.5.7, and I could ssh between the two. However, 10.5.5.5 still can not get a DHCP address or (with hardcoded IP) reach 10.5.5.1 on the network node, whereas 10.5.5.7 can. Here's how things look now: root@os-compute-01:~# ovs-dpctl show br-int system@br-int: lookups: hit:236399 missed:45742 lost:0 flows: 1 port 0: br-int (internal) port 2: qvo7dcd14b3-70 port 9: qvo0b459c65-a0 port 10: qvo4f36c3ea-5c port 11: qvo62721ee8-08 port 12: qvocf833d2a-9e port 13: patch-tun (patch: peer=patch-int) root@os-compute-01:~# ovs-vsctl show 3a52a17f-9846-4b32-b309-b49faf91bfc4 Bridge br-int Port br-int Interface br-int type: internal Port qvo0b459c65-a0 tag: 1 Interface qvo0b459c65-a0 Port qvo62721ee8-08 tag: 1 Interface qvo62721ee8-08 Port qvo4f36c3ea-5c tag: 1 Interface qvo4f36c3ea-5c Port qvocf833d2a-9e tag: 1 Interface qvocf833d2a-9e Port qvo7dcd14b3-70 tag: 1 Interface qvo7dcd14b3-70 Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Bridge br-tun Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Port gre-1 Interface gre-1 type: gre options: {in_key=flow, out_key=flow, remote_ip=10.10.10.1} ovs_version: 1.4.0+build0 root@os-compute-01:~# brctl show bridge name bridge id STP enabled interfaces br-int .222603554b47 no qvo0b459c65-a0 qvo4f36c3ea-5c qvo62721ee8-08 qvo7dcd14b3-70 qvocf833d2a-9e br-tun .3abeb87cdb47 no qbr0b459c65-a0 8000.3af05347af11 no qvb0b459c65-a0 vnet2 qbr4f36c3ea-5c 8000.e6a5faf9a181 no qvb4f36c3ea-5c vnet1 qbr62721ee8-08 8000.8af675d45ed7 no qvb62721ee8-08 vnet0 qbr7dcd14b3-70 8000.aabc605c1b2c no qvb7dcd14b3-70 vnet4 qbrcf833d2a-9e 8000.36e77dfc6018 no qvbcf833d2a-9e vnet3 root@os-compute-01:~# On Tue, Mar 5, 2013 at 8:02 AM, Sylvain Bauza sylvain.ba...@digimind.comwrote: You get it. This is the bug I mentioned related to compute nodes. Folks, anyone knowing the bug tracking numbre, btw ? 'ovs-dpctl show' shows you that only qvo7dcd14b3-70 is bridged to br-int (and mapped to vnet4, which I guess is the vnet device for the correct VM). Could you please try : sudo ovs-vsctl add-port br-int qvo0b459c65-a0 sudo ovs-vsctl add-port br-int qvo4f36c3ea-5c sudo ovs-vsctl add-port br-int qvo62721ee8-08 sudo ovs-vsctl add-port br-int qvocf833d2a-9e sudo service quantum-plugin-openvswitch-agent restart and check that your VMs get network back ? -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Rebooted, now can't ping my guest
On Tue, Mar 5, 2013 at 11:19 AM, Sylvain Bauza sylvain.ba...@digimind.comwrote: You should be close to the solution. Looking at your GRE tunnels, I only see a one-to-one tunnel in between your compute node and your network node (provided your netnode is 10.10.10.1). Could you please confirm that your controller is either on the compute node or on the network node ? My network node is 10.10.10.1. My controller is an independent node, and not on the 10.10.10.x network (following this architecturehttp://docs.openstack.org/folsom/basic-install/content/basic-install_requirements.htmlalthough I have changed the external subnets). One could suggest to restart nova-compute and check. I tried, and it did nothing. I think something else might be going on...I do seem to be getting the correct ARP entries for 10.5.5.5 on my network node: root@os-network:/var/log/quantum# arp -an ? (10.5.5.4) at incomplete on tap45ffdc5f-da ? (10.5.5.3) at incomplete on tap45ffdc5f-da ? (10.10.10.2) at 00:50:56:81:25:73 [ether] on eth1 ? (192.168.0.1) at 00:50:56:81:48:02 [ether] on eth0 ? (10.5.5.7) at fa:16:3e:06:17:3c [ether] on tap45ffdc5f-da ? (10.5.5.5) at fa:16:3e:8d:2c:f9 [ether] on qr-9f9041ce-65 ? (10.5.5.5) at fa:16:3e:8d:2c:f9 [ether] on tap45ffdc5f-da ? (10.42.38.1) at 00:07:b4:01:b5:01 [ether] on eth3 root@os-network:/var/log/quantum# arp -i qr-9f9041ce-65 -d 10.5.5.5 root@os-network:/var/log/quantum# arp -i tap45ffdc5f-da -d 10.5.5.5 root@os-network:/var/log/quantum# arp -an ? (10.5.5.4) at incomplete on tap45ffdc5f-da ? (10.5.5.3) at incomplete on tap45ffdc5f-da ? (10.10.10.2) at 00:50:56:81:25:73 [ether] on eth1 ? (192.168.0.1) at 00:50:56:81:48:02 [ether] on eth0 ? (10.5.5.7) at fa:16:3e:06:17:3c [ether] on tap45ffdc5f-da ? (10.5.5.5) at fa:16:3e:8d:2c:f9 [ether] on tap45ffdc5f-da ? (10.42.38.1) at 00:07:b4:01:b5:01 [ether] on eth3 root@os-network:/var/log/quantum# arp -an ? (10.5.5.4) at incomplete on tap45ffdc5f-da ? (10.5.5.3) at incomplete on tap45ffdc5f-da ? (10.10.10.2) at 00:50:56:81:25:73 [ether] on eth1 ? (192.168.0.1) at 00:50:56:81:48:02 [ether] on eth0 ? (10.5.5.7) at fa:16:3e:06:17:3c [ether] on tap45ffdc5f-da ? (10.5.5.5) at fa:16:3e:8d:2c:f9 [ether] on qr-9f9041ce-65 ? (10.5.5.5) at fa:16:3e:8d:2c:f9 [ether] on tap45ffdc5f-da ? (10.42.38.1) at 00:07:b4:01:b5:01 [ether] on eth3 root@os-network:/var/log/quantum# I'm not clear on what the interfaces are, but q-9f9041ce-65 is 10.5.5.1 on the network node, so he seems to be seeing the traffic. tap45ffdc5f-da is listed as 10.5.5.2, and I have no idea what function that is performing. root@os-network:/var/log/quantum# ping 10.5.5.7 PING 10.5.5.7 (10.5.5.7) 56(84) bytes of data. 64 bytes from 10.5.5.7: icmp_req=1 ttl=64 time=1.93 ms 64 bytes from 10.5.5.7: icmp_req=2 ttl=64 time=2.08 ms ^C --- 10.5.5.7 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 1.931/2.007/2.083/0.076 ms root@os-network:/var/log/quantum# arp -an ? (10.5.5.4) at fa:16:3e:e0:17:f0 [ether] on tap45ffdc5f-da ? (10.5.5.3) at incomplete on tap45ffdc5f-da ? (10.10.10.2) at 00:50:56:81:25:73 [ether] on eth1 ? (192.168.0.1) at 00:50:56:81:48:02 [ether] on eth0 ? (10.5.5.7) at fa:16:3e:06:17:3c [ether] on tap45ffdc5f-da ? (10.5.5.5) at fa:16:3e:8d:2c:f9 [ether] on tap45ffdc5f-da ? (10.42.38.1) at 00:07:b4:01:b5:01 [ether] on eth3 root@os-network:/var/log/quantum# ping 10.5.5.5 PING 10.5.5.5 (10.5.5.5) 56(84) bytes of data. ^C --- 10.5.5.5 ping statistics --- 9 packets transmitted, 0 received, 100% packet loss, time 8062ms root@os-network:/var/log/quantum# Also, could you please tcpdump your network node on your management IP and check if you see GRE packets coming from your compute node (while pinging or trying to get a lease) ? Threw a sniff up at http://pastebin.com/giwZysxW. There were 4 pings from 10.5.5.7 (starting line 47), followed by 4 pings from 10.5.5.5. Interesting to see the 10.5.5.3 and .4 references...I don't have passwords for those images (sshed in with the keys), so I rebooted them while sniffing here: http://pastebin.com/xpbgnhxu The network node ARP table never populated with .3 or .4, either. It looks like quantum-openvswitch-agent is started: root@os-compute-01:~# ps -ef | egrep quantum | egrep -v grep quantum 11504 1 1 09:27 ?00:01:44 python /usr/bin/quantum-openvswitch-agent --config-file=/etc/quantum/quantum.conf --config-file=/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini --log-file=/var/log/quantum/openvswitch-agent.log root@os-compute-01:~# ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Fwd: Rebooted, now can't ping my guest
Sorry, replied directly: -- Forwarded message -- From: The King in Yellow yellowk...@gmail.com Date: Tue, Mar 5, 2013 at 12:56 PM Subject: Re: [Openstack] Rebooted, now can't ping my guest To: Sylvain Bauza sylvain.ba...@digimind.com In fact, when I ping 10.5.5.2, tcpdump on the network node actually shows responses... root@os-network:/var/log/quantum# tcpdump -i tap45ffdc5f-da host 10.5.5.5 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tap45ffdc5f-da, link-type EN10MB (Ethernet), capture size 65535 bytes : 12:50:12.936292 ARP, Request who-has os-network.local tell 10.5.5.5, length 28 12:50:12.936329 ARP, Reply os-network.local is-at fa:16:3e:36:2e:54 (oui Unknown), length 28 12:50:14.936338 ARP, Request who-has os-network.local tell 10.5.5.5, length 28 12:50:14.936378 ARP, Reply os-network.local is-at fa:16:3e:36:2e:54 (oui Unknown), length 28 12:50:15.936299 ARP, Request who-has os-network.local tell 10.5.5.5, length 28 12:50:15.936343 ARP, Reply os-network.local is-at fa:16:3e:36:2e:54 (oui Unknown), length 28 12:50:16.936008 ARP, Request who-has os-network.local tell 10.5.5.5, length 28 12:50:16.936045 ARP, Reply os-network.local is-at fa:16:3e:36:2e:54 (oui Unknown), length 28 above, pinging 10.5.5.1 ; below, pinging 10.5.5.2 12:50:17.044161 IP 10.5.5.5 os-network.local: ICMP echo request, id 18433, seq 0, length 64 12:50:17.044222 IP os-network.local 10.5.5.5: ICMP echo reply, id 18433, seq 0, length 64 12:50:18.044399 IP 10.5.5.5 os-network.local: ICMP echo request, id 18433, seq 1, length 64 12:50:18.044455 IP os-network.local 10.5.5.5: ICMP echo reply, id 18433, seq 1, length 64 12:50:19.044847 IP 10.5.5.5 os-network.local: ICMP echo request, id 18433, seq 2, length 64 12:50:19.044899 IP os-network.local 10.5.5.5: ICMP echo reply, id 18433, seq 2, length 64 12:50:20.045370 IP 10.5.5.5 os-network.local: ICMP echo request, id 18433, seq 3, length 64 12:50:20.045427 IP os-network.local 10.5.5.5: ICMP echo reply, id 18433, seq 3 Unfortunately, the CirrOS image 10.5.5.5 is using doesn't seem to have the arp utility, nor does 'ip -n' work. I guess it isn't getting the ARP replies for 10.5.5.1, and somehow got an ARP entry for 10.5.5.2 and isn't seeing the echo replies. Seems to imply some sort of issue from the network node back to him? Found ARP entries via cat /proc/1/net/arp, which seems to confirm that-- entry for 10.5.5.1 is 00:00:00:00:00:00. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Rebooted, now can't ping my guest
On Mon, Mar 4, 2013 at 3:18 AM, Sylvain Bauza sylvain.ba...@digimind.comwrote: Is the network node also acting as a Compute node ? No, I am running three separate nodes-- network, compute and controller. The issue you were mentioning was related to the tap virtual device (for DHCP leases) : if the network node goes down, then the DHCP lease is expiring on the vm without being reack, and then your instance is loosing its IP address. By recreating the bridges upon reboot on the network node, the tap interface will be back up. On the VMs, only a DHCP request is enough, not a reboot (or even a compute node reboot). I know there is also a second bug related to virtio bridges on the compute nodes. This is still a bit unclear to me, but upon compute node reboot, virtio bridges are also not reattached, only new instances created afterwards. Could you please run 'ovs-dpctl show br-int' (provided br-int is the right bridge), 'ovs-vsctl show' and 'brctl show' ? This is on the compute node, where I assume the issue is. For the record, I have five vms running here-- four created before rebuilding the networking, and one after. Only the one after is working. root@os-compute-01:/var/log# ovs-dpctl show br-int system@br-int: lookups: hit:235944 missed:33169 lost:0 flows: 0 port 0: br-int (internal) port 1: patch-tun (patch: peer=patch-int) port 2: qvo7dcd14b3-70 root@os-compute-01:/var/log# ovs-vsctl show 3a52a17f-9846-4b32-b309-b49faf91bfc4 Bridge br-tun Port br-tun Interface br-tun type: internal Port gre-1 Interface gre-1 type: gre options: {in_key=flow, out_key=flow, remote_ip=10.10.10.1} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-int Port br-int Interface br-int type: internal Port qvo7dcd14b3-70 tag: 1 Interface qvo7dcd14b3-70 Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} ovs_version: 1.4.0+build0 root@os-compute-01:/var/log# brctl show bridge name bridge id STP enabled interfaces br-int .222603554b47 no qvo7dcd14b3-70 br-tun .36c126165e42 no qbr0b459c65-a0 8000.3af05347af11 no qvb0b459c65-a0 vnet2 qbr4f36c3ea-5c 8000.e6a5faf9a181 no qvb4f36c3ea-5c vnet1 qbr62721ee8-08 8000.8af675d45ed7 no qvb62721ee8-08 vnet0 qbr7dcd14b3-70 8000.aabc605c1b2c no qvb7dcd14b3-70 vnet4 qbrcf833d2a-9e 8000.36e77dfc6018 no qvbcf833d2a-9e vnet3 root@os-compute-01:/var/log# Thank you for the assistance! Lot of new stuff here I'm trying to come up to speed on. Le 01/03/2013 21:28, The King in Yellow a écrit : On Fri, Mar 1, 2013 at 10:11 AM, Sylvain Bauza sylvain.ba...@digimind.com wrote: There is a known bug for the network bridges, when rebooting : https://bugs.launchpad.net/quantum/+bug/1091605 Try to delete/recreate your br-int/br-ex and then restart openvswitch_plugin/l3/dhcp agents, it should fix the issue. Thanks! Now, I can create a new instance, and that works. My previous instances don't work, however. What do I need to do to get them reattached? root@os-network:/var/log/quantum# ping 10.5.5.6 PING 10.5.5.6 (10.5.5.6) 56(84) bytes of data. ^C --- 10.5.5.6 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1008ms root@os-network:/var/log/quantum# ping 10.5.5.7 PING 10.5.5.7 (10.5.5.7) 56(84) bytes of data. 64 bytes from 10.5.5.7: icmp_req=1 ttl=64 time=2.13 ms 64 bytes from 10.5.5.7: icmp_req=2 ttl=64 time=1.69 ms 64 bytes from 10.5.5.7: icmp_req=3 ttl=64 time=1.93 ms 64 bytes from 10.5.5.7: icmp_req=4 ttl=64 time=1.01 ms ^C --- 10.5.5.7 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms rtt min/avg/max/mdev = 1.013/1.692/2.132/0.424 ms root@os-network:/var/log/quantum# ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Rebooted, now can't ping my guest
In my case, it actually appears that my vms aren't up-- the instances panel says they are up, but looking at the console, it appears they aren't getting an IP address. This is a new instance: Begin: Running /scripts/init-bottom ... done. [2.849416] EXT4-fs (vda1): re-mounted. Opts: (null) cloud-init start-local running: Thu, 28 Feb 2013 13:29:09 +. up 10.41 seconds no instance data found in start-local cloud-init-nonet waiting 120 seconds for a network device. cloud-init-nonet gave up waiting for a network device. ci-info: lo: 1 127.0.0.1 255.0.0.0 . ci-info: eth0 : 1 . . fa:16:3e:e0:17:f0 route_info failed Waiting for network configuration... It looks like it made an OVS port, though. This is on the compute node, openvswitch-agent.log: 2013-02-28 08:34:19DEBUG [quantum.agent.linux.utils] Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvo4f36c3ea-5c', 'external_ids'] Exit code: 0 Stdout: '{attached-mac=fa:16:3e:e0:17:f0, iface-id=4f36c3ea-5c49-4625-a830-0c81f27ba139, iface-status=active, vm-uuid=239d3051-255e-4213-9511-af0a82fcc744}\n' Stderr: '' 2013-02-28 08:34:19DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface qvo62721ee8-08 external_ids 2013-02-28 08:34:19DEBUG [quantum.agent.linux.utils] : root@os-compute-01:/var/log/quantum# ovs-vsctl show 3a52a17f-9846-4b32-b309-b49faf91bfc4 Bridge br-int Port qvo62721ee8-08 tag: 1 Interface qvo62721ee8-08 Port qvo1ed73bcc-9d tag: 1 Interface qvo1ed73bcc-9d Port qvoce0c94a9-ef tag: 1 Interface qvoce0c94a9-ef Port qvo135e78dd-8e tag: 4095 Interface qvo135e78dd-8e Port qvof37b7a55-a3 tag: 1 Interface qvof37b7a55-a3 Port br-int Interface br-int type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port qvoaed25b41-9c tag: 1 Interface qvoaed25b41-9c Port qvo4f36c3ea-5c tag: 1 Interface qvo4f36c3ea-5c Bridge br-tun Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port gre-1 Interface gre-1 type: gre options: {in_key=flow, out_key=flow, remote_ip=10.10.10.1} Port br-tun Interface br-tun type: internal ovs_version: 1.4.0+build0 root@os-compute-01:/var/log/quantum# I supposed it should be getting address via DHCP from quantum-dhcp-agent on the network node? It was running, nothing regarding this MAC in the logs. I restarted quantum-dhcp-agent and rebooted, no change. In fact, I got two CirrOS vms up, logged on the console and manually IPed them (10.5.5.10/24 and 10.5.5.11/24), and they can't ping each other. I would expect them to, right? They should both be connected to OVS switch br-int, right? Any pointers? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Rebooted, now can't ping my guest
I have been working on creating an OpenStack environment according to the Basic Install http://docs.openstack.org/folsom/basic-install/content/index.htmldoc. It was working fine last night! In order to make sure I didn't mess anything up, I downed controller/network/compute nodes and cloned them (they are nested on ESXi 5.0u1). Upon coming back up, I can't ping my guests. I'm on the network node, pinging 10.5.5.3, which is a running guest. I'm guessing the GRE tunnel isn't coming between the compute and network node, since the br-* interfaces down? (After this, I manually ip link set up all br-* interfaces on both compute and network-- nothing) I have no experience with either Quantum or Open vSwitch, so I don't know what this is telling me. I'm rather at a loss-- can anybody point me in the right direction here? I don't see anything in the quantum logs right now that seems to indicate an error-- openvswitch-agent.log is cycling through things like the following, though: 2013-02-27 18:19:43DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface qr-9f9041ce-65 external_ids 2013-02-27 18:19:43DEBUG [quantum.agent.linux.utils] Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qr-9f9041ce-65', 'external_ids'] Exit code: 0 Stdout: '{attached-mac=fa:16:3e:e2:38:da, iface-id=9f9041ce-654d-4706-a208-60cf5fca5d90, iface-status=active}\n' Stderr: '' 2013-02-27 18:19:43DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface tap45ffdc5f-da external_ids 2013-02-27 18:19:43DEBUG [quantum.agent.linux.utils] Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'tap45ffdc5f-da', 'external_ids'] Exit code: 0 Stdout: '{attached-mac=fa:16:3e:36:2e:54, iface-id=45ffdc5f-dad9-444a-aff4-3d39b607f828, iface-status=active}\n' Stderr: '' 2013-02-27 18:19:45DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 list-ports br-int 2013-02-27 18:19:45DEBUG [quantum.agent.linux.utils] Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int'] Exit code: 0 Stdout: 'patch-tun\nqr-9f9041ce-65\ntap45ffdc5f-da\n' Stderr: '' 2013-02-27 18:19:45DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface patch-tun external_ids 2013-02-27 18:19:45DEBUG [quantum.agent.linux.utils] Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'patch-tun', 'external_ids'] Exit code: 0 Stdout: '{}\n' Stderr: '' Here are the output of ifconfig -a, ovs-vsctl show, and ovs-cfctl of each bridge on the network node: root@os-network:~# ifconfig -a br-ex Link encap:Ethernet HWaddr 00:50:56:81:66:d8 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:23 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1380 (1.3 KB) TX bytes:0 (0.0 B) br-intLink encap:Ethernet HWaddr 5e:5a:c3:07:44:42 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) br-tunLink encap:Ethernet HWaddr 56:2d:9f:6c:ac:4f BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr 00:50:56:81:28:f4 inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::250:56ff:fe81:28f4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:535 errors:0 dropped:10 overruns:0 frame:0 TX packets:554 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:137612 (137.6 KB) TX bytes:108783 (108.7 KB) eth1 Link encap:Ethernet HWaddr 00:50:56:81:44:e7 inet addr:10.10.10.1 Bcast:10.10.10.255 Mask:255.255.255.0 inet6 addr: fe80::250:56ff:fe81:44e7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:76 errors:0 dropped:9 overruns:0 frame:0 TX packets:40 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:14531 (14.5 KB) TX