** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1263338
Title: neutron (devstack) network connections to launched instances fail Status in OpenStack Neutron (virtual network service): Invalid Bug description: When running neutron on devstack, I run into the issue where I can successfully launch an instance, but cannot connect to it, not even from the host running devstack. Commands like 'ping' and 'ssh' appear to get no response at all. Steps to reproduce: ./stack.sh export OS_USERNAME=admin export OS_PASSWORD=password export OS_TENANT_NAME=demo export OS_AUTH_URL=http://192.168.126.142:5000/v2.0/ source /usr/local/src/devstack/openrc admin nova secgroup-add-rule default tcp 22 22 0.0.0.0/0 nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0 wget https://launchpadlibrarian.net/83303699/cirros-0.3.0-i386-disk.img glance image-create --name=cirros-0.3.0-i386 --is-public=true --container-format=bare --disk-format=qcow2 < cirros-0.3.0-i386-disk.img nova boot --flavor m1.nano --image cirros-0.3.0-i386 myvm nova show myvm +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | localhost.localdomain | | OS-EXT-SRV-ATTR:hypervisor_hostname | localhost.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000001 | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2013-12-21T14:19:29.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2013-12-21T14:19:22Z | | flavor | m1.nano (42) | | hostId | d751bc233fcbbf6622883cbe4ccfee1b1b41107c693836dc0b3666b1 | | id | 5ba5e118-47ba-4791-be59-309eb6405dff | | image | cirros-0.3.0-i386 (2c9602d1-b3b4-436e-bd92-532b3b03e541) | | key_name | None | | metadata | {} | | name | myvm | | os-extended-volumes:volumes_attached | [] | | private network | 10.0.0.3 | | progress | 0 | | security_groups | default | | status | ACTIVE | | tenant_id | 2c79cf76ab5f488388de486a586aa23f | | updated | 2013-12-21T14:19:29Z | | user_id | 4f3c34c88167426baa94e20b26ccac8b | +--------------------------------------+----------------------------------------------------------+ ping 10.0.0.3 PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data. ^C --- 10.0.0.3 ping statistics --- 8 packets transmitted, 0 received, 100% packet loss, time 7008ms ssh 10.0.0.3 ^C nova secgroup-list +--------------------------------------+---------+-------------+ | Id | Name | Description | +--------------------------------------+---------+-------------+ | 9e35010a-33b5-4437-8580-384975ce75c5 | default | default | +--------------------------------------+---------+-------------+ nova secgroup-list-rules default +-------------+-----------+---------+-----------+--------------+ | IP Protocol | From Port | To Port | IP Range | Source Group | +-------------+-----------+---------+-----------+--------------+ | icmp | -1 | -1 | 0.0.0.0/0 | | | | | | | default | | tcp | 22 | 22 | 0.0.0.0/0 | | | | | | | default | +-------------+-----------+---------+-----------+--------------+ But this works : sudo ip netns exec qdhcp-fc342f8f-0211-4e86-9a2f-ec64c719ba67 ping 10.0.0.3 PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data. 64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=1.26 ms 64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.408 ms ^C --- 10.0.0.3 ping statistics --- 3 packets transmitted, 2 received, 33% packet loss, time 2001ms rtt min/avg/max/mdev = 0.408/0.836/1.264/0.428 ms sudo ip netns exec qrouter-4d26556c-9b29-4162-ba91-ce704b771fa6 ping 10.0.0.3 PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data. 64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=1.98 ms 64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.441 ms ^C --- 10.0.0.3 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.441/1.213/1.986/0.773 ms Here is the rest of the configuration. ifconfig -a br-ex: flags=67<UP,BROADCAST,RUNNING> mtu 1500 inet 172.24.4.225 netmask 255.255.255.128 broadcast 0.0.0.0 ether a2:8f:9a:28:63:4f txqueuelen 0 (Ethernet) RX packets 12 bytes 976 (976.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3 bytes 270 (270.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 br-int: flags=67<UP,BROADCAST,RUNNING> mtu 1500 inet6 fe80::80bb:edff:fe2d:adc6 prefixlen 64 scopeid 0x20<link> ether be:88:dc:b0:e1:44 txqueuelen 0 (Ethernet) RX packets 35 bytes 3258 (3.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 83757 bytes 73811498 (70.3 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 83757 bytes 73811498 (70.3 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ovs-system: flags=4098<BROADCAST,MULTICAST> mtu 1500 ether 0a:ea:20:c3:4a:d5 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 p3p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.126.142 netmask 255.255.255.0 broadcast 192.168.126.255 inet6 fe80::20c:29ff:fe6e:32be prefixlen 64 scopeid 0x20<link> ether 00:0c:29:6e:32:be txqueuelen 1000 (Ethernet) RX packets 78553 bytes 71897578 (68.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 79025 bytes 10351867 (9.8 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 19 base 0x2000 tap256fc450-51: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::fc16:3eff:fefa:4513 prefixlen 64 scopeid 0x20<link> ether fe:16:3e:fa:45:13 txqueuelen 500 (Ethernet) RX packets 44 bytes 4285 (4.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 46 bytes 4887 (4.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 192.168.126.2 0.0.0.0 UG 0 0 0 p3p1 172.24.4.128 * 255.255.255.128 U 0 0 0 br-ex 192.168.126.0 * 255.255.255.0 U 1 0 0 p3p1 ip netns list qrouter-3eeb65d2-307a-496b-aa90-035bdaa77c5f qdhcp-1c0789cd-0148-44a5-9297-c3e5a16e1514 ovs-vsctl show ace3db50-ed2e-44ec-81aa-f427cc26a394 Bridge br-ex Port "qg-7a28824e-75" Interface "qg-7a28824e-75" type: internal Port br-ex Interface br-ex type: internal Bridge br-int Port "tap256fc450-51" tag: 1 Interface "tap256fc450-51" Port "tap201dde18-1e" tag: 1 Interface "tap201dde18-1e" type: internal Port br-int Interface br-int type: internal Port "qr-74a237e6-76" tag: 1 Interface "qr-74a237e6-76" type: internal ovs_version: "1.11.0" sudo brctl show bridge name bridge id STP enabled interfaces arp -a ? (192.168.126.1) at 00:50:56:c0:00:08 [ether] on p3p1 ? (192.168.126.2) at 00:50:56:e6:61:ac [ether] on p3p1 ? (192.168.126.254) at 00:50:56:fc:bc:9a [ether] on p3p1 When I ping the instance (10.0.0.3) from the host I run openstack on (localhost.localdomain, 192.168.126.142) I dont see any arp request/reply packets on any of the interfaces: sudo tcpdump -vv -i br-int arp tcpdump: WARNING: br-int: no IPv4 address assigned tcpdump: listening on br-int, link-type EN10MB (Ethernet), capture size 65535 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel sudo tcpdump -vv -i br-ex arp tcpdump: listening on br-ex, link-type EN10MB (Ethernet), capture size 65535 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel sudo tcpdump -vv -i tap256fc450-51 arp tcpdump: WARNING: tap256fc450-51: no IPv4 address assigned tcpdump: listening on tap256fc450-51, link-type EN10MB (Ethernet), capture size 65535 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel And this is my local.conf cat local.conf [[local|localrc]] ADMIN_PASSWORD=password MYSQL_PASSWORD=password RABBIT_PASSWORD=password SERVICE_PASSWORD=password SERVICE_TOKEN=tokentoken RECLONE=yes LOGFILE=$DEST/logs/stack.sh.log SCREEN_LOGDIR=$DEST/logs/screen LOGDAYS=1 #VERBOSE=True HOST_IP_IFACE=p3p1 PUBLIC_INTERFACE=p3p1 VLAN_INTERFACE=p3p1 FLAT_INTERFACE=p3p1 HOST_IP=192.168.126.142 FIXED_RANGE=10.0.0.0/24 FIXED_NETWORK_SIZE=254 FLOATING_RANGE=192.168.42.128/25 disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron enable_service q-lbaas [[post-config|$NOVA_CONF]] [DEFAULT] debug = False And the firewall rules # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination neutron-openvswi-INPUT all -- anywhere anywhere nova-api-INPUT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere INPUT_direct all -- anywhere anywhere INPUT_ZONES_SOURCE all -- anywhere anywhere INPUT_ZONES all -- anywhere anywhere ACCEPT icmp -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-openvswi-FORWARD all -- anywhere anywhere nova-filter-top all -- anywhere anywhere nova-api-FORWARD all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere FORWARD_direct all -- anywhere anywhere FORWARD_IN_ZONES_SOURCE all -- anywhere anywhere FORWARD_IN_ZONES all -- anywhere anywhere FORWARD_OUT_ZONES_SOURCE all -- anywhere anywhere FORWARD_OUT_ZONES all -- anywhere anywhere ACCEPT icmp -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-openvswi-OUTPUT all -- anywhere anywhere nova-filter-top all -- anywhere anywhere nova-api-OUTPUT all -- anywhere anywhere OUTPUT_direct all -- anywhere anywhere Chain FORWARD_IN_ZONES (1 references) target prot opt source destination FWDI_public all -- anywhere anywhere FWDI_public all -- anywhere anywhere Chain FORWARD_IN_ZONES_SOURCE (1 references) target prot opt source destination Chain FORWARD_OUT_ZONES (1 references) target prot opt source destination FWDO_public all -- anywhere anywhere FWDO_public all -- anywhere anywhere Chain FORWARD_OUT_ZONES_SOURCE (1 references) target prot opt source destination Chain FORWARD_direct (1 references) target prot opt source destination Chain FWDI_public (2 references) target prot opt source destination FWDI_public_log all -- anywhere anywhere FWDI_public_deny all -- anywhere anywhere FWDI_public_allow all -- anywhere anywhere Chain FWDI_public_allow (1 references) target prot opt source destination Chain FWDI_public_deny (1 references) target prot opt source destination Chain FWDI_public_log (1 references) target prot opt source destination Chain FWDO_external (0 references) target prot opt source destination FWDO_external_log all -- anywhere anywhere FWDO_external_deny all -- anywhere anywhere FWDO_external_allow all -- anywhere anywhere Chain FWDO_external_allow (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain FWDO_external_deny (1 references) target prot opt source destination Chain FWDO_external_log (1 references) target prot opt source destination Chain FWDO_public (2 references) target prot opt source destination FWDO_public_log all -- anywhere anywhere FWDO_public_deny all -- anywhere anywhere FWDO_public_allow all -- anywhere anywhere Chain FWDO_public_allow (1 references) target prot opt source destination Chain FWDO_public_deny (1 references) target prot opt source destination Chain FWDO_public_log (1 references) target prot opt source destination Chain INPUT_ZONES (1 references) target prot opt source destination IN_public all -- anywhere anywhere IN_public all -- anywhere anywhere Chain INPUT_ZONES_SOURCE (1 references) target prot opt source destination Chain INPUT_direct (1 references) target prot opt source destination Chain IN_dmz (0 references) target prot opt source destination IN_dmz_log all -- anywhere anywhere IN_dmz_deny all -- anywhere anywhere IN_dmz_allow all -- anywhere anywhere Chain IN_dmz_allow (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW ACCEPT tcp -- anywhere anywhere tcp ctstate NEW ACCEPT udp -- anywhere anywhere udp ctstate NEW Chain IN_dmz_deny (1 references) target prot opt source destination Chain IN_dmz_log (1 references) target prot opt source destination Chain IN_external (0 references) target prot opt source destination IN_external_log all -- anywhere anywhere IN_external_deny all -- anywhere anywhere IN_external_allow all -- anywhere anywhere Chain IN_external_allow (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW ACCEPT tcp -- anywhere anywhere tcp ctstate NEW ACCEPT udp -- anywhere anywhere udp ctstate NEW Chain IN_external_deny (1 references) target prot opt source destination Chain IN_external_log (1 references) target prot opt source destination Chain IN_home (0 references) target prot opt source destination IN_home_log all -- anywhere anywhere IN_home_deny all -- anywhere anywhere IN_home_allow all -- anywhere anywhere Chain IN_home_allow (1 references) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:ipp ctstate NEW ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW ACCEPT udp -- anywhere anywhere udp dpt:netbios-ns ctstate NEW ACCEPT udp -- anywhere anywhere udp dpt:netbios-dgm ctstate NEW ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW ACCEPT tcp -- anywhere anywhere tcp ctstate NEW ACCEPT udp -- anywhere anywhere udp ctstate NEW Chain IN_home_deny (1 references) target prot opt source destination Chain IN_home_log (1 references) target prot opt source destination Chain IN_internal (0 references) target prot opt source destination IN_internal_log all -- anywhere anywhere IN_internal_deny all -- anywhere anywhere IN_internal_allow all -- anywhere anywhere Chain IN_internal_allow (1 references) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:ipp ctstate NEW ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW ACCEPT udp -- anywhere anywhere udp dpt:netbios-ns ctstate NEW ACCEPT udp -- anywhere anywhere udp dpt:netbios-dgm ctstate NEW ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW ACCEPT tcp -- anywhere anywhere tcp ctstate NEW ACCEPT udp -- anywhere anywhere udp ctstate NEW Chain IN_internal_deny (1 references) target prot opt source destination Chain IN_internal_log (1 references) target prot opt source destination Chain IN_public (2 references) target prot opt source destination IN_public_log all -- anywhere anywhere IN_public_deny all -- anywhere anywhere IN_public_allow all -- anywhere anywhere Chain IN_public_allow (1 references) target prot opt source destination ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW ACCEPT tcp -- anywhere anywhere tcp ctstate NEW ACCEPT udp -- anywhere anywhere udp ctstate NEW Chain IN_public_deny (1 references) target prot opt source destination Chain IN_public_log (1 references) target prot opt source destination Chain IN_work (0 references) target prot opt source destination IN_work_log all -- anywhere anywhere IN_work_deny all -- anywhere anywhere IN_work_allow all -- anywhere anywhere Chain IN_work_allow (1 references) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:ipp ctstate NEW ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW ACCEPT tcp -- anywhere anywhere tcp ctstate NEW ACCEPT udp -- anywhere anywhere udp ctstate NEW Chain IN_work_deny (1 references) target prot opt source destination Chain IN_work_log (1 references) target prot opt source destination Chain OUTPUT_direct (1 references) target prot opt source destination Chain neutron-filter-top (2 references) target prot opt source destination neutron-openvswi-local all -- anywhere anywhere Chain neutron-openvswi-FORWARD (1 references) target prot opt source destination neutron-openvswi-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-out tap256fc450-51 --physdev-is-bridged neutron-openvswi-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-in tap256fc450-51 --physdev-is-bridged Chain neutron-openvswi-INPUT (1 references) target prot opt source destination neutron-openvswi-o256fc450-5 all -- anywhere anywhere PHYSDEV match --physdev-in tap256fc450-51 --physdev-is-bridged Chain neutron-openvswi-OUTPUT (1 references) target prot opt source destination Chain neutron-openvswi-i256fc450-5 (1 references) target prot opt source destination DROP all -- anywhere anywhere state INVALID RETURN all -- anywhere anywhere state RELATED,ESTABLISHED RETURN tcp -- anywhere anywhere tcp dpt:ssh RETURN icmp -- anywhere anywhere RETURN udp -- 10.0.0.2 anywhere udp spt:bootps dpt:bootpc neutron-openvswi-sg-fallback all -- anywhere anywhere Chain neutron-openvswi-local (1 references) target prot opt source destination Chain neutron-openvswi-o256fc450-5 (2 references) target prot opt source destination RETURN udp -- anywhere anywhere udp spt:bootpc dpt:bootps neutron-openvswi-s256fc450-5 all -- anywhere anywhere DROP udp -- anywhere anywhere udp spt:bootps dpt:bootpc DROP all -- anywhere anywhere state INVALID RETURN all -- anywhere anywhere state RELATED,ESTABLISHED RETURN all -- anywhere anywhere neutron-openvswi-sg-fallback all -- anywhere anywhere Chain neutron-openvswi-s256fc450-5 (1 references) target prot opt source destination RETURN all -- 10.0.0.3 anywhere MAC FA:16:3E:FA:45:13 DROP all -- anywhere anywhere Chain neutron-openvswi-sg-chain (2 references) target prot opt source destination neutron-openvswi-i256fc450-5 all -- anywhere anywhere PHYSDEV match --physdev-out tap256fc450-51 --physdev-is-bridged neutron-openvswi-o256fc450-5 all -- anywhere anywhere PHYSDEV match --physdev-in tap256fc450-51 --physdev-is-bridged ACCEPT all -- anywhere anywhere Chain neutron-openvswi-sg-fallback (2 references) target prot opt source destination DROP all -- anywhere anywhere Chain nova-api-FORWARD (1 references) target prot opt source destination Chain nova-api-INPUT (1 references) target prot opt source destination ACCEPT tcp -- anywhere localhost.localdomain tcp dpt:8775 Chain nova-api-OUTPUT (1 references) target prot opt source destination Chain nova-api-local (1 references) target prot opt source destination Chain nova-filter-top (2 references) target prot opt source destination nova-api-local all -- anywhere anywhere To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1263338/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp