Hi Darragh,
Even I am facing the same issue of request getting timed out and even updates getting hanged up for very long time. I followed your step of reducing the MTU size from 1500 to 1454 and now everything works fine. I tried this on Ubuntu instances. This seems to be an issue with the Grizzly release. I had already started email-thread earlier for this but was unable to find the root cause. Here is the link to it:- https://lists.launchpad.net/openstack/msg23993.html Thank you for your suggestion of reducing the MTU size as it solved the problem. You must file a bug for this so that this issue can be tracked. Thanks and Regards Rahul Sharma On Thu, May 30, 2013 at 2:23 AM, Darragh O'Reilly < dara2002-openst...@yahoo.com> wrote: > Hi Farhan, > > I was able to reproduce this with curl from the cirros 0.3.1 that supports > ssl. > > cirros$ curl -L github.com # -L follow redirects > > it just hangs and I get these ICMPs on the netnode's physical nic. > > 20:33:10.811485 IP (tos 0xc0, ttl 63, id 13647, offset 0, flags [none], > proto ICMP (1), length 576) > 192.168.101.2 > 204.232.175.90: ICMP 192.168.101.2 unreachable - need > to frag (mtu 1454), length 556 > IP (tos 0x0, ttl 51, id 54729, offset 0, flags [DF], proto TCP (6), length > 1500) > 204.232.175.90.443 > 192.168.101.2.41237: Flags [.], seq 1:1449, ack > 225, win 7, options [nop,nop,TS val 4208725487 ecr 171322], length 1448 > > So I reduced the mtu from the default 1500 to 1454 on the instance and now > 'curl -L github.com' works > > cirros$ sudo ip link set mtu 1454 dev eth0 > > Will need to look into this more. Maybe to do with the GRE tunnels > (+~20bytes?) or iptables. Anyway try reducing the mtu for now. > > Darragh. > > > ----- Original Message ----- > > From: Farhan Patwa <farhan.pa...@utsa.edu> > > To: Darragh O'Reilly <dara2002-openst...@yahoo.com>; OpenStack Maillist > <openstack@lists.launchpad.net> > > Cc: > > Sent: Wednesday, 29 May 2013, 18:14 > > Subject: Re: [Openstack] VM Issues on Grizzly Install on Ubuntu 12.04 > > > > Hi Darragh, > > Thank you soo Much! That was it! Now I am able to connect to the VM with > > no issues. > > > > But I am back to another network issue I had when I had Folsom installed > > on the same setup. > > I would really appreciate if you can provide any pointers here. > > > > > > I able to spawn VM get IP, set floating IP and now am trying to do some > > development within the VM. > > I am unable to connect to certain sites and ports: > > git clone https://github.com/openstack-dev/devstack.git - <-- This just > > times out. > > > > > ########################################################################### > > ############# > > This is what works: > > Wget google.com > > Wget openstack.com > > > ########################################################################### > > ############# > > This is what hangs and times out: > > > > Wget yahoo.com > > Wget paypal.com > > Wget facebook.com > > Wget github.com > > ubuntu@fpatwa-1:~$ wget github.com > > --2013-05-10 19:08:19-- http://github.com/ > > Resolving github.com (github.com)... 204.232.175.90 > > Connecting to github.com (github.com)|204.232.175.90|:80... connected. > > HTTP request sent, awaiting response... 301 Moved Permanently > > Location: https://github.com/ [following] > > --2013-05-10 19:08:20-- https://github.com/ > > Connecting to github.com (github.com)|204.232.175.90|:443... connected. > > > > > ########################################################################### > > ############# > > > > The same commands works on the network node. > > > > > > The pattern that I can see is that any SSL website fails (port 443) but > > then something like yahoo fails also and its at port 80. > > > > > > Here are my security rules: > > +-------------+-----------+---------+-----------+--------------+ > > | IP Protocol | From Port | To Port | IP Range | Source Group | > > +-------------+-----------+---------+-----------+--------------+ > > | icmp | -1 | -1 | 0.0.0.0/0 | | > > | tcp | 1 | 65535 | 0.0.0.0/0 | | > > | tcp | 22 | 22 | 0.0.0.0/0 | | > > | udp | 1 | 65535 | 0.0.0.0/0 | | > > +-------------+-----------+---------+-----------+--------------+ > > > > > > > > I have messed around with all kinds of combinations of security rules but > > no luck so far. > > > > Thanks, > > > > -Farhan. > > > > > > > > > > On 5/28/13 3:28 PM, "Darragh O'Reilly" > > <dara2002-openst...@yahoo.com> > > wrote: > > > >> Hi, > >> > >> the ping error "connect: Network is unreachable" means a route > > could not > >> be found. > >> > >> The gateway 10.245.124.253 for the external subnet is not in the subnet > >> CIDR 10.245.124.64/26. > >> > >> > >> So I guess a default route was not setup here: > >> netnode$ ip netns exec <router ns> route -n > >> > >> You will need to create the subnet with a CIDR that includes the gateway > >> ip - something like this: > >> quantum subnet-create <ext-net-id> 10.245.124.192/26 --gateway > >> 10.245.124.253 --enable_dhcp False > >> > >> Darragh. > >> > >> > >> ----- Original Message ----- > >>> From: Farhan Patwa <farhan.pa...@utsa.edu> > >>> To: Darragh OReilly <darragh.orei...@yahoo.com>; OpenStack > > Maillist > >>> <openstack@lists.launchpad.net> > >>> Cc: > >>> Sent: Tuesday, 28 May 2013, 19:52 > >>> Subject: Re: [Openstack] VM Issues on Grizzly Install on Ubuntu 12.04 > >>> > >>> Hi Darragh, > >>> Thanks a lot for your reply and suggestions. > >>> I am not able to ping the gateway ip from the namespace. > >>> Also eth0 is up but br-ex has unknown state? > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> root@openstack-2:~# ip link > >>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state > > UNKNOWN > >>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > >>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq > > state UP > >>> qlen > >>> 1000 > >>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff > >>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq > > state UP > >>> qlen > >>> 1000 > >>> link/ether 78:2b:cb:27:1f:c9 brd ff:ff:ff:ff:ff:ff > >>> 4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue > >>> state > >>> UNKNOWN > >>> link/ether f2:3b:f7:1b:b0:46 brd ff:ff:ff:ff:ff:ff > >>> 6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue state > >>> UNKNOWN > >>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff > >>> 32: br-tun: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue > >>> state > >>> UNKNOWN > >>> link/ether 7e:6c:65:0f:c9:43 brd ff:ff:ff:ff:ff:ff > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> > >>> Here is the result of the tcpdump as ping is being done: > >>> > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> root@openstack-2:~# ip netns exec > >>> qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping -c1 10.245.124.253 > >>> connect: Network is unreachable > >>> > >>> root@openstack-2:~# tcpdump -nei eth0 > >>> tcpdump: WARNING: eth0: no IPv4 address assigned > >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol > >>> decode > >>> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 > > bytes > >>> 13:46:31.399055 00:26:88:7a:40:87 > 01:80:c2:00:00:00, 802.3, length > > 60: > >>> LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: > > STP > >>> 802.1w, Rapid STP, Flags [Proposal], bridge-id > >>> 8000.00:26:88:7a:40:81.8205, length 43 > >>> 13:46:33.259195 c2:35:07:e7:b0:10 > ff:ff:ff:ff:ff:ff, ethertype ARP > >>> (0x0806), length 60: Reply 10.245.0.10 is-at c2:35:07:e7:b0:10, length > >>> 46 > >>> 13:46:33.313988 00:26:88:7a:40:87 > 01:80:c2:00:00:00, 802.3, length > > 60: > >>> LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: > > STP > >>> 802.1w, Rapid STP, Flags [Proposal], bridge-id > >>> 8000.00:26:88:7a:40:81.8205, length 43 > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> > >>> > >>> The other information that you wanted is: > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> root@openstack-2:~# ip link > >>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state > > UNKNOWN > >>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > >>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq > > state UP > >>> qlen > >>> 1000 > >>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff > >>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq > > state UP > >>> qlen > >>> 1000 > >>> link/ether 78:2b:cb:27:1f:c9 brd ff:ff:ff:ff:ff:ff > >>> 4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue > >>> state > >>> UNKNOWN > >>> link/ether f2:3b:f7:1b:b0:46 brd ff:ff:ff:ff:ff:ff > >>> 6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue state > >>> UNKNOWN > >>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff > >>> 32: br-tun: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue > >>> state > >>> UNKNOWN > >>> link/ether 7e:6c:65:0f:c9:43 brd ff:ff:ff:ff:ff:ff > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> root@openstack-2:~# ip netns exec > >>> qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ip address > >>> 25: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state > > UNKNOWN > >>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > >>> inet 127.0.0.1/8 scope host lo > >>> inet6 ::1/128 scope host > >>> valid_lft forever preferred_lft forever > >>> 39: qr-eebfe1cb-0f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > > qdisc > >>> noqueue state UNKNOWN > >>> link/ether fa:16:3e:08:16:19 brd ff:ff:ff:ff:ff:ff > >>> inet 50.50.1.1/24 brd 50.50.1.255 scope global qr-eebfe1cb-0f > >>> inet6 fe80::f816:3eff:fe08:1619/64 scope link > >>> valid_lft forever preferred_lft forever > >>> 40: qg-910fef3b-cb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > > qdisc > >>> noqueue state UNKNOWN > >>> link/ether fa:16:3e:e3:d5:fa brd ff:ff:ff:ff:ff:ff > >>> inet 10.245.124.65/26 brd 10.245.124.127 scope global > > qg-910fef3b-cb > >>> inet 10.245.124.67/32 brd 10.245.124.67 scope global > qg-910fef3b-cb > >>> inet6 fe80::f816:3eff:fee3:d5fa/64 scope link > >>> valid_lft forever preferred_lft forever > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> root@openstack-2:~# quantum net-show > >>> 37d27ee8-36a9-4cdb-9966-9b5571526b41 > >>> +---------------------------+--------------------------------------+ > >>> | Field | Value | > >>> +---------------------------+--------------------------------------+ > >>> | admin_state_up | True | > >>> | id | 37d27ee8-36a9-4cdb-9966-9b5571526b41 | > >>> | name | ext_net | > >>> | provider:network_type | gre | > >>> | provider:physical_network | | > >>> | provider:segmentation_id | 1 | > >>> | router:external | True | > >>> | shared | True | > >>> | status | ACTIVE | > >>> | subnets | dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750 | > >>> | tenant_id | 2990df1bd46c4dda915b43558d591a2f | > >>> +---------------------------+--------------------------------------+ > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> > >>> > >>> > >>> root@openstack-2:~# quantum subnet-show > >>> dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750 > >>> > >>> > +------------------+----------------------------------------------------- > >>> + > >>> | Field | Value > >>> | > >>> > >>> > +------------------+----------------------------------------------------- > >>> + > >>> | allocation_pools | {"start": "10.245.124.65", > >>> "end": "10.245.124.126"} | > >>> | cidr | 10.245.124.64/26 > >>> | > >>> | dns_nameservers | 10.245.0.10 > >>> | > >>> | enable_dhcp | False > >>> | > >>> | gateway_ip | 10.245.124.253 > >>> | > >>> | host_routes | > >>> | > >>> | id | dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750 > >>> | > >>> | ip_version | 4 > >>> | > >>> | name | > >>> | > >>> | network_id | 37d27ee8-36a9-4cdb-9966-9b5571526b41 > >>> | > >>> | tenant_id | 2990df1bd46c4dda915b43558d591a2f > >>> | > >>> > >>> > +------------------+----------------------------------------------------- > >>> + > >>> > >>> > ######################################################################### > >>> ## > >>> ####################### > >>> > >>> > >>> > >>> Thanks, > >>> > >>> -Farhan. > >>> > >>> > >>> > >>> > >>> > >>> On 5/27/13 4:08 AM, "Darragh OReilly" > >>> <darragh.orei...@yahoo.com> wrote: > >>> > >>>> > >>>> I'd check the external network config first. > >>>> > >>>> You should be able to ping the external subnet's gateway from > > the > >>>> router > >>>> namespace. > >>>> This gateway should correspond to some real external > > gateway/router. > >>>> > >>>> quantum subnet-show <ext sub id> -c gateway_ip # > > 10.245.124.1 ? > >>>> ip netns exec <router-ns> ping -c1 <ext sub gateway> > >>>> > >>>> If that is not working use tcpdump as you ping. Br-ex is using > > eth0, is > >>>> eth0 up? tcpdump -nei eth0 > >>>> > >>>> > >>>> If you are still having problems, post the above output and the > >>>> following: > >>>> > >>>> # network node > >>>> ip link > >>>> ip netns exec <router-ns> ip address > >>>> > >>>> quantum net-show <uuid of external net> > >>>> quantum subnet-show <uuid of external subnet> > >>>> > >>>> > >>>>> ________________________________ > >>>>> From: Farhan Patwa <farhan.pa...@utsa.edu> > >>>>> To: OpenStack Maillist <openstack@lists.launchpad.net> > >>>>> Sent: Friday, 24 May 2013, 20:28 > >>>>> Subject: [Openstack] VM Issues on Grizzly Install on Ubuntu > > 12.04 > >>>>> > >>>>> > >>>>> > >>>>> Hello, > >>>>> I followed the following guide to install Grizzly release on > > 3-node > >>>>> setup. > >>>>> > >>>>> > http://docs.openstack.org/grizzly/basic-install/apt/content/basic-insta > >>>>> ll > >>>>> _intro.html > >>>>> > >>>>> > >>>>> I am stuck at my last issue with Quantum networking (at least > > that¹s > >>>>> what I think). > >>>>> The VM instance comes up and gets the private IP and the > > metadata. > >>>>> Also I have assigned the floating IP to it but am not able to > > ping > >>>>> either IP except when I use: > >>>>> > >>>>> > >>>>> ip netns exec qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping > >>>>> 50.50.1.3 <- fixed IP private network > >>>>> ip netns exec qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping > >>>>> 10.24.124.4 <- floating IP external network > >>>>> > >>>>> > >>>>> Based on that I think the security rules are okay > >>>>> The router is tied to the specified tenant and using gateway of > > the > >>>>> external network. > >>>>> I think the issue is routing table or maybe firewall related > > but not > >>>>> sure how to debug this. > >>>>> > >>>>> > >>>>> Some details of my environment are below. > >>>>> Any one have any words of wisdom/guidance? > >>>>> > >>>>> > >>>>> Thanks, > >>>>> > >>>>> > >>>>> -Farhan. > >>>>> > >>>>> > >>>>> Management Network: 192.168.0.0/24 > >>>>> Data Network: 10.5.5.0/24 > >>>>> External Network: 10.245.124.0/24 > >>>>> > >>>>> > >>>>> Network Node: (192.168.0.2) > >>>>> ovs-vsctl show > >>>>> ea4fa894-5986-40f2-b10b-55eef2222408 > >>>>> Bridge br-tun > >>>>> Port patch-int > >>>>> Interface patch-int > >>>>> type: patch > >>>>> options: {peer=patch-tun} > >>>>> Port "gre-1" > >>>>> Interface "gre-1" > >>>>> type: gre > >>>>> options: {in_key=flow, out_key=flow, > >>>>> remote_ip="192.168.0.3"} > >>>>> Port br-tun > >>>>> Interface br-tun > >>>>> type: internal > >>>>> Bridge br-int > >>>>> Port "tap3fca71a9-c8" > >>>>> tag: 4095 > >>>>> Interface "tap3fca71a9-c8" > >>>>> type: internal > >>>>> Port patch-tun > >>>>> Interface patch-tun > >>>>> type: patch > >>>>> options: {peer=patch-int} > >>>>> Port "tap4b8a22a2-9c" > >>>>> tag: 4095 > >>>>> Interface "tap4b8a22a2-9c" > >>>>> type: internal > >>>>> Port "tap633ed611-a9" > >>>>> tag: 1 > >>>>> Interface "tap633ed611-a9" > >>>>> type: internal > >>>>> Port "qr-eebfe1cb-0f" > >>>>> tag: 1 > >>>>> Interface "qr-eebfe1cb-0f" > >>>>> type: internal > >>>>> Port br-int > >>>>> Interface br-int > >>>>> type: internal > >>>>> Bridge br-ex > >>>>> Port "eth0" > >>>>> Interface "eth0" > >>>>> Port br-ex > >>>>> Interface br-ex > >>>>> type: internal > >>>>> Port "qg-910fef3b-cb" > >>>>> Interface "qg-910fef3b-cb" > >>>>> type: internal > >>>>> ovs_version: "1.4.0+build0" > >>>>> > >>>>> > >>>>> Kernel IP routing table > >>>>> Destination Gateway Genmask Flags Metric > > Ref > >>>>> Use > >>>>> Iface > >>>>> 0.0.0.0 192.168.0.253 0.0.0.0 UG 0 0 > >>>>> 0 > >>>>> eth1 > >>>>> 10.5.5.0 0.0.0.0 255.255.255.0 U 0 0 > >>>>> 0 > >>>>> eth1 > >>>>> 10.245.124.0 0.0.0.0 255.255.255.0 U 0 0 > >>>>> 0 > >>>>> br-ex > >>>>> 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 > >>>>> 0 > >>>>> eth1 > >>>>> > >>>>> > >>>>> Compute Node: (192.168.0.3) > >>>>> ovs-vsctl show > >>>>> f0fe78a5-dfd0-4f6b-87be-466dac0b4473 > >>>>> Bridge br-tun > >>>>> Port patch-int > >>>>> Interface patch-int > >>>>> type: patch > >>>>> options: {peer=patch-tun} > >>>>> Port br-tun > >>>>> Interface br-tun > >>>>> type: internal > >>>>> Port "gre-2" > >>>>> Interface "gre-2" > >>>>> type: gre > >>>>> options: {in_key=flow, out_key=flow, > >>>>> remote_ip="192.168.0.2"} > >>>>> Bridge br-int > >>>>> Port patch-tun > >>>>> Interface patch-tun > >>>>> type: patch > >>>>> options: {peer=patch-int} > >>>>> Port br-int > >>>>> Interface br-int > >>>>> type: internal > >>>>> Port "tap6514a8cc-b2" > >>>>> tag: 1 > >>>>> Interface "tap6514a8cc-b2" > >>>>> ovs_version: "1.4.0+build0" > >>>>> > >>>>> > >>>>> Kernel IP routing table > >>>>> Destination Gateway Genmask Flags Metric > > Ref > >>>>> Use > >>>>> Iface > >>>>> 0.0.0.0 192.168.0.253 0.0.0.0 UG 0 0 > >>>>> 0 > >>>>> eth1 > >>>>> 10.5.5.0 0.0.0.0 255.255.255.0 U 0 0 > >>>>> 0 > >>>>> eth1 > >>>>> 10.245.124.0 0.0.0.0 255.255.255.0 U 0 0 > >>>>> 0 > >>>>> eth0 > >>>>> 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 > >>>>> 0 > >>>>> eth1 > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> Mailing list: https://launchpad.net/~openstack > >>>>> Post to : openstack@lists.launchpad.net > >>>>> Unsubscribe : https://launchpad.net/~openstack > >>>>> More help : https://help.launchpad.net/ListHelp > >>>>> > >>>>> > >>>>> > >>>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> Mailing list: https://launchpad.net/~openstack > >>> Post to : openstack@lists.launchpad.net > >>> Unsubscribe : https://launchpad.net/~openstack > >>> More help : https://help.launchpad.net/ListHelp > >>> > >> > > > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp