Everyone, thanks for all the help, but I found the problem, and am going to go pound my head against the wall this evening or have a martini.
My cloud config uses VLAN tagging. Somehow, during the upgrade to new hardware and the essex to havana migration, the tag profile in my Brocade switch got set back to the default tag value of 8100. This prevents packets tagged by Openstack from being switched between ports. Simple fix, and I should have suspected this when I didn’t see any traffic on the virtual router. Thanks again everyone and regards, /ross On Feb 11, 2014, at 5:40 PM, Lillie Ross-CDSR11 <ross.lil...@motorolasolutions.com> wrote: > As a further follow-on… > > Forget my question about namespaces on the compute node. Dumb. Realized it > the minute I hit send. > > Regarding my instance not receiving a DHCP response, I did the following test. > > In the namespace for my dhcp server on the network controller, I issued the > following command: > > # ip netns exec qdhcp-05137211-1660-44e1-ae50-107900090e05 tcpdump -i all > > Then, in another process, I boot up and instance of cirros, e.g. > > # nova boot —flavor m1.tiny —key-name root —image cirros tvm > > Nova shows the instance booting, and finally running with the correct DHCP > address, however the process running tcpdump in the namespace shows nothing. > > Any ideas of where to start digging? I know this is a stupid config bug - I > just can’t see it. > > Thanks again, > Ross > > On Feb 11, 2014, at 4:49 PM, Ross Lillie <ross.lil...@motorolasolutions.com> > wrote: > >> Oops! forgot to hit “reply all”. Sorry for the duplicates… Also adding >> additional observations/questions. >> >> When I attach to the compute node, I don’t see any network namespaces. Is >> this normal? Admittedly, I haven’t read up on all the gory details of >> neutron (which I probably need to do this evening). >> >> Original message follows: >> >> ------------------------- >> >> Hi Yuanle, >> >> OK, checking the console log, it doesn’t appear that my instance is getting >> the dhcp assigned address… >> >> -------- >> Starting network... >> udhcpc (v1.20.1) started >> Sending discover... >> Sending discover... >> Sending discover... >> No lease, failing >> WARN: /etc/rc3.d/S40-network failed >> cirros-ds 'net' up at 181.14 >> checking http://169.254.169.254/2009-04-04/instance-id >> failed 1/20: up 181.15. reques >> ———— >> >> and later, in summary I see… >> >> === network info === >> if-info: lo,up,127.0.0.1,8,::1 >> if-info: eth0,up,,8,fe80::f816:3eff:fe56:3612 >> === datasource: None None === >> === cirros: current=0.3.1 uptime=221.94 === >> route: fscanf >> === pinging gateway failed, debugging connection === >> ############ debug start ############## >> ### /etc/init.d/sshd start >> Starting dropbear sshd: OK >> route: fscanf >> ### ifconfig -a >> eth0 Link encap:Ethernet HWaddr FA:16:3E:56:36:12 >> inet6 addr: fe80::f816:3eff:fe56:3612/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:6 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:468 (468.0 B) TX bytes:1112 (1.0 KiB) >> >> lo Link encap:Local Loopback >> inet addr:127.0.0.1 Mask:255.0.0.0 >> inet6 addr: ::1/128 Scope:Host >> UP LOOPBACK RUNNING MTU:16436 Metric:1 >> RX packets:12 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:1020 (1020.0 B) TX bytes:1020 (1020.0 B) >> >> ### route -n >> Kernel IP routing table >> Destination Gateway Genmask Flags Metric Ref Use Iface >> >> I don’t see anything in the logs that indicates a problem to me, so I’m >> missing something. Checking the DHCP logs on the network/controller node I >> see the address is being allocated. The instance, however, isn’t seeing the >> DHCP server. Multi-cast issue? Any guidance is appreciated and thanks again. >> >> Regards, >> /ross >> >> On Feb 10, 2014, at 7:58 PM, sylecn <syl...@gmail.com> wrote: >> >>> Hi Ross, >>> >>> 1. Make sure you have enabled ping (ICMP) in security groups. >>> The default security groups does not allow ping. >>> >>> neutron security-group-rule-create --direction ingress --protocol icmp >>> $SG_ID >>> >>> I suggest you explicitly create security group and use that when you >>> boot instance. In this case, I see two security groups named >>> "default". Better add that rule for both. >>> >>> 2. Check whether you can ping the fixed ip. >>> Run on the neutron node: >>> >>> sudo ip netns exec qrouter-43c596c4-65fe-4c22-a48a-0a6e200abf78 ping -c 4 >>> 10.0.1.2 >>> >>> 3. Check console log of the vm. Did it boot correctly? Did it get IP from >>> DHCP? >>> >>> nova console-log tvm1 >>> >>> Thanks, >>> Yuanle >>> >>> >>> >>> On Tue, Feb 11, 2014 at 8:52 AM, Lillie Ross-CDSR11 >>> <ross.lil...@motorolasolutions.com> wrote: >>> If this issue has already been discussed, please excuse. >>> >>> I’m somewhat confused about neutron configuration and tenancy. Correct me >>> if I’m wrong. >>> >>> First, I’ve create a private network under the ‘admin’ tenant named >>> ‘admin-net'. I’ve associated a subnet named admin-net.1 with the admin-net >>> with a CIDR of 10.0.1.0/24. >>> >>> Next, I created a network with router:external set to True associated with >>> our campus network named ‘campus-net’. This network was created under the >>> ‘service’ tenant’. I also create a router named ‘campus-gw’ under the >>> ‘service’ tenant and set it’s gateway to be the ‘campus-net’ network. >>> >>> Finally, I create a floating address pool under the ‘admin’ tenant named >>> ‘admin-net.float', and add it as an interface to the ‘campus-gw’ router. I >>> also create a default security group under the ‘admin’ tenant to allow SSH >>> and ICMP access. >>> >>> When I boot an image, as a member of the admin tenant, the instance is >>> correctly assigned an IP address from the admin tenant’s private network. >>> I next allocate (nova floating-ip-create admin-net.float) a floating IP >>> address and associated it my running instance. >>> >>> However, I’m unable to ping the running instance, and I see no indication >>> of the end of the tunnel being established on the network/controller node. >>> >>> I’m not that well versed with network namespaces nor the openvswitch >>> commands. >>> >>> 2 questions. Does my overall configuration sound correct? And how best to >>> diagnose what’s going on here? Any pointers would be helpful. Additional >>> details can be provided as needed. Thanks loads in advance. >>> >>> Regards, >>> /ross >>> >>> — >>> >>> (neutron) net-list >>> +--------------------------------------+------------+----------------------------------------------------+ >>> | id | name | subnets >>> | >>> +--------------------------------------+------------+----------------------------------------------------+ >>> | 2426f4d8-a983-4f50-ab5a-fd2a37e5cd94 | campus-net | >>> a948538d-c2c2-4c02-9116-b89a79f0c73a 173.23.0.0/16 | >>> | e6984375-f35b-4636-a293-43d0d296e0ff | admin-net | >>> 2ced890b-944f-4f6e-8f7a-3f5a4d07c2bb 10.0.1.0/24 | >>> +--------------------------------------+------------+----------------------------------------------------+ >>> (neutron) subnet-list >>> +--------------------------------------+--------------------+---------------+---------------------------------------------------+ >>> | id | name | cidr >>> | allocation_pools | >>> +--------------------------------------+--------------------+---------------+---------------------------------------------------+ >>> | 2ced890b-944f-4f6e-8f7a-3f5a4d07c2bb | admin-net.1 | 10.0.1.0/24 >>> | {"start": "10.0.1.2", "end": "10.0.1.254"} | >>> | a948538d-c2c2-4c02-9116-b89a79f0c73a | admin-net.floating | 173.23.0.0/16 >>> | {"start": "173.23.182.2", "end": "173.23.182.15"} | >>> +--------------------------------------+--------------------+---------------+---------------------------------------------------+ >>> (neutron) router-list >>> +--------------------------------------+-----------+-----------------------------------------------------------------------------+ >>> | id | name | external_gateway_info >>> | >>> +--------------------------------------+-----------+-----------------------------------------------------------------------------+ >>> | 43c596c4-65fe-4c22-a48a-0a6e200abf78 | campus-gw | {"network_id": >>> "2426f4d8-a983-4f50-ab5a-fd2a37e5cd94", "enable_snat": true} | >>> +--------------------------------------+-----------+-----------------------------------------------------------------------------+ >>> (neutron) router-show campus-gw >>> +-----------------------+-----------------------------------------------------------------------------+ >>> | Field | Value >>> | >>> +-----------------------+-----------------------------------------------------------------------------+ >>> | admin_state_up | True >>> | >>> | external_gateway_info | {"network_id": >>> "2426f4d8-a983-4f50-ab5a-fd2a37e5cd94", "enable_snat": true} | >>> | id | 43c596c4-65fe-4c22-a48a-0a6e200abf78 >>> | >>> | name | campus-gw >>> | >>> | routes | >>> | >>> | status | ACTIVE >>> | >>> | tenant_id | service >>> | >>> +-----------------------+-----------------------------------------------------------------------------+ >>> (neutron) security-group-list >>> +--------------------------------------+---------+-------------+ >>> | id | name | description | >>> +--------------------------------------+---------+-------------+ >>> | 0d66a3e2-7a0f-4caf-8b63-c3c8f3106242 | default | default | >>> | c87230fa-9193-47a7-8ade-cec5f7f6b958 | default | default | >>> +--------------------------------------+---------+-------------+ >>> (neutron) >>> root@cirrus3:/var/log/neutron# nova list >>> +--------------------------------------+------+--------+------------+-------------+----------------------------------+ >>> | ID | Name | Status | Task State | Power >>> State | Networks | >>> +--------------------------------------+------+--------+------------+-------------+----------------------------------+ >>> | ccdf7541-3a74-4289-a8ce-9fe5cffe9dbd | tvm1 | ACTIVE | None | >>> Running | admin-net=10.0.1.2, 173.23.182.3 | >>> +--------------------------------------+------+--------+------------+-------------+----------------------------------+ >>> root@cirrus3:/var/log/neutron# nova show tvm1 >>> +--------------------------------------+----------------------------------------------------------+ >>> | Property | Value >>> | >>> +--------------------------------------+----------------------------------------------------------+ >>> | status | ACTIVE >>> | >>> | updated | 2014-02-11T00:03:25Z >>> | >>> | OS-EXT-STS:task_state | None >>> | >>> | OS-EXT-SRV-ATTR:host | cn1 >>> | >>> | key_name | root >>> | >>> | image | cirros >>> (57a9f5d6-8b07-4bdb-b8a0-900de339d804) | >>> | admin-net network | 10.0.1.2, 173.23.182.3 >>> | >>> | hostId | >>> 982cd20cde9c5f8514c95b5ca8530258fa9454cdc988a8b007a6d20b | >>> | OS-EXT-STS:vm_state | active >>> | >>> | OS-EXT-SRV-ATTR:instance_name | instance-00000021 >>> | >>> | OS-SRV-USG:launched_at | 2014-02-11T00:03:25.000000 >>> | >>> | OS-EXT-SRV-ATTR:hypervisor_hostname | cn1 >>> | >>> | flavor | m1.tiny (1) >>> | >>> | id | >>> ccdf7541-3a74-4289-a8ce-9fe5cffe9dbd | >>> | security_groups | [{u'name': u'default'}] >>> | >>> | OS-SRV-USG:terminated_at | None >>> | >>> | user_id | 090a2de6e74b4573bd29318d4f494191 >>> | >>> | name | tvm1 >>> | >>> | created | 2014-02-11T00:02:47Z >>> | >>> | tenant_id | ec54b7cadcab4620bbb6d568be7bd4a8 >>> | >>> | OS-DCF:diskConfig | MANUAL >>> | >>> | metadata | {} >>> | >>> | os-extended-volumes:volumes_attached | [] >>> | >>> | accessIPv4 | >>> | >>> | accessIPv6 | >>> | >>> | progress | 0 >>> | >>> | OS-EXT-STS:power_state | 1 >>> | >>> | OS-EXT-AZ:availability_zone | nova >>> | >>> | config_drive | >>> | >>> +--------------------------------------+----------------------------------------------------------+ >>> root@cirrus3:/var/log/neutron# >>> >>> -- >>> Ross Lillie >>> Distinguished Member of Technical Staff >>> Motorola Solutions, Inc. >>> >>> motorolasolutions.com >>> O: +1.847.576.0012 >>> M: +1.847.980.2241 >>> E: ross.lil...@motorolasolutions.com >>> >>> >>> <MSI-Email-Identity-sm.png> >>> >>> >>> _______________________________________________ >>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openstack@lists.openstack.org >>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >>> >> > _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
