[Yahoo-eng-team] [Bug 1570113] Re: compute nodes do not get dvrha router update from server
Re-opening this bug as still can be reproduce with master branch by following below. Master branch : April 19 commit 21a2dbc25d2eeeae43faf6c7e5ebd2e4f7fd2be2 1) Create network/subnet. 2) Create external network & subnet. 3) Create DVR HA router. 4) Set router gateway. 5) Attached router interface to above created tenant network. 6) Boot VM in the above created network and noticed that router namesapce is not being created on the compute node. Example: neutron net-create n3 neutron subnet-create --name s3 n3 3.3.3.0/24 neutron router-create r3 --distributed True --ha True neutron router-gateway-set r3 ext-net neutron router-interface-add r3 s3 nova boot --image cirros-0.3.4-x86_64-uec --flavor 42 --nic net-id=b8fa3205-16f2-4e5a-aaf5-c0ca15195771 vm3 VM is running: stack@ctl1:/opt/stack/neutron$ nova list +--+--+++-++ | ID | Name | Status | Task State | Power State | Networks | +--+--+++-++ | 6ab2d1ad-284c-429f-b68e-37ccb82039b9 | vm3 | ACTIVE | - | Running | n3=3.3.3.5 | +--+--+++-++ from compute node: stack@cn:/opt/stack/neutron$ virsh list IdName State 4 instance-0006 running stack@cn:/opt/stack/neutron$ ip netns stack@cn:/opt/stack/neutron$ Workaround: If we boot VM first and then add router's interface to tenant network then router namespace is being created. ** Changed in: neutron Status: Invalid => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1570113 Title: compute nodes do not get dvrha router update from server Status in neutron: Confirmed Bug description: branch: master (April 13th 2016): commit 2a305c563073a3066aac3f07aab3c895ec2cd2fb Topology: 1 controller (neutronserver) 3 network nodes (l3_agent in dvr_snat mode) 2 compute nodes (l3_agent in dvr mode) behavior: when a dvr/ha router is created and a vm instantiated on the compute node, the l3_agent on the compute node does not instantiate a local router. (the qr-namespace along with all the interfaces). expected: when a dvr/ha router is created and a vm is instantiated on a compute node, the l3_agent running on that compute node should instantiate a local router. The qr-router-id namespace along with all the appropriate interfaces should be present locally on the compute node. Similar in behavior to a dvr only router. (without ha). To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1570113/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1557909] [NEW] SNAT namespace is not getting cleared after the manual move of SNAT with dead agent
Public bug reported: Stale snat namespace on the controller after recovery of dead l3 agent. Note: Only on Stable/LIBERTY Branch: Setup: Multiple controller (DVR_SNAT) setup. Steps: 1) Create tenant network, subnet and router. 2) Create a external network 3) Attached internal & external network to a router 4) Create VM on above tenant network. 5) Make sure VM can reach outside using CSNAT. 6) Find router hosting l3 agent and stop the l3 agent. 7) Manually move router to other controller (dvr_snat mode). SNAT namespace should be create on new controller node. 8) Start the l3 agent on the controller (the one that stopped in step6) 9) Notice that snat namespace is now available on 2 controller and it is not getting deleted from the agent which is not hosting it. Example: | cfa97c12-b975-4515-86c3-9710c9b88d76 | L3 agent | vm2-ctl2-936 | :-) | True | neutron-l3-agent | | df4ca7c5-9bae-4cfb-bc83-216612b2b378 | L3 agent | vm1-ctl1-936 | :-) | True | neutron-l3-agent | mysql> select * from csnat_l3_agent_bindings; +--+--+-+--+ | router_id| l3_agent_id | host_id | csnat_gw_port_id | +--+--+-+--+ | 0fb68420-9e69-41bb-8a88-8ab53b0faabb | cfa97c12-b975-4515-86c3-9710c9b88d76 | NULL| NULL | +--+--+-+--+ On vm1-ctl1-936 Stale SNAT namespace on Initially hosting controller. ubuntu@vm1-ctl1-936:~/devstack$ sudo ip netns snat-0fb68420-9e69-41bb-8a88-8ab53b0faabb qrouter-0fb68420-9e69-41bb-8a88-8ab53b0faabb On vm2-ctl2-936 (2nd Controller) ubuntu@vm2-ctl2-936:~$ ip netns snat-0fb68420-9e69-41bb-8a88-8ab53b0faabb qrouter-0fb68420-9e69-41bb-8a88-8ab53b0faabb ** Affects: neutron Importance: Undecided Status: New ** Tags: l3-dvr-backlog ** Tags added: l3-dvr-backlog -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1557909 Title: SNAT namespace is not getting cleared after the manual move of SNAT with dead agent Status in neutron: New Bug description: Stale snat namespace on the controller after recovery of dead l3 agent. Note: Only on Stable/LIBERTY Branch: Setup: Multiple controller (DVR_SNAT) setup. Steps: 1) Create tenant network, subnet and router. 2) Create a external network 3) Attached internal & external network to a router 4) Create VM on above tenant network. 5) Make sure VM can reach outside using CSNAT. 6) Find router hosting l3 agent and stop the l3 agent. 7) Manually move router to other controller (dvr_snat mode). SNAT namespace should be create on new controller node. 8) Start the l3 agent on the controller (the one that stopped in step6) 9) Notice that snat namespace is now available on 2 controller and it is not getting deleted from the agent which is not hosting it. Example: | cfa97c12-b975-4515-86c3-9710c9b88d76 | L3 agent | vm2-ctl2-936 | :-) | True | neutron-l3-agent | | df4ca7c5-9bae-4cfb-bc83-216612b2b378 | L3 agent | vm1-ctl1-936 | :-) | True | neutron-l3-agent | mysql> select * from csnat_l3_agent_bindings; +--+--+-+--+ | router_id| l3_agent_id | host_id | csnat_gw_port_id | +--+--+-+--+ | 0fb68420-9e69-41bb-8a88-8ab53b0faabb | cfa97c12-b975-4515-86c3-9710c9b88d76 | NULL| NULL | +--+--+-+--+ On vm1-ctl1-936 Stale SNAT namespace on Initially hosting controller. ubuntu@vm1-ctl1-936:~/devstack$ sudo ip netns snat-0fb68420-9e69-41bb-8a88-8ab53b0faabb qrouter-0fb68420-9e69-41bb-8a88-8ab53b0faabb On vm2-ctl2-936 (2nd Controller) ubuntu@vm2-ctl2-936:~$ ip netns snat-0fb68420-9e69-41bb-8a88-8ab53b0faabb qrouter-0fb68420-9e69-41bb-8a88-8ab53b0faabb To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1557909/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1518444] [NEW] DVR: router namespace is not getting removed once all VMs from a compute node migrates to other node
Public bug reported: Setup: 1) Multimode setup with 1 controller & 2 compute nodes running linux+KVM. 2) NFS for shared storage. (instances_path = /opt/stack/data/nova/instances is shared) Steps: 1) Create 2 private networks. 2) Create a DVR router and add an interface to each of the above network. 3) Create 1st VM on private network 1 and on compute node1 4) Create 2nd VM on private network 2 and on compute node 2 5) Migrate VM2 from compute node 2 to compute node 1 (nova live-migrate VM2) 6) Notice that after VM2 migrates to compute node1, router namespace is still there on the compute node 2. Example: Before migration: VM11 & VM12 are hosted on the different compute nodes (CN-1 & CN-2). stack@CTL:~$ nova show vm11 | grep OS-EXT-SRV-ATTR:host | OS-EXT-SRV-ATTR:host | CN-1 | | OS-EXT-SRV-ATTR:hostname | vm11 | stack@CTL:~$ nova show vm12 | grep OS-EXT-SRV-ATTR:host | OS-EXT-SRV-ATTR:host | CN-2 | | OS-EXT-SRV-ATTR:hostname | vm12 | Router namespace is present on both the compute nodes: stack@CN-1:~$ ip netns qrouter-9d439e4e-c4c6-4901-ba32-0e793b4df3d8 stack@CN-2:~$ sudo ip netns qrouter-9d439e4e-c4c6-4901-ba32-0e793b4df3d8 After migrating VM12 to CN-1:(Both VMs are now hosted on CN-1) stack@CTL:~$ nova show vm11 | grep OS-EXT-SRV-ATTR:host | OS-EXT-SRV-ATTR:host | CN-1 | | OS-EXT-SRV-ATTR:hostname | vm11 | stack@CTL:~$ nova show vm12 | grep OS-EXT-SRV-ATTR:host | OS-EXT-SRV-ATTR:host | CN-1 | | OS-EXT-SRV-ATTR:hostname | vm12 | Router namespace is still present on the compute node2 which is not hosting any VMs. stack@CTL:~$ nova list +--+--+++-++ | ID | Name | Status | Task State | Power State | Networks | +--+--+++-++ | 0a2f82e0-3edd-47c5-aa24-a29d5b826a55 | vm11 | ACTIVE | - | Running | n1=1.1.1.4 | | 1274d128-c39c-4598-a8f6-d4629a259bbc | vm12 | ACTIVE | - | Running | n2=2.2.2.3 | stack@CN-2:~/devstack$ sudo ip netns qrouter-9d439e4e-c4c6-4901-ba32-0e793b4df3d8 ** Affects: neutron Importance: Undecided Status: New ** Tags: l3-dvr-backlog ** Tags added: l3-dvr-backlog -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1518444 Title: DVR: router namespace is not getting removed once all VMs from a compute node migrates to other node Status in neutron: New Bug description: Setup: 1) Multimode setup with 1 controller & 2 compute nodes running linux+KVM. 2) NFS for shared storage. (instances_path = /opt/stack/data/nova/instances is shared) Steps: 1) Create 2 private networks. 2) Create a DVR router and add an interface to each of the above network. 3) Create 1st VM on private network 1 and on compute node1 4) Create 2nd VM on private network 2 and on compute node 2 5) Migrate VM2 from compute node 2 to compute node 1 (nova live-migrate VM2) 6) Notice that after VM2 migrates to compute node1, router namespace is still there on the compute node 2. Example: Before migration: VM11 & VM12 are hosted on the different compute nodes (CN-1 & CN-2). stack@CTL:~$ nova show vm11 | grep OS-EXT-SRV-ATTR:host | OS-EXT-SRV-ATTR:host | CN-1 | | OS-EXT-SRV-ATTR:hostname | vm11 | stack@CTL:~$ nova show vm12 | grep OS-EXT-SRV-ATTR:host | OS-EXT-SRV-ATTR:host | CN-2 | | OS-EXT-SRV-ATTR:hostname | vm12 | Router namespace is present on both the compute nodes: stack@CN-1:~$ ip netns qrouter-9d439e4e-c4c6-4901-ba32-0e793b4df3d8 stack@CN-2:~$ sudo ip netns qrouter-9d439e4e-c4c6-4901-ba32-0e793b4df3d8 After migrating VM12 to CN-1:(Both VMs are now hosted on CN-1) stack@CTL:~$ nova show vm11 | grep OS-EXT-SRV-ATTR:host | OS-EXT-SRV-ATTR:host | CN-1 | | OS-EXT-SRV-ATTR:hostname | vm11 | stack@CTL:~$ nova show vm12 | grep OS-EXT-SRV-ATT
[Yahoo-eng-team] [Bug 1479130] [NEW] DVR:Removing interface from router with ext gw set does not remove interface from SNAT namespace
Public bug reported: Steps to reproduce: 1) Create one private and one public network. 2) Create DVR Router. 3) Add internal interface to router. 4) Set gateway to router. (qrouter & snat namespace should be created). 5) Remove internal interface from router (by port or by subnet) 6) Notice that corresponding SNAT interface for the internal network from SNAT namespace is still there. So if we add internal interface again to a router then 2 SNAT interfaces for internal network will be there in the SNAT Namespace, which breaks external traffic for private subnet. $ neutron net-list +--+-+--+ | id | name| subnets | +--+-+--+ | 6a180ace-23a5-4300-89b2-e54872b4994c | n1 | f16081e0-5674-4caf-aeef-19f1ca3ab4cf 192.168.20.0/24 | | acf1512c-683b-435c-a161-5c5eba916fa0 | ext-net | 8bf3aa4a-8791-44d1-8a7a-0c99a9412c09 10.10.20.0/24 | +--+-+--+ $ neutron router-list +--+--+---+-+---+ | id | name | external_gateway_info | distributed | ha| +--+--+---+-+---+ | 4948fdfa-6f67-4ede-8e9a-dc960c08b4fd | r1 | null | True | False | +--+--+---+-+---+ $ neutron router-interface-add r1 s1 Added interface 59f3fd7b-5125-41a3-95fe-368890f955e4 to router r1. $ neutron router-gateway-set r1 ext-net Set gateway for router r1 $ ip netns snat-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd qrouter-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd $ neutron router-interface-delete r1 s1 Removed interface from router r1 It remove interface from qrouter namespace $ sudo ip netns exec qrouter-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd ifconfig loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Not removing sg interface from sname namespace. sudo ip netns exec snat-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd ifconfig loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) qg-9c6eb6ec-17 Link encap:Ethernet HWaddr fa:16:3e:77:4c:43 inet addr:10.10.20.107 Bcast:10.10.20.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe77:4c43/64 Scope:Link UP BROADCAST RUNNING MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:1300 (1.3 KB) sg-4f5377ff-fc Link encap:Ethernet HWaddr fa:16:3e:ae:ac:d2 inet addr:192.168.20.3 Bcast:192.168.20.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:feae:acd2/64 Scope:Link UP BROADCAST RUNNING MTU:1500 Metric:1 RX packets:12 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:992 (992.0 B) TX bytes:952 (952.0 B) Re-adding internal interface to router will have 2 sg ports inside the SNAT namespace. $ neutron router-interface-add r1 s1 Added interface 57d66312-c222-4df2-9120-273a9a540925 to router r1. $ sudo ip netns exec snat-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd ifconfig loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) qg-9c6eb6ec-17 Link encap:Ethernet HWaddr fa:16:3e:77:4c:43 inet addr:10.10.20.107 Bcast:10.10.20.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe77:4c43/64 Scope:Link UP BROADCAST RUNNING MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:18 errors:
