[Yahoo-eng-team] [Bug 1926531] [NEW] SNAT namespace prematurely created then deleted on hosts, resulting in removal of RFP/FPR link to FIP namespace
Public bug reported: Seems like collateral from https://bugs.launchpad.net/neutron/+bug/1850779 I think this fix causes problems. We have multiple nodes that are DVR_SNAT mode. Snat namespace is scheduled to 1 of them. When l3-agent is restarted on the othre nodes, now, initialize() is invoked always for DvrEdgeRouter which creates the SNAT namespace prematurely. This in turn causes external_gateway_added() to later detect that this host is NOT hosting snat router, but the namespace exists, so it removes it by triggering external_gateway_removed(dvr_edge_router --> dvr_local_router) Problem is that the dvr_local_router code for external_gateway_removed() ends up DELETING the rfp/fpr pair and severs the qrouter connection to fip namespace (and deletes all the FIP routes in fip namespace as a result). Prior to this bug fix, _create_snat_namespace was only invoked in _create_dvr_gateway(), which was only invoked when the node was actually hosting SNAT for the router. Even without the breaking issue of deleting the rtr_2_fip link, this fix unnecessarily creates SNAT namespace on every host, only for it to be deleted. FYI this is for non-HA routers 1. Where the qrouter to FIP link is deleted: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L599 This results in connectivity breakage 2. Above #1 is triggered by code here in edge router which sees snat namespace, but SNAT is scheduled to different host: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_edge_router.py#L56 3. SNAT namespace is created on wrong host because of bug fix for 1850779 which moved it to DvrEdgeRouter intilization ** Affects: neutron Importance: Undecided Status: New ** Tags: l3-dvr-backlog l3-ha -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1926531 Title: SNAT namespace prematurely created then deleted on hosts, resulting in removal of RFP/FPR link to FIP namespace Status in neutron: New Bug description: Seems like collateral from https://bugs.launchpad.net/neutron/+bug/1850779 I think this fix causes problems. We have multiple nodes that are DVR_SNAT mode. Snat namespace is scheduled to 1 of them. When l3-agent is restarted on the othre nodes, now, initialize() is invoked always for DvrEdgeRouter which creates the SNAT namespace prematurely. This in turn causes external_gateway_added() to later detect that this host is NOT hosting snat router, but the namespace exists, so it removes it by triggering external_gateway_removed(dvr_edge_router --> dvr_local_router) Problem is that the dvr_local_router code for external_gateway_removed() ends up DELETING the rfp/fpr pair and severs the qrouter connection to fip namespace (and deletes all the FIP routes in fip namespace as a result). Prior to this bug fix, _create_snat_namespace was only invoked in _create_dvr_gateway(), which was only invoked when the node was actually hosting SNAT for the router. Even without the breaking issue of deleting the rtr_2_fip link, this fix unnecessarily creates SNAT namespace on every host, only for it to be deleted. FYI this is for non-HA routers 1. Where the qrouter to FIP link is deleted: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L599 This results in connectivity breakage 2. Above #1 is triggered by code here in edge router which sees snat namespace, but SNAT is scheduled to different host: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_edge_router.py#L56 3. SNAT namespace is created on wrong host because of bug fix for 1850779 which moved it to DvrEdgeRouter intilization To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1926531/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1884708] [NEW] explicity_egress_direction prevents learning of local MACs and causes flooding of ingress packets
Public bug reported:
We took this bug fix: https://bugs.launchpad.net/neutron/+bug/1732067
and then also backported ourselves
https://bugs.launchpad.net/neutron/+bug/1866445
The latter is for iptables based firewall.
We have VLAN based networks, and seeing ingress packets destined to
local MACs being flooded. We are not seeing any local MACs present under
ovs-appctl fdb/show br-int.
Consider following example:
HOST 1:
MAC A = fa:16:3e:c1:01:43
MAC B = fa:16:3e:de:0b:8a
HOST 2:
MAC C = fa:16:3e:d6:3f:31
A is talking to C. Snooping on qvo interface of B, we are seeing all the
traffic destined to MAC A (along with other unicast traffic not destined
to or sourced from MAC B. Neither Mac A or B are present in br-int FDB,
despite sending heavy traffic.
Here is ofproto trace for such packet. in_port 8313 is qvo of MAC A:
sudo ovs-appctl ofproto/trace br-int
in_port=8313,tcp,dl_src=fa:16:3e:c1:01:43,dl_dst=fa:16:3e:d6:3f:31
Flow:
tcp,in_port=8313,vlan_tci=0x,dl_src=fa:16:3e:c1:01:43,dl_dst=fa:16:3e:d6:3f:31,nw_src=0.0.0.0,nw_dst=0.0.0.0,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=0,tcp_flags=0
bridge("br-int")
0. in_port=8313, priority 9, cookie 0x9a67096130ac45c2
goto_table:25
25. in_port=8313,dl_src=fa:16:3e:c1:01:43, priority 2, cookie 0x9a67096130ac45c2
goto_table:60
60. in_port=8313,dl_src=fa:16:3e:c1:01:43, priority 9, cookie 0x9a67096130ac45c2
resubmit(,61)
61.
in_port=8313,dl_src=fa:16:3e:c1:01:43,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00,
priority 10, cookie 0x9a67096130ac45c2
push_vlan:0x8100
set_field:4098->vlan_vid
output:1
bridge("br-ext")
0. in_port=2, priority 2, cookie 0xab09adf2af892674
goto_table:1
1. priority 0, cookie 0xab09adf2af892674
goto_table:2
2. in_port=2,dl_vlan=2, priority 4, cookie 0xab09adf2af892674
set_field:4240->vlan_vid
NORMAL
-> forwarding to learned port
bridge("br-vlan")
-
0. priority 1, cookie 0x651552fc69601a2d
goto_table:3
3. priority 1, cookie 0x651552fc69601a2d
NORMAL
-> forwarding to learned port
Final flow:
tcp,in_port=8313,dl_vlan=2,dl_vlan_pcp=0,vlan_tci1=0x,dl_src=fa:16:3e:c1:01:43,dl_dst=fa:16:3e:d6:3f:31,nw_src=0.0.0.0,nw_dst=0.0.0.0,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=0,tcp_flags=0
Megaflow:
recirc_id=0,eth,ip,in_port=8313,vlan_tci=0x/0x1fff,dl_src=fa:16:3e:c1:01:43,dl_dst=fa:16:3e:d6:3f:31,nw_frag=no
Datapath actions: push_vlan(vid=144,pcp=0),51
Because it took output: action from table=61, added by fix
explicitly_egress_direct, the local MAC is not learned. But on ingress, the
packet is hitting table=60's NORMAL action, causing it to be flooded because it
never knows where to send the local MAC.
sudo ovs-appctl ofproto/trace br-int
in_port=1,dl_vlan=144,dl_src=fa:16:3e:d6:3f:31,dl_dst=fa:16:3e:c1:01:43
Flow:
in_port=1,dl_vlan=144,dl_vlan_pcp=0,vlan_tci1=0x,dl_src=fa:16:3e:d6:3f:31,dl_dst=fa:16:3e:c1:01:43,dl_type=0x
bridge("br-int")
0. in_port=1,dl_vlan=144, priority 3, cookie 0x9a67096130ac45c2
set_field:4098->vlan_vid
goto_table:60
60. priority 3, cookie 0x9a67096130ac45c2
NORMAL
-> no learned MAC for destination, flooding
bridge("br-vlan")
-
0. in_port=4, priority 2, cookie 0x651552fc69601a2d
goto_table:1
1. priority 0, cookie 0x651552fc69601a2d
goto_table:2
2. in_port=4, priority 2, cookie 0x651552fc69601a2d
drop
bridge("br-tun")
0. in_port=1, priority 1, cookie 0xf1baf24d000c6f7c
goto_table:1
1. priority 0, cookie 0xf1baf24d000c6f7c
goto_table:2
2. dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 0, cookie
0xf1baf24d000c6f7c
goto_table:20
20. priority 0, cookie 0xf1baf24d000c6f7c
goto_table:22
22. priority 0, cookie 0xf1baf24d000c6f7c
drop
Final flow:
in_port=1,dl_vlan=2,dl_vlan_pcp=0,vlan_tci1=0x,dl_src=fa:16:3e:d6:3f:31,dl_dst=fa:16:3e:c1:01:43,dl_type=0x
Megaflow:
recirc_id=0,eth,in_port=1,dl_vlan=144,dl_vlan_pcp=0,dl_src=fa:16:3e:d6:3f:31,dl_dst=fa:16:3e:c1:01:43,dl_type=0x
Datapath actions:
pop_vlan,push_vlan(vid=2,pcp=0),7,pop_vlan,46,26,57,58,13,6,61,66,68,22,23,72,78,79,34,81,83,2,18,87,33,88,90,91,94,95,99,100,101,102,103,106,108,113,115,116,125,132,133,134,144,145,146,147,165,168,169,170,173,174,175,178,201,203,204,205,216,222,148,150,200,160,181,54,159,151,110,182,114,233,241,212,238,154,11,213,70,29,37,131,45,93,14,139,48,105,152,129,28,12,107,172,196,3,4,62,40,183,124,20,32,67,82,135,153,84,98,109,111,123,5,65,119,120,104,122,128,130,137,142,143,121,141,176,177,179,184,186,190
dump-flows br-int indicates it first hits this rule:
cookie=0x6832197111786c03, duration=107845.507s, table=0,
n_packets=98500552445, n_bytes=66585173373354, idle_age=0,
hard_age=65534, priority=3,in_port=1,dl_vlan=144
actions=mod_vlan_vid:1,resubmit(,60)
then at table=60, the only rule it matches is the final NORMAL r
[Yahoo-eng-team] [Bug 1878719] [NEW] DHCP Agent's iptables CHECKSUM rule causes skb_warn_bad_offload kernel
Public bug reported: We are hitting this kernel issue due to a DHCP agent CHECKSUM rule that is probably obsolete/not needed: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1840619 Upgrading the kernel is one workaround, but more disruptive, especially since still using CentOS7, and kernel fix only made it into 4.19. We should just remove this rule altogether. As per the kernel issue: "The changes are limited only to users which have CHECKSUM rules enabled in their iptables configs. Openstack commonly configures such rules on deployment, even though they are not necessary, as almost all packets have their checksum calculated by NICs these days, and CHECKSUM is only around to service old dhcp clients which would discard UDP packets with empty checksums. This commit was selected for upstream -stable 4.18.13, and has made its way into bionic 4.15.0-58.64 by LP #1836426. There have been no reported problems and those kernels would have had sufficient testing with Openstack and its configured iptables rules. If any users are affected by regression, then they can simply delete any CHECKSUM entries in their iptables configs." I can see the metadata agent's CHECKSUM rule was alreayd removed last year: https://github.com/openstack/neutron/commit/04e995be9898ceaa009344509dc16ca7f589d814 Is there any reason the DHCP agent's was not? Is it safe to just remove this function and where it is invoked from altogether? https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L1739 https://github.com/openstack/neutron/blob/cb55643a0695ebc5b41f50f6edb1546bcc676b71/neutron/agent/linux/dhcp.py#L1691 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1878719 Title: DHCP Agent's iptables CHECKSUM rule causes skb_warn_bad_offload kernel Status in neutron: New Bug description: We are hitting this kernel issue due to a DHCP agent CHECKSUM rule that is probably obsolete/not needed: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1840619 Upgrading the kernel is one workaround, but more disruptive, especially since still using CentOS7, and kernel fix only made it into 4.19. We should just remove this rule altogether. As per the kernel issue: "The changes are limited only to users which have CHECKSUM rules enabled in their iptables configs. Openstack commonly configures such rules on deployment, even though they are not necessary, as almost all packets have their checksum calculated by NICs these days, and CHECKSUM is only around to service old dhcp clients which would discard UDP packets with empty checksums. This commit was selected for upstream -stable 4.18.13, and has made its way into bionic 4.15.0-58.64 by LP #1836426. There have been no reported problems and those kernels would have had sufficient testing with Openstack and its configured iptables rules. If any users are affected by regression, then they can simply delete any CHECKSUM entries in their iptables configs." I can see the metadata agent's CHECKSUM rule was alreayd removed last year: https://github.com/openstack/neutron/commit/04e995be9898ceaa009344509dc16ca7f589d814 Is there any reason the DHCP agent's was not? Is it safe to just remove this function and where it is invoked from altogether? https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L1739 https://github.com/openstack/neutron/blob/cb55643a0695ebc5b41f50f6edb1546bcc676b71/neutron/agent/linux/dhcp.py#L1691 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1878719/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866139] [NEW] GARP not sent on provider network after live migration
Public bug reported: Using Rocky, with OVS. Live migrated a VM on regular VLAN based provider network. Network connectivity was stopped, no GARP packets observed on tcpdump. Things started working after VM initiated traffic, causing MAC to be relearned. Looking at the code, send_ip_addr_adv_notif(), in ip_lib.py is responsible for using arping utility to send out a GARP. But this is only referenced in l3-agent code. This is a provider network. No routers, no floating IPs. I see this very old bug in OVN: https://bugs.launchpad.net/networking-ovn/+bug/1545897 But we are not using OVN, and that fix was fixed in OVN code itself. This is Openstack with OVS agent. How is live migration and GARP handled for fixed IPs? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1866139 Title: GARP not sent on provider network after live migration Status in neutron: New Bug description: Using Rocky, with OVS. Live migrated a VM on regular VLAN based provider network. Network connectivity was stopped, no GARP packets observed on tcpdump. Things started working after VM initiated traffic, causing MAC to be relearned. Looking at the code, send_ip_addr_adv_notif(), in ip_lib.py is responsible for using arping utility to send out a GARP. But this is only referenced in l3-agent code. This is a provider network. No routers, no floating IPs. I see this very old bug in OVN: https://bugs.launchpad.net/networking-ovn/+bug/1545897 But we are not using OVN, and that fix was fixed in OVN code itself. This is Openstack with OVS agent. How is live migration and GARP handled for fixed IPs? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1866139/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1864711] [NEW] DHCP port rescheduling causes ports to grow, internal DNS to be broken
Public bug reported: Suppose we have DHCP servers per network 2. And we have a # of DHCP agents > 2. During a time of network instability, RabbitMQ issues, or even a DHCP host temporarily going down the DHCP port will get rescheduled. Except it looks like it's not so much as getting rescheduled, but a brand new port with IP/MAC is created on a new host. The old port is only updated and marked as reserved, not deleted. This causes two issues: 1. The # of DHCP ports grows. Even when the old host starts heartbeating again, it's port is not deleted. For example we had an environment with 3 DHCP servers per network, and a dozen or so DHCP hosts. It was observed that for some networks, there were 10+ DHCP ports allocated. 2. DNS is broken temporarily for VMs that still point to the old IPs. /etc/resolv.conf can only store 3 servers, and either way, Linux's 5 second default DNS timeout means the first server going down or second server going down causes a 5+ or 10+ delay, which breaks many other apps. I'm not sure if this is a bug, or by design. For example if the same IP/mac were re-used, we could have a conflict on the data plane. Neutron-server has no idea if DHCP/DNS services are actually down - it just knows it's not receiving heartbeats over the control plane. Is that why a new port is allocated? Prefer to mitigate the risk of conflict? As for why the old ports aren't deleted or scaled down when connectivity is restored, is this by design too? ** Affects: neutron Importance: Undecided Status: New ** Tags: dns l3-ipam-dhcp -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1864711 Title: DHCP port rescheduling causes ports to grow, internal DNS to be broken Status in neutron: New Bug description: Suppose we have DHCP servers per network 2. And we have a # of DHCP agents > 2. During a time of network instability, RabbitMQ issues, or even a DHCP host temporarily going down the DHCP port will get rescheduled. Except it looks like it's not so much as getting rescheduled, but a brand new port with IP/MAC is created on a new host. The old port is only updated and marked as reserved, not deleted. This causes two issues: 1. The # of DHCP ports grows. Even when the old host starts heartbeating again, it's port is not deleted. For example we had an environment with 3 DHCP servers per network, and a dozen or so DHCP hosts. It was observed that for some networks, there were 10+ DHCP ports allocated. 2. DNS is broken temporarily for VMs that still point to the old IPs. /etc/resolv.conf can only store 3 servers, and either way, Linux's 5 second default DNS timeout means the first server going down or second server going down causes a 5+ or 10+ delay, which breaks many other apps. I'm not sure if this is a bug, or by design. For example if the same IP/mac were re-used, we could have a conflict on the data plane. Neutron-server has no idea if DHCP/DNS services are actually down - it just knows it's not receiving heartbeats over the control plane. Is that why a new port is allocated? Prefer to mitigate the risk of conflict? As for why the old ports aren't deleted or scaled down when connectivity is restored, is this by design too? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1864711/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1862851] [NEW] update_floatingip_statuses: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Public bug reported:
Running Rocky, in a DVR environment
We are seeing repeated errors like this in neutron-server logs. Not sure
at this point what the effect is, or harmless. This is in a build test
environment where a lot of automated VMs get spawned and deleted then
associated/disassociated with existing floating IPs, as well as a lot of
VMs and floating IPs that are created and deleted manually by user. The
logs aren't indicating what the floating IP is, what host, or what
port/VM they were bound to: http://paste.openstack.org/show/789444/
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api
[req-3a00ad99-d6ed-46cc-9f18-6835854799c8 - - - - -] DB exceeded retry limit.:
StaleDataError: UPDATE statement on table 'standardattributes' expected to
update 1 row(s); 0 were matched.
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api Traceback (most recent call
last):
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 142, in
wrapper
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api return f(*args,
**kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183,
in wrapped
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api LOG.debug("Retry
wrapper got retriable exception: %s", e)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api self.force_reraise()
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179,
in wrapped
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api return f(*dup_args,
**dup_kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py",
line 273, in update_floatingip_statuses
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api status)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/api.py", line 123, in
wrapped
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api return method(*args,
**kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1524,
in update_floatingip_status
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api context, {'status':
status}, id=floatingip_id)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line
669, in update_object
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api context, values,
validate_filters=False, **kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line
304, in update_object
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api
cls._update_objects(obj, values)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line
296, in _update_objects
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api obj.update()
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line
342, in decorator
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api return func(self,
*args, **kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line
876, in update
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api
self._get_composite_keys()))
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/db/api.py", line
82, in update_object
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api
db_obj.save(session=context.session)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py",
line 50, in save
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api session.flush()
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
2254, in flush
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api self._flush(objects)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api File
"/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
2380, in _flush
2020-02-11 19:38:52,839
[Yahoo-eng-team] [Bug 1853071] [NEW] AMQP disconnects, q-reports-plugin queue grows, leading to DBDeadlocks while trying to update agent heartbeats
Public bug reported:
Since upgrading to Rocky, we have seen this issue pop up in several
environments, small and large. First we see various AMQP/Rabbit related
errors - missed heartbeats from neutron-server to rabbitmq, then
repeated errors such as Socket Closed, Broken Pipe, etc...
This continues on for a while and all agents report as dead. On the
agent side, we see RPC timeouts when trying to report state. Meanwhile,
the q-reports-plugin queue in rabbit grows, to 10k+ - presumably because
neutron-server can't connect to Rabbit and process messages.
Eventually sometime later, we see "DBDeadlock:
(_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; try
restarting transaction')" errors when neutron-server is trying to update stale
agent heartbeats.
Example of various AMQP related errors - all slightly different:
2019-11-18 07:38:55,200.200 22488 ERROR
oslo.messaging._drivers.impl_rabbit [req-cba0d0fa-8e5a-42f1-a93b-
4bb398b22275 - - - - -] [eba472a9-021d-4738-801b-7944aad3e3af] AMQP
server 127.0.0.1:5672 closed the connection. Check login credentials:
Socket closed: IOError: Socket closed
2019-11-18 07:40:22,454.454 22489 ERROR
oslo.messaging._drivers.impl_rabbit [-] [6eb8d074-02c7-4add-
8d91-768cbae60fdc] AMQP server on 127.0.0.1:5672 is unreachable: Too
many heartbeats missed. Trying again in 1 seconds.: ConnectionForced:
Too many heartbeats missed
2019-11-18 07:40:22,586.586 22489 ERROR
oslo.messaging._drivers.impl_rabbit [req-0b9f092f-
27f2-4be1-bdf5-2c5208e54321 - - - - -] [4b43df2c-cc3e-4442-807c-
dcfd4057cb3d] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 32]
Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe
2019-11-18 07:42:06,010.010 22487 WARNING
oslo.messaging._drivers.impl_rabbit [-] Unexpected error during
heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2019-11-18 07:58:26,692.692 22489 WARNING
oslo.messaging._drivers.impl_rabbit [-] Unexpected error during
heartbeart thread processing, retrying...: IOError: Socket closed
2019-11-18 07:58:26,696.696 22489 ERROR
oslo.messaging._drivers.impl_rabbit [-]
[84273ffb-1610-44b1-aff7-d5e4606b7f59] AMQP server on 127.0.0.1:5672 is
unreachable: . Trying again in 1 seconds.:
RecoverableConnectionError:
Along with following Broken Pipe stacktrace in oslo messaging:
http://paste.openstack.org/show/786312/
This continues for some time (30 min - 1 hour) and all agents report as
dead, and we see following errors in rabbitmq broker logs: First missed
heartbeat errors, then handshake_timeout errors:
2019-11-18 07:41:01.448 [error] <0.6126.71> closing AMQP connection <0.6126.71>
(127.0.0.1:39817 -> 127.0.0.1:5672 -
neutron-server:22487:ee468e25-42d7-45b8-aea0-4f6fb58a9034):
missed heartbeats from client, timeout: 60s
2019-11-18 07:41:07.665 [error] <0.18727.72> closing AMQP connection
<0.18727.72> (127.0.0.1:51762 -> 127.0.0.1:5672):
{handshake_timeout,frame_header}
Eventually we see Rabbitmq q-reports has grown and neutron reporting
following DBDeadlock stacktrace:
2019-11-18 08:51:14,505.505 22493 ERROR oslo_db.api [req-
231004a2-d988-47b3-9730-d6b5276fdcf8 - - - - -] DB exceeded retry
limit.: DBDeadlock: (_mysql_exceptions.OperationalError) (1205, 'Lock
wait timeout exceeded; try restarting transaction') [SQL: u'UPDATE
agents SET heartbeat_timestamp=%s WHERE agents.id = %s'] [parameters:
(datetime.datetime(2019, 11, 18, 8, 50, 23, 804716), '223c754e-9d7f-
4df3-b5a5-9be4eb8692b0')] (Background on this error at:
http://sqlalche.me/e/e3q8)
Full stacktrace here: http://paste.openstack.org/show/786313/
The only way to recover is we stop neutron-server and rabbitmq, kill any
neutron workers still dangling (which they usually are), then restart. But then
we see problem manifest days or a week later.
Rabbitmq is on same host as neutron-server - it is all localhost
communication. So we are unsure of why it can't heartbeat or connect.
Also the subsequent DBDeadlock leads me to think there is some
syncrhonization issue when neutron gets overwhelmed with outstanding RPC
messages.
** Affects: neutron
Importance: Undecided
Status: New
** Tags: db
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1853071
Title:
AMQP disconnects, q-reports-plugin queue grows, leading to
DBDeadlocks while trying to update agent heartbeats
Status in neutron:
New
Bug description:
Since upgrading to Rocky, we have seen this issue pop up in several
environments, small and large. First we see various AMQP/Rabbit
related errors - missed heartbeats from neutron-server to rabbitmq,
then repeated errors such as Socket Closed, Broken Pipe, etc...
This continues on for a while and all agents report as dead. On the
agent side, we see RPC timeouts when trying to report state.
Meanwhile, the q-reports-plugin queue in rabbit grows, to 10k+ -
presumably beca
[Yahoo-eng-team] [Bug 1852504] [NEW] DHCP reserved ports that were unscheduled are advertised as DNS servers
Public bug reported: We have 2 DHCP servers per network. After network outages, and when hosts come back online, the number of ACTIVE DHCP servers grow. This happened again after more outages, with some networks having up to 9-10+ DHCP ports, many in ACTIVE state, despite neutron-server's neutron.conf only having dhcp_agents_per_network = 2 It turns out these are "reserved_dhcp_port" as indicated by the device_id. As you can see here: https://github.com/openstack/neutron/blob/master/neutron/db/agentschedulers_db.py#L399 When a network is rescheduled to a new DHCP agent, the old port is not deleted, not is its status marked as DOWN. All that is done is it is marked as reserved and the port updated. However VMs on the network now get advertised all the DHCP ports on the network as internal DNS servers, several stale entries in /etc/resolv.conf in our case. Problem is some of these DHCP agents have been unscheduled so the DNS servers don't actually exist. Also in the VMs, more than 3 entries are not queried. As you can see here, is resolv.conf on a VM: [root@arjunpmk-master ~]# vim /etc/resolv.conf # Generated by NetworkManager search mpt1.pf9.io nameserver 10.128.144.16 nameserver 10.128.144.23 nameserver 10.128.144.15 # NOTE: the libc resolver may not support more than 3 nameservers. # The nameservers listed below may not be recognized. nameserver 10.128.144.7 nameserver 10.128.144.4 nameserver 10.128.144.8 nameserver 10.128.144.9 nameserver 10.128.144.17 nameserver 10.128.144.12 nameserver 10.128.144.45 nameserver 10.128.144.46 nameserver 10.128.144.51 Here you can see all the DHCP ports for the network of this VM: [root@df-us-mpt1-kvm arjun(admin)]# openstack port list --network ead88ed3-f1e0-4498-8c1e-6d091083ae33 --device-owner network:dhcp +--+--+---+--++ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--+--+---+--++ | 02ff0f4c-f39d-4207-90b4-2a69585f4c8a | | fa:16:3e:a9:36:82 | ip_address='10.128.144.16', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | 0b612f86-ad06-4bce-a333-bc18f3e9e7b1 | | fa:16:3e:bb:d8:3d | ip_address='10.128.144.23', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | DOWN | | 402338ac-2ca6-4312-a2df-a306fc589f10 | | fa:16:3e:a3:a8:57 | ip_address='10.128.144.15', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | 5d2edc73-4eff-44c0-8993-125636973384 | | fa:16:3e:6c:cd:2b | ip_address='10.128.144.7', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | 78241da3-9674-479a-8b45-a580c7f8b117 | | fa:16:3e:d0:9d:ef | ip_address='10.128.144.4', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | 7b41bf47-d4d4-434a-b704-4c67182ffcaa | | fa:16:3e:4c:cf:54 | ip_address='10.128.144.8', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | 96897190-1aa8-4c17-a7d1-c3744f1bf962 | | fa:16:3e:e8:55:29 | ip_address='10.128.144.45', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | af87dde6-fb46-4516-9569-e46496398b64 | | fa:16:3e:0e:61:14 | ip_address='10.128.144.9', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | c2a2112d-c6ef-4411-a415-1a453d74a838 | | fa:16:3e:d0:39:67 | ip_address='10.128.144.46', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | DOWN | | c8298fbd-06e7-4488-a3e1-874e9341d4cf | | fa:16:3e:d6:3c:ac | ip_address='10.128.144.51', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | DOWN | | d6f0206f-ae3c-4ebf-95cb-104dad786724 | | fa:16:3e:ab:ab:22 | ip_address='10.128.144.17', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | ACTIVE | | e2be0f98--4645-b58a-435e5513a4d3 | | fa:16:3e:b4:ba:c0 | ip_address='10.128.144.12', subnet_id='9757ae4a-ccfb-49b0-a9cc-53b8664631a6' | DOWN | +--+--+---+--++ If I view the first DNS server for the VM's resolv.conf (10.128.144.16), you can see its status is ACTIVE but its actually a reserved port. This is the same case for 2nd nameserver entry. Luckily the 3rd entry is valid, but this causes timeouts and all DNS lookups to take 10 seconds since first two fail. VMs on other networks aren't so lucky, where all 3 nameservers are reserved. Expectation: Only DHCP ports that are actually scheduled (not reserved) should be advertised as DNS nameservers. I don't know if this means marking the port as DOWN, or deleting the port when unscheduled. maybe status needs to also be updated here? https://github.com/openstack/neutron/blob/master
[Yahoo-eng-team] [Bug 1838699] [NEW] Removing a subnet from DVR router also removes DVR MAC flows for other router on network
Public bug reported: This bug builds on issue seen in https://bugs.launchpad.net/neutron/+bug/1838697 In that issue, if you create a tenant network, some VMs, and attach it to 2 DVR routers, only the DVR MAC rules exist for the first router. With this issue, simply removing the subnet or deleting the second router ends up deleting all the DVR MAC flows for the first router. It deleted both the table=1 and table=60 rules for ALL local endpoints on that network. For example: fa:16:3e:ce:f8:cd = MAC of a VM on this particular host fa:16:3e:5c:44:da = MAC of router_interface_distributed port of 1st router fa:16:3e:19:67:9e = MAC of router_interface_distributed port on 2nd router When simple network is attached to 2 routers: [r...@pf9-kvm-neutron.platform9.net arjun(admin)]# openstack port list --network 8cd0e19e-9041-4a62-9cc9-6bfb5b10f955 --long +--+--+---+-++--+--+--+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--+--+---+-++--+--+--+ | 16e971ae-0ce9-4f4a-aaab-6ab3fc71bf93 | | fa:16:3e:79:66:c8 | ip_address='10.23.23.9', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | 1ef66f53-7818-4281-b407-9be7d55b3b17 | | fa:16:3e:ce:f8:cd | ip_address='10.23.23.7', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | 21553560-5491-4036-9d03-65d7bedb28dc | | fa:16:3e:0a:ff:1b | ip_address='10.23.23.2', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:dhcp | | | 386d3d98-6c86-4748-9c2e-8b60fbe3f6cc | | fa:16:3e:c9:19:14 | ip_address='10.23.23.25', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | 4e211475-91e0-4627-8342-837210219fbc | | fa:16:3e:19:67:9e | ip_address='10.23.23.199', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | ecd04202-0111-4e29-8e2f-39a203123c75 | network:router_interface_distributed | | | 7be10a79-e581-4ba9-95c9-870e845dbea0 | | fa:16:3e:0b:9b:e3 | ip_address='10.23.23.28', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | be9d8d83-0c55-49aa-836e-bb4f483bde48 | | fa:16:3e:21:76:67 | ip_address='10.23.23.4', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:dhcp | | | d266f85c-14b1-4c47-a357-44cd0fa4b557 | | fa:16:3e:c4:f0:ce | ip_address='10.23.23.3', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:dhcp | | | de2fb0b6-9756-4418-8501-be202afbf006 | | fa:16:3e:e7:f6:6c | ip_address='10.23.23.14', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | f00fa134-da4d-4663-8d94-52de0840f9d4 | | fa:16:3e:2e:3c:8a | ip_address='10.23.23.5', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | f33d9ba4-cfdc-42f3-aff4-e5221f84ac03 | | fa:16:3e:c9:86:97 | ip_address='10.23.23.6', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | f763ba3f-fae2-4608-8ef9-10ccc023eacc | | fa:16:3e:5c:44:da | ip_address='10.23.23.1', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:router_interface_distributed | | +--+--+---+-++--+--+--+ [root@chef ~]# ovs-ofctl dump-flows br-int | grep fa:16:3e:ce:f8:cd cookie=0xbdf055421ffc2398, duration=222.793s, table=1, n_packets=0, n_bytes=0, idle_age=1843, priority=4,dl_vlan=13,dl_dst=fa:16:3e:ce:f8:cd actions=mod_dl_src:fa:16:3e:5c:44:da,resubm
[Yahoo-eng-team] [Bug 1838697] [NEW] DVR Mac conversion rules are only added for the first router a network is attached to
Public bug reported: This is seen on stable/pike, have not checked latest or stein. 1. Create a basic tenant network and create a DVR router, attach them. Spin up some VMs: [r...@pf9-kvm-neutron.platform9.net arjun(admin)]# openstack port list --network 8cd0e19e-9041-4a62-9cc9-6bfb5b10f955 --long +--+--+---+++--+--+--+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--+--+---+++--+--+--+ | 16e971ae-0ce9-4f4a-aaab-6ab3fc71bf93 | | fa:16:3e:79:66:c8 | ip_address='10.23.23.9', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | 1ef66f53-7818-4281-b407-9be7d55b3b17 | | fa:16:3e:ce:f8:cd | ip_address='10.23.23.7', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | 21553560-5491-4036-9d03-65d7bedb28dc | | fa:16:3e:0a:ff:1b | ip_address='10.23.23.2', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:dhcp | | | 386d3d98-6c86-4748-9c2e-8b60fbe3f6cc | | fa:16:3e:c9:19:14 | ip_address='10.23.23.25', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | 7be10a79-e581-4ba9-95c9-870e845dbea0 | | fa:16:3e:0b:9b:e3 | ip_address='10.23.23.28', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | be9d8d83-0c55-49aa-836e-bb4f483bde48 | | fa:16:3e:21:76:67 | ip_address='10.23.23.4', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:dhcp | | | d266f85c-14b1-4c47-a357-44cd0fa4b557 | | fa:16:3e:c4:f0:ce | ip_address='10.23.23.3', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:dhcp | | | de2fb0b6-9756-4418-8501-be202afbf006 | | fa:16:3e:e7:f6:6c | ip_address='10.23.23.14', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | f00fa134-da4d-4663-8d94-52de0840f9d4 | | fa:16:3e:2e:3c:8a | ip_address='10.23.23.5', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | f33d9ba4-cfdc-42f3-aff4-e5221f84ac03 | | fa:16:3e:c9:86:97 | ip_address='10.23.23.6', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | bd5274ad-2ff9-443a-9226-473cf129e915 | compute:None | | | f763ba3f-fae2-4608-8ef9-10ccc023eacc | | fa:16:3e:5c:44:da | ip_address='10.23.23.1', subnet_id='f012101e-91ac-4b85-947e-0f9eca83d5e8' | ACTIVE | | network:router_interface_distributed | | +--+--+---+++--+--+--+ 2. Check on a host where one of the VMs is. Search for the router port's MAC in br-int. For inbound packets (arriving from br-tun), there is a table=1 rule that converts from remote per-host DVR MAC to the common MAC of the network:router_distributed port: [root@chef ~]# ovs-ofctl dump-flows br-int | grep fa:16:3e:5c:44:da cookie=0xbdf055421ffc2398, duration=256.770s, table=1, n_packets=0, n_bytes=0, idle_age=1877, priority=4,dl_vlan=13,dl_dst=fa:16:3e:ce:f8:cd actions=mod_dl_src:fa:16:3e:5c:44:da,resubmit(,60) 3. Now create a 2nd router. Attach the same network to this router. Now notice this network has 2 network:router_interface_distributed ports. But the DVR MAC conversion rules are missing for this other router MAC. Only first one is present: [r...@pf9-kvm-neutron.platform9.net arjun(admin)]# openstack port list --network 8cd0e19e-9041-4a62-9cc9-6bfb5b10f955 --long +--+--+---+-++-
[Yahoo-eng-team] [Bug 1823798] [NEW] rfp and fpr interfaces not updated after changing network MTU
Public bug reported: I have a tenant network attached to a router and external network, and some VMs with Floating IPs deployed. I updated the network MTU from 1500 to 1300 (just as a test, but try any other MTU), and restarted nova compute, ovs agent, l3-agent. All interfaces (qvo/qvb/qbr, qr, dhcp taps, sg, fg, qg) got updated except for the rfp-fpr peer link between qrouter and fip namespace: [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ifconfig | grep mtu br-ex: flags=4163 mtu 1500 ens32: flags=4163 mtu 1500 ens34: flags=4163 mtu 1500 ens35: flags=4163 mtu 1500 ens34.1801: flags=4163 mtu 1500 lo: flags=73 mtu 65536 qbr05d85481-ca: flags=4163 mtu 1300 qbr147c3499-25: flags=4163 mtu 1300 qbrbf9399bf-17: flags=4163 mtu 1300 qbrf1ba8ee0-e1: flags=4163 mtu 1300 qvb05d85481-ca: flags=4419 mtu 1300 qvb147c3499-25: flags=4419 mtu 1300 qvbbf9399bf-17: flags=4419 mtu 1300 qvbf1ba8ee0-e1: flags=4419 mtu 1300 qvo05d85481-ca: flags=4419 mtu 1300 qvo147c3499-25: flags=4419 mtu 1300 qvobf9399bf-17: flags=4419 mtu 1300 qvof1ba8ee0-e1: flags=4419 mtu 1300 tap05d85481-ca: flags=4163 mtu 1500 tap147c3499-25: flags=4163 mtu 1500 tapbf9399bf-17: flags=4163 mtu 1500 tapf1ba8ee0-e1: flags=4163 mtu 1500 virbr0: flags=4099 mtu 1500 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec qrouter-b57abb6d-92ff-4ebd-a71f-6b26646eddfa ifconfig | grep mtu lo: flags=73 mtu 65536 qr-9f725d28-82: flags=4163 mtu 1300 qr-a9852f10-d1: flags=4163 mtu 1300 rfp-b57abb6d-9: flags=4163 mtu 1500 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec qdhcp-ada1b967-9604-42ea-88bc-d887ce3e877e ifconfig | grep mtu lo: flags=73 mtu 65536 tap8ecd0a51-63: flags=4163 mtu 1300 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec fip-4ff6386b-c547-465e-a734-49eed1ae6832 ifconfig | grep mtu fg-a7668bc9-97: flags=4163 mtu 1300 fpr-b57abb6d-9: flags=4163 mtu 1500 lo: flags=73 mtu 65536 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec snat-b57abb6d-92ff-4ebd-a71f-6b26646eddfa ifconfig | grep mtu lo: flags=73 mtu 65536 qg-d69632ce-e4: flags=4163 mtu 1300 sg-83643ed0-34: flags=4163 mtu 1300 sg-9eac9eee-69: flags=4163 mtu 1300 VM taps are still at 1500, but will change if I powercycle VMs or migrate. rfp/fprs remain at 1500 and unsure of how to get their MTUs to update ** Affects: neutron Importance: Undecided Status: New ** Tags: l3-dvr-backlog -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1823798 Title: rfp and fpr interfaces not updated after changing network MTU Status in neutron: New Bug description: I have a tenant network attached to a router and external network, and some VMs with Floating IPs deployed. I updated the network MTU from 1500 to 1300 (just as a test, but try any other MTU), and restarted nova compute, ovs agent, l3-agent. All interfaces (qvo/qvb/qbr, qr, dhcp taps, sg, fg, qg) got updated except for the rfp-fpr peer link between qrouter and fip namespace: [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ifconfig | grep mtu br-ex: flags=4163 mtu 1500 ens32: flags=4163 mtu 1500 ens34: flags=4163 mtu 1500 ens35: flags=4163 mtu 1500 ens34.1801: flags=4163 mtu 1500 lo: flags=73 mtu 65536 qbr05d85481-ca: flags=4163 mtu 1300 qbr147c3499-25: flags=4163 mtu 1300 qbrbf9399bf-17: flags=4163 mtu 1300 qbrf1ba8ee0-e1: flags=4163 mtu 1300 qvb05d85481-ca: flags=4419 mtu 1300 qvb147c3499-25: flags=4419 mtu 1300 qvbbf9399bf-17: flags=4419 mtu 1300 qvbf1ba8ee0-e1: flags=4419 mtu 1300 qvo05d85481-ca: flags=4419 mtu 1300 qvo147c3499-25: flags=4419 mtu 1300 qvobf9399bf-17: flags=4419 mtu 1300 qvof1ba8ee0-e1: flags=4419 mtu 1300 tap05d85481-ca: flags=4163 mtu 1500 tap147c3499-25: flags=4163 mtu 1500 tapbf9399bf-17: flags=4163 mtu 1500 tapf1ba8ee0-e1: flags=4163 mtu 1500 virbr0: flags=4099 mtu 1500 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec qrouter-b57abb6d-92ff-4ebd-a71f-6b26646eddfa ifconfig | grep mtu lo: flags=73 mtu 65536 qr-9f725d28-82: flags=4163 mtu 1300 qr-a9852f10-d1: flags=4163 mtu 1300 rfp-b57abb6d-9: flags=4163 mtu 1500 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec qdhcp-ada1b967-9604-42ea-88bc-d887ce3e877e ifconfig | grep mtu lo: flags=73 mtu 65536 tap8ecd0a51-63: flags=4163 mtu 1300 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec fip-4ff6386b-c547-465e-a734-49eed1ae6832 ifconfig | grep mtu fg-a7668bc9-97: flags=4163 mtu 1300 fpr-b57abb6d-9: flags=4163 mtu 1500 lo: flags=73 mtu 65536 [root@dvr-tn-vl-c7-10-4-211-218platform9 ~]# ip netns exec snat-b57abb6d-92ff-4ebd-a71f-6b26646eddfa ifconfig | grep mtu lo: flags=73 mtu 65536 qg-d69632ce-e4: flags=4163 mtu 1300 sg-83643ed0-34: flags=4163 mtu 1300 sg-9eac9eee-69: flags=4163 mtu 1300 VM taps are still at 1500, but will change
[Yahoo-eng-team] [Bug 1806770] [NEW] DHCP Agent should not release DHCP lease when client ID is not set on port
Public bug reported: DHCP agent has a really strict enforcement of client ID, which is part of the DHCP extra options. If a VM advertises a client ID, DHCP agent will automatically release it's lease whenever *any* other port is updated/deleted. This happens even if no client ID is set on the port. When reload_allocations() is called, DHCP agent parses the current leases file, the hosts file, and gets the list all the ports in the network from DB, computing 3 different sets. The set from leases file (v4_leases) will have some client ID. The set from port DB will have None. As a result the set subtraction does not filter out the entry, and the port's DHCP lease is constantly released, whenever the VM renews its lease and any other port in the network is deleted: https://github.com/openstack/neutron/blob/stable/pike/neutron/agent/linux/dhcp.py#L850 v4_leases = set() for (k, v) in cur_leases.items(): # IPv4 leases have a MAC, IPv6 ones do not, so we must ignore if netaddr.IPAddress(k).version == constants.IP_VERSION_4: # treat '*' as None, see note in _read_leases_file_leases() client_id = v['client_id'] if client_id is '*': client_id = None v4_leases.add((k, v['iaid'], client_id)) new_leases = set() for port in self.network.ports: client_id = self._get_client_id(port) for alloc in port.fixed_ips: new_leases.add((alloc.ip_address, port.mac_address, client_id)) # If an entry is in the leases or host file(s), but doesn't have # a fixed IP on a corresponding neutron port, consider it stale. entries_to_release = (v4_leases | old_leases) - new_leases if not entries_to_release: return I believe that this client ID and releasing of lease should only occur if a client id is set in the port's DHCP Extra opts and there is a mismatch. Otherwise, ignore whatever client ID the VM advertises. This can cause issues where when the VM later asks to renew its lease when the expiry period is coming up (I think about halfway thru), dnsmasq sends an DHCP NAK and the lease is re-negotiated and existing networking connections can get disrupted. It also just causes DHCP agent to do unneccessary work, releasing a ton of leases when it technically shouldn't. Setting the client ID in the port's DHCP extra opts is not an good solution: 1. In some cases, like Windows VMs, the client ID is advertised as the MAC by default. In fact, there is a Windows bug which prevents you from even turning this off: https://support.microsoft.com/en-us/help/3004537 /dhcp-client-always-includes-option-61-in-the-dhcp-request-in-windows-8 Linux VMs dont have this on by default, when I checked, but they may be enabled in some templates unknown to users 2. End users will usually just be deploying a VM, with the port being auto created by Nova. They don't know or need to know about advanced networking concepts like DHCP client IDs. 3. We can't expect everyone to modify their existing app templates, or end users to make API calls, to update ports everytime they deploy a VM So, client ID should only be enforce, and leases released, if it's actually set on the port. In that case it means someone knows what they are doing, and we want to check for a mismatch. If its None, I suspect in 99.% of cases the operator does not know or care about client ID field. ** Affects: neutron Importance: Undecided Assignee: Arjun Baindur (abaindur) Status: New ** Tags: l3-ipam-dhcp ** Changed in: neutron Assignee: (unassigned) => Arjun Baindur (abaindur) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1806770 Title: DHCP Agent should not release DHCP lease when client ID is not set on port Status in neutron: New Bug description: DHCP agent has a really strict enforcement of client ID, which is part of the DHCP extra options. If a VM advertises a client ID, DHCP agent will automatically release it's lease whenever *any* other port is updated/deleted. This happens even if no client ID is set on the port. When reload_allocations() is called, DHCP agent parses the current leases file, the hosts file, and gets the list all the ports in the network from DB, computing 3 different sets. The set from leases file (v4_leases) will have some client ID. The set from port DB will have None. As a result the set subtraction does not filter out the entry, and the port's DHCP lease is constantly released, whenever the VM renews its lease and any other port in the network is deleted: https://github.com/openstack/neutron/blob/stable/pike/neutron/agent/linux/dhcp.py#L850 v4_lease
[Yahoo-eng-team] [Bug 1802006] [NEW] Floating IP attach/detach fails for non-admin user and unbound port with router in different tenant
Public bug reported:
Seeing this on pike, but code looks same in master so issue still likely
exists.
We have a shared external network connected to router in TenantA. Now
create a network, either shared in tenantA or owned by tenantB, and
attach to tenantA's router (an admin user will have to do this).
Now suppose a non-admin user in the different tenantB creates a Floating
IP on shared ext network. They then try to attach it to a port. It
passes if the port is bound to a VM. It fails if the port is unbound.
For example, pre-create a port on a network/subnet available to this
tenant, and then try the following /floatingips/ PUT API call. It will
fail. Then bring up a VM on same network, and attach Floating IP to it's
port, this will pass:
curl -k -X PUT -i
https://localhost/neutron/v2.0/floatingips/cc56cbb4-2c3b-4d53-9506-1baaa8e7b2d6
-H "X-Auth-Token: $TOK" -H "Content-Type: application/json" -d '{"floatingip":
{"port_id": "6af4bb1c-85b4-42c0-8b96-6efb90443aa7"}}'
HTTP/1.1 404 Not Found
Server: nginx/1.12.2
Date: Tue, 06 Nov 2018 07:31:54 GMT
Content-Type: application/json
Content-Length: 135
Connection: keep-alive
X-Openstack-Request-Id: req-31b02819-844c-4d41-a471-938f092b4a57
Access-Control-Allow-Credentials: true
{"NeutronError": {"message": "Router b819cfbb-7e8b-4bce-964e-
ec4b29614241 could not be found", "type": "RouterNotFound", "detail":
""}}
curl -k -X PUT -i
https://localhost/neutron/v2.0/floatingips/44361b51-7928-441e-8478-4fd35919e5c3
-H "X-Auth-Token: $TOK" -H "Content-Type: application/json" -d '{"floatingip":
{"port_id": "7ea1b40a-e3b1-490d-8a02-d5e2cf18b89c"}}'
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Tue, 06 Nov 2018 07:15:10 GMT
Content-Type: application/json
Content-Length: 584
Connection: keep-alive
X-Openstack-Request-Id: req-50cb5289-fcbb-4a65-91d4-33f59b0f4632
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: X-Subject-Token
{"floatingip": {"router_id": "b819cfbb-7e8b-4bce-964e-ec4b29614241",
"status": "DOWN", "description": "", "tags": [], "updated_at":
"2018-11-06T07:15:09Z", "dns_domain": "", "floating_network_id":
"6580471a-cac0-4f03-ae2e-77ddfb76b181", "fixed_ip_address":
"10.168.1.14", "floating_ip_address": "10.4.252.154", "revision_number":
16, "port_id": "7ea1b40a-e3b1-490d-8a02-d5e2cf18b89c", "id":
"44361b51-7928-441e-8478-4fd35919e5c3", "dns_name": "", "created_at":
"2018-11-06T02:53:23Z", "tenant_id": "5e09d64a521b440e9ffbec28f5fb7de0",
"project_id": "5e09d64a521b440e9ffbec28f5fb7de0"}}
Problem is due to new code which allows binding FIP to unbound ports via
SNAT router, from this diff:
https://github.com/openstack/neutron/commit/9515c771e742a5b6d29b17f84f49a0b39706489b
An additional get_router() call is made here, and it needs the elevated
admin context to be passed in. It fails because default policy for
get_router is admin_or_owner, and it can't fetch the SNAT router in
different tenant. This code path is not hit for a VM port, as it is
bound and has a host:
https://github.com/openstack/neutron/blob/master/neutron/db/l3_dvr_db.py#L1098
which invokes get_router() here:
https://github.com/openstack/neutron/blob/master/neutron/db/l3_hascheduler_db.py#L45
Need to pass in context.elevated() in either one of those 2 places -
thinking the first location might be better?
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
Seeing this on pike, but code looks same in master so issue still likely
exists.
We have a shared external network connected to router in TenantA. Now
create a network, either shared in tenantA or owned by tenantB, and
attach to tenantA's router (an admin user will have to do this).
Now suppose a non-admin user in the different tenantB creates a Floating
IP on shared ext network. They then try to attach it to a port. It
passes if the port is bound to a VM. It fails if the port is unbound.
For example, pre-create a port on a network/subnet available to this
tenant, and then try the following /floatingips/ PUT API call. It will
fail. Then bring up a VM on same network, and attach Floating IP to it's
port, this will pass:
curl -k -X PUT -i
https://localhost/neutron/v2.0/floatingips/cc56cbb4-2c3b-4d53-9506-1baaa8e7b2d6
-H "X-Auth-Token: $TOK" -H "Content-Type: application/json" -d '{"floatingip":
{"port_id": "6af4bb1c-85b4-42c0-8b96-6efb90443aa7"}}'
HTTP/1.1 404 Not Found
Server: nginx/1.12.2
Date: Tue, 06 Nov 2018 07:31:54 GMT
Content-Type: application/json
Content-Length: 135
Connection: keep-alive
X-Openstack-Request-Id: req-31b02819-844c-4d41-a471-938f092b4a57
Access-Control-Allow-Credentials: true
{"NeutronError": {"message": "Router b819cfbb-7e8b-4bce-964e-
ec4b29614241 could not be found", "type": "RouterNotFound", "detail":
""}}
curl -k -X PUT -i
https://localhost/neutron/v2.0/floatingips/44361b51-7928-441e-8478-4fd35919e5c3
-H "X-Auth-Token: $TOK" -H "Content-Type: application/json" -d '{"floatin
[Yahoo-eng-team] [Bug 1783908] [NEW] dnsmasq does not remove leases for deleted VMs - leases and host files point to different MACS
Public bug reported: We see this sporadically, sometimes it works, sometimes it doesn't. We delete VMs, then create new ones. The new fixed IP can't get an IP via DHCP, because the leases file still points to MAC of some old, deleted VMs. The host file is correctly updated. For example we see in syslog/messages: Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: not using configured address 10.96.0.21 because it is leased to fa:16:3e:ae:c4:b0 Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: DHCPDISCOVER(tap6d5fc8f6-28) fa:16:3e:2f:29:37 no address available And tcpdump on tap interface of qdhcp indicates it is receiving the DHCP requests, but nothing is sent out. On inspection of the leases and host file, we see the MAC address differs for the fixed IP: [root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21 fa:16:3e:2f:29:37,test-pf9-vmha-one-region-c7-702498-490-2.platform9.sys.,10.96.0.21 [root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21 1532649702 fa:16:3e:ae:c4:b0 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 * On another dhcp-agent host, we saw the leases file pointed to some OTHER old third MAC. We then deleted this VM. We observed that the lease was removed on one host, but it remained on the other host. So it appears to be random. Enabling DEBUG logs shows the dhcp_release command is being invoked, without error: 2018-07-26 16:52:06.214 10660 DEBUG neutron.agent.dhcp.agent [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Calling driver for network: 52f8298e-f867-4581-8ae7-fac0c9196ae4 action: reload_allocations call_driver /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:133 2018-07-26 16:52:06.217 10660 DEBUG neutron.agent.linux.utils [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Running command: ['sudo', '/opt/pf9/pf9-neutron/bin/neutron-rootwrap', '/opt/pf9/etc/pf9-neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-52f8298e-f867-4581-8ae7-fac0c9196ae4', 'dhcp_release', 'tap4d6bb403-59', '10.96.0.21', 'fa:16:3e:2f:29:37'] create_process /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92 2018-07-26 16:52:06.498 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Building host file: /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:685 2018-07-26 16:52:06.505 10660 DEBUG neutron.agent.linux.dhcp [req-22449d0c-c0e3-4571-b4f9-a3150f8596c5 5412536c76f942d980e1834f48340f77 f175f441ebbb4c2b8fedf6469d6415fc - - -] Done building host file /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host _output_hosts_file /opt/pf9/pf9-neutron/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:724 The entry was removed from hosts file, but remains in leases despite above dhcp_release remaining. If we spin up a new VM with this fixed IP, it can't get an IP due to old lease: root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/host | grep 10.96.0.21 root@krusty:~# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fac0c9196ae4/leases | grep 10.96.0.21 1532735433 fa:16:3e:2f:29:37 10.96.0.21 test-pf9-vmha-one-region-c7-702498-490-2 * root@krusty:~# ** Affects: neutron Importance: Undecided Status: New ** Tags: l3-ipam-dhcp ** Summary changed: - dnsmasq not removes leases for deleted VMs + dnsmasq does not remove leases for deleted VMs - leases and host files point to different MACS -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1783908 Title: dnsmasq does not remove leases for deleted VMs - leases and host files point to different MACS Status in neutron: New Bug description: We see this sporadically, sometimes it works, sometimes it doesn't. We delete VMs, then create new ones. The new fixed IP can't get an IP via DHCP, because the leases file still points to MAC of some old, deleted VMs. The host file is correctly updated. For example we see in syslog/messages: Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: not using configured address 10.96.0.21 because it is leased to fa:16:3e:ae:c4:b0 Jul 26 16:45:04 stan dnsmasq-dhcp[18540]: DHCPDISCOVER(tap6d5fc8f6-28) fa:16:3e:2f:29:37 no address available And tcpdump on tap interface of qdhcp indicates it is receiving the DHCP requests, but nothing is sent out. On inspection of the leases and host file, we see the MAC address differs for the fixed IP: [root@stan ~]# cat /var/opt/pf9/neutron/dhcp/52f8298e-f867-4581-8ae7-fa
[Yahoo-eng-team] [Bug 1783654] [NEW] DVR process flow not installed on physical bridge for shared tenant network
Public bug reported: Seems like collateral from https://bugs.launchpad.net/neutron/+bug/1751396 In DVR, the distributed gateway port's IP and MAC are shared in the qrouter across all hosts. The dvr_process_flow on the physical bridge (which replaces the shared router_distributed MAC address with the unique per-host MAC when its the source), is missing, and so is the drop rule which instructs the bridge to drop all traffic destined for the shared distributed MAC. Because of this, we are seeing the router MAC on the network infrastructure, causing it on flap on br-int on every compute host: root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec1 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec2 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec1 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec0 Where port 1 is phy-br-vlan, connecting to the physical bridge, and port 11 is the correct local qr-interface. Because these dvr flows are missing on br-vlan, pkts w/ source mac ingress into the host and br-int learns it upstream. The symptom is when pinging a VM's floating IP, we see occasional packet loss (10-30%), and sometimes the responses are sent upstream by br-int instead of the qrouter, so the ICMP replies come with fixed IP of the replier since no NAT'ing took place, and on the tenant network rather than external network. When I force net_shared_only to False here, the problem goes away: https://github.com/openstack/neutron/blob/stable/pike/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py#L436 It should we noted we *ONLY* need to do this on our dvr_snat host. The dvr process's are missing on every compute host. But if we shut qrouter on the snat host, FIP functionality works and DVR mac stops flapping on others. Or if we apply fix only to snat host, it works. Perhaps there is something on SNAT node that is unique ** Affects: neutron Importance: Undecided Status: New ** Tags: l3-dvr-backlog ovs -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1783654 Title: DVR process flow not installed on physical bridge for shared tenant network Status in neutron: New Bug description: Seems like collateral from https://bugs.launchpad.net/neutron/+bug/1751396 In DVR, the distributed gateway port's IP and MAC are shared in the qrouter across all hosts. The dvr_process_flow on the physical bridge (which replaces the shared router_distributed MAC address with the unique per-host MAC when its the source), is missing, and so is the drop rule which instructs the bridge to drop all traffic destined for the shared distributed MAC. Because of this, we are seeing the router MAC on the network infrastructure, causing it on flap on br-int on every compute host: root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec1 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec2 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec0 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 1 4 fa:16:3e:42:a2:ec1 root@milhouse:~# ovs-appctl fdb/show br-int | grep fa:16:3e:42:a2:ec 11 4 fa:1
[Yahoo-eng-team] [Bug 1776778] [NEW] Floating IPs broken after upgrade to Centos 7.5 - DNAT not working
Public bug reported: Since upgrading to Centos 7.5, floating IP functionality has been completely busted. Packets arrive inbound to qrouter from fip namespace via RFP, but are not DNAT'd or routed, as we see nothing going out qr- interface. For outbound packets leaving the VM, they are fine, but then all responses are again dropped inbound to qrouter after arriving on rfp. It appears the DNAT rules in the "-t nat" iptables within qrouter are not being hit (packet counters are zero). SNAT functionality works when we remove floating IP from the VM (VM can then ping outbound). So problem seems isolated to DNAT / qrouter receiving packets from fip? We are able to reproduce this 100% consistently, whenever we update our working centos 7.2 / centos 7.4 hosts to 7.5. Nothing changes except a "yum update". All routes, rules, iptables are identical on a working older host vs. broken centos 7.5 host. I added some basic rules to log packets at top of PREROUTING chain in raw, mangle, and nat tables. Filtering either by my source IP, or all packets on -i rfp ingress interface. While packet counters increment for raw and mangle, they remain at 0 for nat, indicating the nat iptable is not invoked for PREROUTING. Floating IP = 10.8.17.52, Fixed IP = 192.168.94.9. [root@centos7-neutron-template ~]# ip netns exec qrouter-f48d5536-eefa-4410-b17b-1b3d14426323 tcpdump -l -evvvnn -i rfp-f48d5536-e tcpdump: listening on rfp-f48d5536-e, link-type EN10MB (Ethernet), capture size 262144 bytes 13:42:00.345440 7a:3b:f1:c7:5d:4e > aa:24:89:9e:c8:f0, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 62, id 1832, offset 0, flags [DF], proto ICMP (1), length 84) 10.4.165.22 > 10.8.17.52: ICMP echo request, id 5771, seq 1, length 64 13:42:01.344047 7a:3b:f1:c7:5d:4e > aa:24:89:9e:c8:f0, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 1833, offset 0, flags [DF], proto ICMP (1), length 84) 10.4.165.22 > 10.8.17.52: ICMP echo request, id 5771, seq 2, length 64 13:42:02.398300 7a:3b:f1:c7:5d:4e > aa:24:89:9e:c8:f0, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 1834, offset 0, flags [DF], proto ICMP (1), length 84) 10.4.165.22 > 10.8.17.52: ICMP echo request, id 5771, seq 3, length 64 13:42:03.344345 7a:3b:f1:c7:5d:4e > aa:24:89:9e:c8:f0, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 1835, offset 0, flags [DF], proto ICMP (1), length 84) 10.4.165.22 > 10.8.17.52: ICMP echo request, id 5771, seq 4, length 64 ^C 4 packets captured 4 packets received by filter 0 packets dropped by kernel [root@centos7-neutron-template ~]# ip netns exec qrouter-f48d5536-eefa-4410-b17b-1b3d14426323 tcpdump -l -evvvnn -i qr-295f9857-21 tcpdump: listening on qr-295f9857-21, link-type EN10MB (Ethernet), capture size 262144 bytes ***CRICKETS*** [root@centos7-neutron-template ~]# ip netns exec qrouter-f48d5536-eefa-4410-b17b-1b3d14426323 ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: rfp-f48d5536-e: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether aa:24:89:9e:c8:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 169.254.106.114/31 scope global rfp-f48d5536-e valid_lft forever preferred_lft forever inet6 fe80::a824:89ff:fe9e:c8f0/64 scope link valid_lft forever preferred_lft forever 59: qr-295f9857-21: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:3d:f1:12 brd ff:ff:ff:ff:ff:ff inet 192.168.94.1/24 brd 192.168.94.255 scope global qr-295f9857-21 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe3d:f112/64 scope link valid_lft forever preferred_lft forever [root@centos7-neutron-template ~]# ip netns exec qrouter-f48d5536-eefa-4410-b17b-1b3d14426323 ip route 169.254.106.114/31 dev rfp-f48d5536-e proto kernel scope link src 169.254.106.114 192.168.94.0/24 dev qr-295f9857-21 proto kernel scope link src 192.168.94.1 [root@centos7-neutron-template ~]# ip netns exec qrouter-f48d5536-eefa-4410-b17b-1b3d14426323 ip rule 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 57481: from 192.168.94.9 lookup 16 3232259585: from 192.168.94.1/24 lookup 3232259585 [root@centos7-neutron-template ~]# ip netns exec qrouter-f48d5536-eefa-4410-b17b-1b3d14426323 ip route show table 16 default via 169.254.106.115 dev rfp-f48d5536-e [root@centos7-neutron-template ~]# ip netns exec qrouter-f48d5536-eefa-4410-b17b-1b3d14426323 ip neighbor 169.254.106.115 dev rfp-f48d5536-e lladdr 7a:3b:f1:c7:5d:4e STALE 192.168.94.4 dev qr-295f9857-21 lladdr fa:16:3e:cf:a1:08 PERMANENT 192.168.94.13 dev qr-295f9857-21 lladdr fa:16:3e:91:37:54 PERMANENT 192.168.94.2 dev qr-295f9857-21 lladdr fa:16:3e:b2:18:5e PERMANENT 192.168.94.9 dev qr-29
[Yahoo-eng-team] [Bug 1681979] [NEW] L2pop flows are lost after OVS agent restart
Public bug reported:
In OVS agent, there is a race condition between l2pop's add_fdb_entries
notification and provision_local_vlan when we create a vlanmanager
mapping. This results in either unicast, flooding, or both entries not
being populated on the host. Without the flooding entries, connectivity
is lost.
They are lost semi-permanently after this as l2Pop mechanism driver only
sends full list of fdb entries after a port_update_up, but only on 1st
agent port, or after OVS reboot (where we again hit same race condition,
or it partially fixed flows).
Legacy testbed w/ 3 nodes. 4 tenant networks:
1. The add_fdb_entries code path will create the tunnel port(s) in
add_fdb_tun, then invoke add_fdb_flow to add the BC/UC l2pop flows and -
but only if it can get a Vlanmanager mapping:
def fdb_add(self, context, fdb_entries):
LOG.info("2. fdb_add received")
for lvm, agent_ports in self.get_agent_ports(fdb_entries):
agent_ports.pop(self.local_ip, None)
LOG.info("2. fdb_add: agent_ports = %s", agent_ports)
LOG.info("2. fdb_add: lvm = %s", lvm)
if len(agent_ports):
if not self.enable_distributed_routing:
with self.tun_br.deferred() as deferred_br:
LOG.info("2. fdb_add: about to call fdb_add_tun w/ lvm
= %s", lvm)
self.fdb_add_tun(context, deferred_br, lvm,
agent_ports, self._tunnel_port_lookup)
else:
self.fdb_add_tun(context, self.tun_br, lvm,
agent_ports, self._tunnel_port_lookup)
def get_agent_ports(self, fdb_entries, local_vlan_map=None):
"""Generator to yield port info.
For each known (i.e found in VLAN manager) network in
fdb_entries, yield (lvm, fdb_entries[network_id]['ports']) pair.
:param fdb_entries: l2pop fdb entries
:param local_vlan_map: Deprecated.
"""
lvm_getter = self._get_lvm_getter(local_vlan_map)
for network_id, values in fdb_entries.items():
try:
lvm = lvm_getter(network_id, local_vlan_map)
except vlanmanager.MappingNotFound:
LOG.info("get_agent_ports: vlanmanager.MappingNotFound
EXCEPTION! netid = %s, local_vlan_map = %s", network_id, local_vlan_map)
continue
agent_ports = values.get('ports')
LOG.info("get_agent_ports: got lvm= %s", lvm)
yield (lvm, agent_ports)
2. If the vlan mapping isn't found, the tunnel port creation is skipped, as are
flows.
3. When we create VLAN mapping in provision_local_vlan(), the
install_flood_to_tun however is skipped if there are currently no tunnel
ports created:
def provision_local_vlan(self, net_uuid, network_type, physical_network,
segmentation_id):
...
if network_type in constants.TUNNEL_NETWORK_TYPES:
LOG.info("ARJUN: network_type = %s", network_type)
if self.enable_tunneling:
# outbound broadcast/multicast
ofports = list(self.tun_br_ofports[network_type].values())
LOG.info("ARJUN: provision_local_vlan: ofports = %s
enable_tunneling = %s", ofports, self.enable_tunneling)
if ofports:
LOG.info("ARJUN: installing FLOODING_ENTRY: lvid = %s
segment_id = %s", lvid, segmentation_id)
self.tun_br.install_flood_to_tun(lvid,
segmentation_id,
ofports)
# inbound from tunnels: set lvid in the right table
# and resubmit to Table LEARN_FROM_TUN for mac learning
4. Finally, the cleanup stale flows logic removes all old flows. At this
point br-tun is left with missing flooding and/or unicast flows.
5. If #3 always happens first for all networks, we are good. Otherwise flows
are lost:
Unicast only flows missing if (but flood added):
- Network Vlanmanager mapping is allocated *after* it's
add_fdb_entries, but some other network sets up tunnel ports on br-tun
Broadcast AND UC flows missing if:
- A network tries to add fdb flows before vlanmanager allocated, and no
other network has created the tunnel ports/ofports on br-tun yet.
Example with 3 tenant networks:
1. add_fdb_entries for network 1 and 2 - no LVM yet, so flow and tunnel ports
not created yet
2. LVM created for network 2, but flood not installed because no ofports
3. LVM created for networks 3
4. add_fdb_entries for network 3, here it properly finds the LVM, and creates
tunnel ports/flows
5. LVM created for network 1, tunnel ofports created, so flood installed - but
unicast missing
After this point, network 3 would be fine, network 2 would me missing
all flows, network 1 would have flood but not unicast.
The ordering seems to vary wildly depending
