Public bug reported: The ARP entries are dropped by DVR routers when the 'qr' device does not exist in the namespace.
There are two ways in the L3 agent the ARP entries are updated. Once when an internal csnat port is created, then arp entries added from the 'dvr_local_router' by calling the "set_subnet_arp_info" which in turn calls the "_update_arp_entry". There is another time, when an arp update "rpc" message comes from the Server to the agent as "add_arp_entry" or "delete_arp_entry" which inturn calls "_update_arp_entry". We have seen log traces that shows that the arp update message comes before the "qr" device is ready. So we get to drop those arp message. We need to kind of cache those arp messages and update the router- namespace when the "qr" device is ready. If you see the message below, we are checking for the device and throwing a warning message that the device is not ready, but the arp entries are not saved anywere. They are dropped. 2015-09-24 18:45:30.150 WARNING neutron.agent.l3.dvr_local_router [req- 0565ce3a-905d-43fa-a6f3-1a07df6c6c2b None None] Arp operation add failed for device qr-b672ffde-cd, since the device does not exist anymore. The device might have been concurrently deleted or not created yet. If you see here the internal_network 'qr' device is added later. 2015-09-24 18:45:30.367 DEBUG neutron.agent.l3.router_info [req- 7e5722e4-5fef-4889-9372-8cf1218522a2 None None] adding internal network: prefix(qr-), port(b672ffde-cd80-49eb-9817-58436fa8e8fd) _internal_network_added /opt/stack/new/neutron/neutron/agent/l3/router_info.py:300 ** Affects: neutron Importance: Undecided Assignee: Swaminathan Vasudevan (swaminathan-vasudevan) Status: In Progress ** Tags: l3-dvr-backlog ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Assignee: (unassigned) => Swaminathan Vasudevan (swaminathan-vasudevan) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1501086 Title: ARP entries dropped by DVR routers when the qr device is not ready or present Status in neutron: In Progress Bug description: The ARP entries are dropped by DVR routers when the 'qr' device does not exist in the namespace. There are two ways in the L3 agent the ARP entries are updated. Once when an internal csnat port is created, then arp entries added from the 'dvr_local_router' by calling the "set_subnet_arp_info" which in turn calls the "_update_arp_entry". There is another time, when an arp update "rpc" message comes from the Server to the agent as "add_arp_entry" or "delete_arp_entry" which inturn calls "_update_arp_entry". We have seen log traces that shows that the arp update message comes before the "qr" device is ready. So we get to drop those arp message. We need to kind of cache those arp messages and update the router- namespace when the "qr" device is ready. If you see the message below, we are checking for the device and throwing a warning message that the device is not ready, but the arp entries are not saved anywere. They are dropped. 2015-09-24 18:45:30.150 WARNING neutron.agent.l3.dvr_local_router [req-0565ce3a-905d-43fa-a6f3-1a07df6c6c2b None None] Arp operation add failed for device qr-b672ffde-cd, since the device does not exist anymore. The device might have been concurrently deleted or not created yet. If you see here the internal_network 'qr' device is added later. 2015-09-24 18:45:30.367 DEBUG neutron.agent.l3.router_info [req- 7e5722e4-5fef-4889-9372-8cf1218522a2 None None] adding internal network: prefix(qr-), port(b672ffde-cd80-49eb-9817-58436fa8e8fd) _internal_network_added /opt/stack/new/neutron/neutron/agent/l3/router_info.py:300 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1501086/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp