On Mon, Nov 30, 2020 at 12:13 PM Renat Nurgaliyev <imple...@gmail.com> wrote: > > On 30.11.20 07:07, Numan Siddique wrote: > > On Mon, Nov 30, 2020 at 7:37 AM Han Zhou <hz...@ovn.org> wrote: > >> On Sat, Nov 28, 2020 at 12:31 PM Tony Liu <tonyliu0...@hotmail.com> wrote: > >>> Hi Renat, > > Hi folks, > >>> > >>> What's this "logical datapath patches that Ilya Maximets submitted"? > >>> Could you share some links? > >>> > >>> There were couple discussions for the similar issue. > >>> [1] raised the issue and results a new option > >>> always_learn_from_arp_request to be added [2]. > >>> [3] results a patch to OVN ML2 driver [4] to set the option added by [1]. > >>> > >>> It seems that it helps to optimize logical_flow table. > >>> I am not sure if it helps on mac_binding as well. > >>> > >>> Is it the same issue we are trying to address here, by either > >>> Numan's local cache or the solution proposed by Dumitru? > >>> > >>> [1] > >> https://mail.openvswitch.org/pipermail/ovs-discuss/2020-May/049994.html > >>> [2] > >> https://github.com/ovn-org/ovn/commit/61ccc6b5fc7c49b512e26347cfa12b86f0ec2fd9#diff-05b24a3133733fb7b0f979698083b8128e8f1f18c3c2bd09002ae788d34a32f5 > >>> [3] http://osdir.com/openstack-discuss/msg16002.html > >>> [4] https://review.opendev.org/c/openstack/neutron/+/752678 > >>> > >>> > >>> Thanks! > >>> Tony > >> Thanks Tony for pointing to the old discussion [0]. I thought setting the > >> option always_learn_from_arp_request to "false" on the logical routers > >> should have solved this scale problem in MAC_Binding table in this scenario. > >> > >> However, it seems the commit a2b88dc513 ("pinctrl: Directly update > >> MAC_Bindings created by self originated GARPs.") have overridden the > >> option. (I haven't tested, but maybe @Dumitru Ceara <dce...@redhat.com> can > >> confirm.) > >> > >> Similarly, for the Logical_Flow explosion, it should have been solved by > >> setting the option dynamic_neigh_routers to "true". > >> > >> I think these two options are exactly for the scenario Renat is > >> reporting. @Renat, could you try setting these options as suggested above > >> using the OVN version before the commit a2b88dc513 to see if it solves your > >> problem? > >> > > When you test it out with the suggested commit, please delete the > > mac_binding entries manually > > as ovn-northd or ovn-controllers don't delete any entries from > > mac_binding table. > > We tested with dynamic_neigh_routers set to true, and we saw some very > positive change, size of Logical_Flows table decresed from 600k > entries to 100k. This is a huge difference, thanks for pointing this > out! > > It did not affect MAC_Binding table with commit a2b88dc513 ("pinctrl: > Directly update MAC_Bindings created by self originated GARPs."), but > that was expected. Just for test purposes we commented out some code > as follows: > > diff --git a/controller/pinctrl.c b/controller/pinctrl.c > index 291202c24..76047939c 100644 > --- a/controller/pinctrl.c > +++ b/controller/pinctrl.c > @@ -4115,10 +4115,10 @@ send_garp_rarp_update(struct ovsdb_idl_txn *ovnsb_idl_txn, > laddrs->ipv4_addrs[i].addr, > binding_rec->datapath->tunnel_key, > binding_rec->tunnel_key); > - send_garp_locally(ovnsb_idl_txn, > - sbrec_mac_binding_by_lport_ip, > - local_datapaths, binding_rec, laddrs->ea, > - laddrs->ipv4_addrs[i].addr); > + //send_garp_locally(ovnsb_idl_txn, > + // sbrec_mac_binding_by_lport_ip, > + // local_datapaths, binding_rec, laddrs->ea, > + // laddrs->ipv4_addrs[i].addr); > > } > free(name); > > Together with dynamic_neigh_routers we achieved quite a stable setup, > with 62 MiB SB database, which is a huge step forward after 1.9 GiB. > MAC_Binding size stays around 2000 entries, in comparison to almost a > million. > > Will it make sense to make behaviour introduced in a2b88dc513 > toggleable via a command line option, before there is a better > solution? > > Thanks, > Renat. >
Thanks Renat for the testing. The result looks good. Just to confirm, in the final test with the code change above, did you also set the "always_learn_from_arp_request" to "false"? I think the logic introduced in a2b88dc513 can add the check for the option "always_learn_from_arp_request" instead of overriding it. Also, regarding to Winson's question: > We moved to ovn 20.09 branch recently and the mac binding issues happen again in our > ovn-k8s scale test cluster. > Is there a quick workaround to make the options " always_learn_from_arp_request “ works again? > Thanks Winson for confirming. As mentioned above, I think the logic of the patch "pinctrl: Directly update MAC_Bindings created by self originated GARPs." can be updated to add the check for this option, to restore the behavior. Before the fix, I think a quick work around for you in 20.09 could be reverting the following patches (I haven't tested though): 1. "ovn-northd: Limit self originated ARP/ND broadcast domain." 2. "pinctrl: Fix segfault seen when creating mac_binding for local GARPs." 3. "pinctrl: Directly update MAC_Bindings created by self originated GARPs." Thanks, Han > >> Regarding the proposals in this thread: > >> - Move MAC_Binding to LS (by Dumitru) > >> This sounds good to me, while I am not sure about all the implications > >> yet, wondering why it was associated with LRP instead in the beginning. > >> > >> - Remove MAC_Binding from SB (by Numan) > >> I am a little concerned about this. The MAC_Binding in SB is required > >> for distributed LR to work for dynamic ARP resolving. Consider a general > >> use case: A - LS1 - LR1 - LS2 - B. A is on HV1 and B is on HV2. Now A sends > >> a packet to B's IP. Assume B's IP is unknown by OVN. The packet is routed > >> by LR1 and on the LRP facing LS2 an ARP is sent out over the LS1 logical > >> network. The above steps happen on HV1. Now the ARP request reaches HV2 and > >> is received by B, so B sends an ARP response. With the current > >> implementation, HV2's OVS flow would learn the MAC-IP binding from the ARP > >> response and update SB DB, and HV1 will get the SB update and install the > >> MAC Binding flow as a result of ARP resolving. The next time A sends a > >> packet to B, the HV1 will directly resolve the ARP from the MAC Binding > >> flows locally and send the IP packet to HV2. The SB DB MAC_Binding table > >> works as a distributed ARP/Neighbor cache. It is a mechanism to sync the > >> ARP cache from the place where it is learned to the place where it is > >> initiated, and all HVs benefit from this without the need to send ARP > >> themselves for the same LRP. In other words, the LRP is distributed, so the > >> ARP resolving is in a distributed fashion. Without this, each HV would > >> initiate ARP request on behalf of the same LRP, which would largely > >> increase the ARP traffic unnecessarily - even more than the traditional > >> network (where one physical router only needs to do one ARP resolving for > >> each neighbor and maintain one copy of ARP cache). And I am not sure if > >> there are other side effects when an endpoint sees unexpectedly frequent > >> ARP requests from the same LRP - would there be any rate limit that even > >> discards repeated ARP requests from the same source? Numan, maybe you have > >> already considered these. Would you share your thoughts? > > Thanks for the comments and highlighting this use case which I missed > > completely. > > > > I was thinking more in lines on the N-S usecase with a distributed > > gateway router port. > > And I completely missed the E-W with an unknown address scenario. If > > we don't consider > > the unknown address scenario, I think moving away from MAC_Binding > > south db tabe would > > be beneficial in the long run. For few reasons > > 1. For better scale. > > 2. To address the mac_binding stale entries (which presently CMS > > have to handle) > > > > For N-S traffic scenario, ovn-controller claiming the gw router port > > will take care of generating the ARP. > > For Floating IP dvr scenario, each compute node will have to generate > > the ARP request to learn a remote. > > I think this should be fine as it is just a one time thing. > > > > Regarding the unknown address scenario, right now ovn controller > > floods the packet to all the unknown logical ports > > of a switch if OVN doesn't know the MAC. All these are unknown logical > > ports belonging to a multicast group. > > > > I think we should solve this case. In the case of Openstack, when port > > security is disabled for a neutron port, the logical > > port will have an unknown address configured. There are a few related > > bugzillas/lauchpad bugs [1]. > > > > I think we should fix this behavior in OVN and ovn should do the mac > > learning on the switch for the unknown ports. And If we do that, > > I think the scenario you mentioned will be addressed. > > > > Maybe we can extend Dumitru's suggestion and have just one approach > > which does the mac learning on the switch (keeping > > the SB Mac_binding table). > > - for unknown logical ports > > - for unknown macs for the N-S routing. > > > > Any thoughts ? > > > > FYI - I have a PoC/RFC patch in progress which adds the mac binding > > cache support - > > https://github.com/numansiddique/ovn/commit/22082d04ca789155ea2edd3c1706bde509ae44da > > > > [1] - https://review.opendev.org/c/openstack/neutron/+/763567/ > > https://bugzilla.redhat.com/show_bug.cgi?id=1888441 > > https://bugs.launchpad.net/neutron/+bug/1904412 > > https://bugzilla.redhat.com/show_bug.cgi?id=1672625 > > > > Thanks > > Numan > > > >> Thanks, > >> Han > >> > >>>> -----Original Message----- > >>>> From: dev <ovs-dev-boun...@openvswitch.org> On Behalf Of Numan Siddique > >>>> Sent: Thursday, November 26, 2020 11:36 AM > >>>> To: Daniel Alvarez Sanchez <dalva...@redhat.com> > >>>> Cc: ovs-dev <ovs-dev@openvswitch.org> > >>>> Subject: Re: [ovs-dev] Scaling of Logical_Flows and MAC_Binding tables > >>>> > >>>> On Thu, Nov 26, 2020 at 4:32 PM Numan Siddique <num...@ovn.org> wrote: > >>>>> On Thu, Nov 26, 2020 at 4:11 PM Daniel Alvarez Sanchez > >>>>> <dalva...@redhat.com> wrote: > >>>>>> On Wed, Nov 25, 2020 at 7:59 PM Dumitru Ceara <dce...@redhat.com> > >>>> wrote: > >>>>>>> On 11/25/20 7:06 PM, Numan Siddique wrote: > >>>>>>>> On Wed, Nov 25, 2020 at 10:24 PM Renat Nurgaliyev > >>>>>>>> <imple...@gmail.com> > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 25.11.20 16:14, Dumitru Ceara wrote: > >>>>>>>>>> On 11/25/20 3:30 PM, Renat Nurgaliyev wrote: > >>>>>>>>>>> Hello folks, > >>>>>>>>>>> > >>>>>>>>>> Hi Renat, > >>>>>>>>>> > >>>>>>>>>>> we run a lab where we try to evaluate scalability potential > >>>>>>>>>>> of OVN > >>>>>>> with > >>>>>>>>>>> OpenStack as CMS. > >>>>>>>>>>> Current lab setup is following: > >>>>>>>>>>> > >>>>>>>>>>> 500 networks > >>>>>>>>>>> 500 routers > >>>>>>>>>>> 1500 VM ports (3 per network/router) > >>>>>>>>>>> 1500 Floating IPs (one per VM port) > >>>>>>>>>>> > >>>>>>>>>>> There is an external network, which is bridged to br-provider > >>>>>>>>>>> on > >>>>>>> gateway > >>>>>>>>>>> nodes. There are 2000 ports > >>>>>>>>>>> connected to this external network (1500 Floating IPs + 500 > >>>>>>>>>>> SNAT > >>>>>>> router > >>>>>>>>>>> ports). So the setup is not > >>>>>>>>>>> very big we'd say, but after applying this configuration via > >>>>>>>>>>> ML2/OVN plugin, northd kicks in and does its job, and after > >>>>>>>>>>> its done, Logical_Flow table gets 645877 entries, which is > >>>>>>>>>>> way too much. But ok, we move on and start one controller on > >>>>>>>>>>> the gateway chassis, and here things get really messy. > >>>>>>>>>>> MAC_Binding table grows from 0 to 999088 entries in one > >>>>>>>>>>> moment, and after its done, the size of SB biggest tables > >>>>>>>>>>> look like this: > >>>>>>>>>>> > >>>>>>>>>>> 999088 MAC_Binding > >>>>>>>>>>> 645877 Logical_Flow > >>>>>>>>>>> 4726 Port_Binding > >>>>>>>>>>> 1117 Multicast_Group > >>>>>>>>>>> 1068 Datapath_Binding > >>>>>>>>>>> 1046 Port_Group > >>>>>>>>>>> 551 IP_Multicast > >>>>>>>>>>> 519 DNS > >>>>>>>>>>> 517 HA_Chassis_Group > >>>>>>>>>>> 517 HA_Chassis > >>>>>>>>>>> ... > >>>>>>>>>>> > >>>>>>>>>>> MAC binding table gets huge, basically it now has an entry > >>>>>>>>>>> for every port that is connected to external network * number > >>>>>>>>>>> of datapaths, which roughly makes it one million entries. > >>>>>>>>>>> This table by itself increases the size of the SB by 200 > >>>>>>>>>>> megabytes. Logical_Flow table also gets very heavy, we have > >>>>>>>>>>> already played a bit with logical datapath patches that Ilya > >>>>>>>>>>> Maximets submitted, and it > >>>>>>> looks > >>>>>>>>>>> much better, but the size of > >>>>>>>>>>> the MAC_Binding table still feels inadequate. > >>>>>>>>>>> > >>>>>>>>>>> We would like to start to work at least on MAC_Binding table > >>>>>>>>>>> optimisation, but it is a bit difficult to start working from > >>>>>>>>>>> scratch. Can someone help us with ideas how this could be > >>>>>>>>>>> optimised? > >>>>>>>>>>> > >>>>>>>>>>> Maybe it would also make sense to group entries in > >>>>>>>>>>> MAC_Binding table > >>>>>>> in > >>>>>>>>>>> the same way like it is proposed for logical flows in Ilya's > >>>>>>>>>>> patch? > >>>>>>>>>>> > >>>>>>>>>> Maybe it would work but I'm not really sure how, right now. > >>>>>>>>>> However, what if we change the way MAC_Bindings are created? > >>>>>>>>>> > >>>>>>>>>> Right now a MAC Binding is created for each logical router > >>>>>>>>>> port but in your case there are a lot of logical router ports > >>>>>>>>>> connected to the single provider logical switch and they all > >>>> learn the same ARPs. > >>>>>>>>>> What if we instead store MAC_Bindings per logical switch? > >>>>>>>>>> Basically sharing all these MAC_Bindings between all router > >>>>>>>>>> ports connected to > >>>>>>> the > >>>>>>>>>> same LS. > >>>>>>>>>> > >>>>>>>>>> Do you see any problem with this approach? > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Dumitru > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> I believe that this approach is way to go, at least nothing > >>>>>>>>> comes to my > >>>>>>> mind > >>>>>>>>> that could go wrong here. We will try to make a patch for that. > >>>>>>> However, if > >>>>>>>>> someone is familiar with the code and knows how to do it fast, > >>>>>>>>> it would > >>>>>>> also > >>>>>>>>> be very nice. > >>>>>>>> This approach should work. > >>>>>>>> > >>>>>>>> I've another idea (I won't call it a solution yet). What if we > >>>>>>>> drop the usage of MAC_Binding altogether ? > >>>>>>> This would be great! > >>>>>>> > >>>>>>>> - When ovn-controller learns a mac_binding, it will not create a > >>>>>>>> row into the SB MAC_binding table > >>>>>>>> - Instead it will maintain the learnt mac binding in its memory. > >>>>>>>> - ovn-controller will still program the table 66 with the flow > >>>>>>>> to set the eth.dst (for the get_arp() action) > >>>>>>>> > >>>>>>>> This has a couple of advantages > >>>>>>>> - Right now we never flush the old/stale mac_binding entries. > >>>>>>>> - If suppose the mac of an external IP has changed, but OVN > >>>>>>>> has an entry for that IP with old mac in the mac_binding table, > >>>>>>>> we will use the old mac, causing the packet to be sent out > >>>>>>>> to the wrong destination and the packet might get lost. > >>>>>>>> - So we will get rid of this problem > >>>>>>>> - We will also save SB DB space. > >>>>>>>> > >>>>>>>> There are few disadvantages > >>>>>>>> - Other ovn-controllers will not add the flows in table 66. I > >>>>>>>> guess this should be fine as each ovn-controller can generate > >>>>>>>> the ARP request and learn the mac. > >>>>>>>> - When ovn-controller restarts we lose the learnt macs and > >>>>>>>> would need to learn again. > >>>>>>>> > >>>>>>>> Any thoughts on this ? > >>>>>> It'd be great to have some sort of local ARP cache but I'm concerned > >>>>>> about the performance implications. > >>>>>> > >>>>>> - How are you going to determine when an entry is stale? > >>>>>> If you slow path the packets to reset the timeout everytime a pkt > >>>>>> with source mac is received, it doesn't look good. Maybe you have > >>>>>> something else in mind. > >>>>> Right now we don't stale any mac_binding entry. If I understand you > >>>>> correctly, your concern is for the scenario where a floating ip is > >>>>> updated with a different mac, how the local cache is updated ? > >>>>> > >>>>> Right now networking-ovn (in the case of openstack) updates the > >>>>> mac_binding entry in the South db for such cases right ? > >>>>> > >>>> FYI - I have started working on this approach as PoC. i.e to use local > >>>> mac_binding cache > >>>> instead of using the SB mac_binding table. > >>>> > >>>> I will update this thread about the progress. > >>>> > >>>> Thanks > >>>> Numan > >>>> > >>>>> Thanks > >>>>> Numan > >>>>> > >>>>>> - > >>>>>> > >>>>>>> There's another scenario that we need to take care of and doesn't > >>>> seem > >>>>>>> too obvious to address without MAC_Bindings. > >>>>>>> > >>>>>>> GARPs were being injected in the L2 broadcast domain of a LS for > >>>> nat > >>>>>>> addresses in case FIPs are reused by the CMS, introduced by: > >>>>>>> > >>>>>>> > >>>>>>> https://github.com/ovn- > >>>> org/ovn/commit/069a32cbf443c937feff44078e8828d7a2702da8 > >>>>>> > >>>>>> Dumitru and I have been discussing the possibility of reverting this > >>>> patch > >>>>>> and rely on CMSs to maintain the MAC_Binding entries associated with > >>>> the > >>>>>> FIPs [0]. > >>>>>> I'm against reverting this patch in OVN [1] for multiple reasons > >>>> being the > >>>>>> most important one the fact that if we rely on workarounds in the > >>>> CMS side, > >>>>>> we'll be creating a control plane dependency for something that is > >>>> pure > >>>>>> dataplane only (ie. if Neutron server is down - outage, upgrades, > >>>> etc. -, > >>>>>> traffic is going to be disrupted). On the other hand one could argue > >>>> that > >>>>>> the same dependency now exists on ovn-controller being up & running > >>>> but I > >>>>>> believe that this is better than a) relying on workarounds on CMSs > >> b) > >>>>>> relying on CMSs availability. > >>>>>> > >>>>>> In the short term I think that moving the MAC_Binding entries to LS > >>>> instead > >>>>>> of LRP as it was suggested up thread would be a good idea and in the > >>>> long > >>>>>> haul, the ARP *local* cache seems to be the right solution. > >>>> Brainstorming > >>>>>> with Dumitru he suggested inspecting the flows regularly to see if > >>>> the > >>>>>> packet count on flows that check if src_mac == X has not increased > >>>> in a > >>>>>> while and then remove the ARP responder flows locally. > >>>>>> > >>>>>> [0] > >>>>>> https://github.com/openstack/networking- > >>>> ovn/commit/5181f1106ff839d08152623c25c9a5f6797aa2d7 > >>>>>> [1] > >>>>>> https://github.com/ovn- > >>>> org/ovn/commit/069a32cbf443c937feff44078e8828d7a2702da8 > >>>>>>> > >>>>>>> Recently, due to the dataplane scaling issue (4K resubmit limit > >>>> being > >>>>>>> hit), we don't flood these packets on non-router ports and instead > >>>>>>> create the MAC Bindings directly from ovn-controller: > >>>>>>> > >>>>>>> > >>>>>>> https://github.com/ovn- > >>>> org/ovn/commit/a2b88dc5136507e727e4bcdc4bf6fde559f519a9 > >>>>>>> Without the MAC_Binding table we'd need to find a way to update or > >>>> flush > >>>>>>> stale bindings when an IP is used for a VIF or FIP. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Dumitru > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> dev mailing list > >>>>>>> d...@openvswitch.org > >>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >>>>>>> > >>>>>>> > >>>>>> _______________________________________________ > >>>>>> dev mailing list > >>>>>> d...@openvswitch.org > >>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >>>>>> > >>>> _______________________________________________ > >>>> dev mailing list > >>>> d...@openvswitch.org > >>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >>> _______________________________________________ > >>> dev mailing list > >>> d...@openvswitch.org > >>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >> _______________________________________________ > >> dev mailing list > >> d...@openvswitch.org > >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >> > > _______________________________________________ > > dev mailing list > > d...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
