Hi folks, Reviving this thread as we're seeing this more and more problematic. Combining the ideas mentioned up thread, Dumitru, Numan, Lucas and I had some internal discussion where we came up with a possible approach and we'd love to get feedback from you:
- Local ovn-controller will always insert an OF rule per MAC_Binding entry to match on src_ip + src_mac that will be sampled with a meter to ovn-controller. - When ovn-controller sees that one entry has not been hit "for a while", it'll delete the OpenFlow rule in table 65 that fills the eth.dst field with the MAC_Binding info. - This will result in further ARP requests from the instance(s) that will refresh the MAC_Binding entries in the database. This could make troubleshooting a bit harder so at some point it'll be great to have a mechanism in OVS where we could disable a flow instead of deleting it. This way, one can tell that the flows in table 65 have been disabled due to the aging mechanism in the local node. Thoughts? Is there any performance consideration regarding the extra flows and meters? Thanks a lot! Daniel On Tue, Jul 9, 2019 at 7:19 AM Ben Pfaff <b...@ovn.org> wrote: > > On Mon, Jul 08, 2019 at 06:19:23PM -0700, Han Zhou wrote: > > On Thu, Jun 27, 2019 at 6:44 AM Ben Pfaff <b...@ovn.org> wrote: > > > > > > On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez wrote: > > > > Lately we've been trying to solve certain issues related to stale > > > > entries in the MAC_Binding table (e.g. [0]). On the other hand, for > > > > the OpenStack + Octavia (Load Balancing service) use case, we see that > > > > a reused VIP can be as well affected by stale entries in this table > > > > due to the fact that it's never bound to a VIF so ovn-controller won't > > > > claim it and send the GARPs to update the neighbors. > > > > > > > > I'm not sure if other scenarios may suffer from this issue but seems > > > > reasonable to have an aging mechanism (as we discussed at some point > > > > in the past) that makes unused/old entries to expire. After talking to > > > > Numan on IRC, since a new pinctrl thread has been introduced recently > > > > [1], it'd be nice to implement this aging mechanism there. > > > > At the same time we'd be also reducing the amount of entries for long > > > > lived systems as it'd grow indefinitely. > > > > > > > > Any thoughts? > > > > > > > > Thanks! > > > > Daniel > > > > > > > > PS. With regards to the 'unused' vs 'old' entries I think it has to be > > > > 'old' rather than 'unused' as I don't see a way to reset the TTL of a > > > > MAC_Binding entry when we see packets coming. The implication is that > > > > we'll be seeing ARPs sent out more often when perhaps they're not > > > > needed. This also leads to the discussion of making the cache timeout > > > > configurable. > > > > > > I've always considered the MAC_Binding implementation incomplete because > > > of this issue and others. ovn/TODO.rst says: > > > > > > * Dynamic IP to MAC binding enhancements. > > > > > > OVN has basic support for establishing IP to MAC bindings > > dynamically, using > > > ARP. > > > > > > * Ratelimiting. > > > > > > From casual observation, Linux appears to generate at most one > > ARP per > > > second per destination. > > > > > > This might be supported by adding a new OVN logical action for > > > rate-limiting. > > > > > > * Tracking queries > > > > > > It's probably best to only record in the database responses to > > queries > > > actually issued by an L3 logical router, so somehow they have to > > be > > > tracked, probably by putting a tentative binding without a MAC > > address > > > into the database. > > > > > > * Renewal and expiration. > > > > > > Something needs to make sure that bindings remain valid and > > expire those > > > that become stale. > > > > > > One way to do this might be to add some support for time to the > > database > > > server itself. > > > > > > * Table size limiting. > > > > > > The table of MAC bindings must not be allowed to grow > > unreasonably large. > > > > > > * MTU handling (fragmentation on output) > > > > > > So, what do we do about it? First, I think that adding support for time > > > to the database server is a terrible idea (even though I think I wrote > > > the above originally). Let's not do that. The following is some > > > "thinking out loud" on the subject. > > > > > > I think there's a challenge around which ovn-controller should take care > > > of a given MAC_Binding. We don't want every ovn-controller expiring > > > every binding. Ideally, we want exactly one ovn-controller expiring a > > > binding. One way would be to add an owner column (but it would be > > > better if we don't need it). > > > > > > If we want to keep track of "unused" bindings, I can imagine a > > > statistical mechanism to do that. Any user of a binding occasionally > > > and probabilistically changes a serial number column that we'd introduce > > > into the MAC_Binding table (this could be optimized to not bother if it > > > has changed recently). The owner checks the serial number every so > > > often and if it hasn't changed then it deletes the row. > > > > > > > Thanks Ben for the advice. Since the user of a binding is simply a OpenFlow > > rule matching, I guess we will need "controller" action to trigger the > > serial number column update in ovn-controller, combined with a meter action > > so that only small number of packets trigger the update. Is this what you > > are suggesting? > > I had not thought that far ahead! That approach would work, although > the trigger percentage would be difficult to figure out--it seems like > really we'd want "every Nth second", not "every Nth packet". Another > approach that might work would be for ovn-controller to notice the > statistics on appropriate OpenFlow flows changing, or to use "learn" > actions as a way to make a controller action trigger only every so > often. _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss