Hi Han, Sounds good. I am looking forward to incremental-processing, and will go from there.
BTW, it would be great if you could let me know how to set probe interval for 3-node cluster, here or in another thread. Thanks! Tony > -----Original Message----- > From: Han Zhou <hz...@ovn.org> > Sent: Tuesday, August 4, 2020 4:02 PM > To: Tony Liu <tonyliu0...@hotmail.com> > Cc: Han Zhou <hz...@ovn.org>; Numan Siddique <num...@ovn.org>; Ben Pfaff > <b...@ovn.org>; Leonid Ryzhyk <lryz...@vmware.com>; ovs-dev <ovs- > d...@openvswitch.org>; ovs-discuss <ovs-discuss@openvswitch.org> > Subject: Re: [ovs-discuss] [OVN] ovn-northd takes much CPU when no > configuration update > > Hi Tony, > > I am glad it is more clear now. For your concern regarding taking too > much time for one round of computing, it is valid, but I guess it is not > directly related to the IDLE probe any more, right? > The OVSDB IDL in fact already does some of the work of caping and > buffering like what you proposed. The IDL will read a limited number of > messages to get processed in each round (and the remaining messages are > buffered in the stream). However, sometimes a single notification > message can contain a huge amount of data. It is hard to split the data > from one single notification, because the data are internally dependent > on each other. > > Without incremental-processing, the size of the data change doesn't > matter much because all data is recomputed anyway. I'd suggest to see > what's the outcome of incremental-processing, and see if any further > improvement is still needed for handling big transactions. > > In my opinion, the special cases of a big data change triggered by > scenarios such as data restore can be handled by operational approaches > instead of implementation. For example, you could adjust the probe > interval before doing data restore, and change it back afterwards. But > of course, if there are good ways to implement we should definitely > consider. > > > Thanks, > Han > > > On Tue, Aug 4, 2020 at 2:00 PM Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > wrote: > > > Hi Han, > > Thanks for clarifications! It's crystal clear. > > My concern, in general, is blocking. For onv-northd, or OVSDB > client, > (I assume all OVSDB clients are using the same library for > connection, > proble, etc.?) when handing current event, it won't be interrupted > to > handle any incoming event, right? How long does it take to handle a > computing event for big chunk of data? How much data can be > buffered > to be computed? Is there estimated maximum time for handle so much > data? > > In case it takes more than 5s to process an event, then the peer > will > drop the connection because of probe timeout. > > With incremental-process, if I restore DB, then that still could be > a > huge incremental, unless the incremental size is controlled. That's > probably why you recommend to restore to existing cluster, to avoid > huge incremental from restoring to a fresh cluster. Am I right? > > What I used to do is to chop big data into pieces and to be handled > by > multiple event loops. That way, other events will have a chance to > get > processed. So big chunk of data won't cause blocking. > > Enlarge probe interval will sort of resolve the issue, but it will > lose > the point of probing. Just like that election timer, enlarge the > timer > avoids often failover, but it also increases the failover time when > real > problem happens. And yes, I agree that it's on control plane and > doesn't > break data plane, but just like in networking world, routing > convergence > is very important. > > I am thinking, in your incremental-processing, if the time for each > event > loop can be capped or controlled, that would be very helpful. The > side > effect of that option is memory consumption. You will need to > buffer more > data. But today, it's lots easier to increase memory to boost > performance. > > > Thanks! > > Tony > > > -----Original Message----- > > From: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > > > Sent: Tuesday, August 4, 2020 12:34 PM > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > Cc: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> >; Numan > Siddique <num...@ovn.org <mailto:num...@ovn.org> >; Ben Pfaff > > <b...@ovn.org <mailto:b...@ovn.org> >; Leonid Ryzhyk > <lryz...@vmware.com <mailto:lryz...@vmware.com> >; ovs-dev <ovs- > > d...@openvswitch.org <mailto:d...@openvswitch.org> >; ovs-discuss > <ovs-discuss@openvswitch.org <mailto:ovs-discuss@openvswitch.org> > > > Subject: Re: [ovs-discuss] [OVN] ovn-northd takes much CPU when > no > > configuration update > > > > > > > > On Tue, Aug 4, 2020 at 11:40 AM Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > Inline... > > > > Thanks! > > > > Tony > > > -----Original Message----- > > > From: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > Sent: Tuesday, August 4, 2020 11:01 AM > > > To: Numan Siddique <num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > >; Ben > > Pfaff <b...@ovn.org <mailto:b...@ovn.org> <mailto:b...@ovn.org > <mailto:b...@ovn.org> > >; Leonid > > > Ryzhyk <lryz...@vmware.com <mailto:lryz...@vmware.com> > <mailto:lryz...@vmware.com <mailto:lryz...@vmware.com> > > > > > Cc: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > >; Han Zhou <hz...@ovn.org > <mailto:hz...@ovn.org> > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > >; ovs- > > > dev <ovs-...@openvswitch.org <mailto:ovs- > d...@openvswitch.org> <mailto:ovs-...@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > >; > > ovs-discuss <ovs-discuss@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > > Subject: Re: [ovs-discuss] [OVN] ovn-northd takes much > CPU when > > no > > > configuration update > > > > > > > > > > > > On Tue, Aug 4, 2020 at 12:38 AM Numan Siddique > <num...@ovn.org <mailto:num...@ovn.org> > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > wrote: > > > > > > > > > > > > > > > On Tue, Aug 4, 2020 at 9:02 AM Tony Liu > > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > wrote: > > > > > > > > > The probe awakes recomputing? > > > There is probe every 5 seconds. Without any > > connection > > > up/down or failover, > > > ovn-northd will recompute everything every > 5 > > seconds, no > > > matter what? > > > Really? > > > > > > Anyways, I will increase the probe interval > for now, > > see if > > > that helps. > > > > > > > > > > > > I think we should optimise this case. I am planning > to look > > into > > > this. > > > > > > Thanks > > > Numan > > > > > > > > > Thanks Numan. > > > I'd like to discuss more on this before we move forward > to change > > > anything. > > > > > > 1) Regarding the problem itself, the CPU cost triggered > by OVSDB > > IDLE > > > probe when there is no configuration change to compute, I > don't > > think it > > > matters that much in real production. It simply wastes > CPU cycles > > when > > > there is nothing to do, so what harm would it do here? > For ovn- > > northd, > > > since it is the centralized component, we would always > ensure > > there is > > > enough CPU available for ovn-north when computing is > needed, and > > this > > > reservation will be wasted anyway when there is no change > to > > compute. So, > > > I'd avoid making any change specifically only to address > this > > issue. I > > > could be wrong, though. I'd like to hear what would be > the real > > concern > > > if this is not addressed. > > > > Is more vCPUs going to help here? Is ovn-northd multi- > thread? > > > > > > > > > > ovn-northd is single threaded. It can be changed to have a > separate > > thread for the probe handling, but I don't see any obvious > benefit. > > > > > > > > I am probably still missing something here. The probe is > there all > > times, > > every 5s. > > > > > > The probe is sent only if there is no activity on the OVSDB > connection > > during the interval, that's why it is called "IDLE" probe. If > there is > > already interaction during the past interval, no probe will be > sent. > > > > > > > > If ovn-northd is in the middle of a computing, is a probe > going > > to make ovn-northd restart the computing? > > > > > > No, it won't. Firstly, it is unlikely that a probe is received > during > > computing, unless the probe interval is set too short. Secondly, > even > > when it happens, the current computing will complete and all > needed > > changes will be enforced to SB DB regardless of the probe > received > > during the computing. The probe will be handled in the next round > of the > > main loop, and it will trigger another round of computing which > is > > useless but not harmful either. There is probably one case I can > think > > of that causes a little latency - when another NB DB change (say, > > change2) comes during the computing triggered by the probe, then > the > > handling for the change2 will be delayed a little until the > computing > > triggered by the probe completes. But the chance is rather low, > > especially if the probe interval is enlarged, and in the unlucky > case, > > the impact is just a little delay in the change handling. > > > > > > > > Or the probe only triggers > > computing when ovn-northd is idle? Even with the latter > case, > > what's the > > intention to trigger computing by probe? > > > > > > > > > > It is not triggered intentionally for the probe. It is just > because ovn- > > northd doesn't distinguish if it is woken up by a probe only or > if there > > are any changes that need to be processed. Many events can wake > up ovn- > > northd, and once it is wake up it will compute everything. I > agree it > > can be optimized (we already optimized this for ovn-controller). > I am > > just wondering if it worth to be optimized specifically. Or we > just get > > it for free as a byproduct when implementing incremental- > processing, > > which is already in the road map. > > > > Does this clarify a little? > > > > > > Thanks, > > Han > > > > > > > > > > > > 2) ovn-northd incremental processing would avoid this CPU > problem > > > naturally. So let's discuss how to move forward for > incremental > > > processing, which is much more important because it also > solves > > the CPU > > > efficiency when handling the changes, and the IDLE probe > problem > > is just > > > a byproduct. I believe the DDlog branch would have solved > this > > problem. > > > However, it seems we are not sure about the current > status of > > DDlog. As > > > you proposed at the last OVN meeting, an alternative is > to > > implement > > > partial incremental-processing using the I-P engine like > ovn- > > controller. > > > While I have no objection to this, we'd better check with > Ben and > > Leonid > > > on the plan to avoid overlapping and waste of work. @Ben > @Leonid, > > would > > > you mind sharing the status here since you were not at > the > > meeting last > > > week? > > > > My point is that, a probe is not supposed to trigger a > computing, > > no matter > > it's full or incremental. > > > > > > > > > > > > > > Thanks, > > > Han > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > Tony > > > > > > > -----Original Message----- > > > > From: Han Zhou <hz...@ovn.org > <mailto:hz...@ovn.org> > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> <mailto:hz...@ovn.org > <mailto:hz...@ovn.org> > > > > > > > Sent: Monday, August 3, 2020 8:22 PM > > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > Cc: Han Zhou <hz...@ovn.org > <mailto:hz...@ovn.org> <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > >; ovs- > > > discuss <ovs-discuss@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > >; > > > > ovs-dev <ovs-...@openvswitch.org > <mailto:ovs-...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:ovs- > <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > Subject: Re: [ovs-discuss] [OVN] ovn- > northd takes > > much CPU > > > when no > > > > configuration update > > > > > > > > Sorry that I didn't make it clear enough. > The > > OVSDB probe > > > itself doesn't > > > > take much CPU, but the probe awakes ovn- > northd > > main loop, > > > which > > > > recompute everything, which is why you > see CPU > > spike. > > > > It will be solved by incremental- > processing, when > > only > > > delta is > > > > processed, and in case of probe handling, > there > > is no > > > change in > > > > configuration, so the delta is zero. > > > > For now, please follow the steps to > adjust probe > > interval, > > > if the CPU of > > > > ovn-northd (when there is no > configuration change) > > is a > > > concern for you. > > > > But please remember that this has no > impact to > > the real CPU > > > usage for > > > > handling configuration changes. > > > > > > > > > > > > Thanks, > > > > Han > > > > > > > > > > > > On Mon, Aug 3, 2020 at 8:11 PM Tony Liu > > > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > wrote: > > > > > > > > > > > > Health check (5 sec internal) > taking 30%- > > 100% CPU is > > > definitely not > > > > acceptable, > > > > if that's really the case. There > must be > > some > > > blocking (and not > > > > yielding CPU) > > > > in coding, which is not supposed to > be > > there. > > > > > > > > Could you point me to the coding > for such > > health > > > check? > > > > Is it single thread? Does it use > any event > > library? > > > > > > > > > > > > Thanks! > > > > > > > > Tony > > > > > > > > > -----Original Message----- > > > > > From: Han Zhou <hz...@ovn.org > <mailto:hz...@ovn.org> > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> <mailto:hz...@ovn.org > <mailto:hz...@ovn.org> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > > Sent: Saturday, August 1, 2020 > 9:11 PM > > > > > To: Tony Liu > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > Cc: ovs-discuss <ovs- > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:ovs-discuss@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > <mailto:ovs-discuss@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > <mailto:ovs- > <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> > > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > >; ovs-dev <ovs- > > > > > d...@openvswitch.org > <mailto:d...@openvswitch.org> > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > Subject: Re: [ovs-discuss] [OVN] > ovn- > > northd takes > > > much CPU when > > > > no > > > > > configuration update > > > > > > > > > > > > > > > > > > > > On Fri, Jul 31, 2020 at 4:14 PM > Tony Liu > > > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > I see the active ovn-northd > takes > > much CPU > > > (30% - 100%) > > > > when there > > > > > is no > > > > > configuration from > OpenStack, > > nothing > > > happening on all > > > > chassis > > > > > nodes either. > > > > > > > > > > Is this expected? What is > it busy > > with? > > > > > > > > > > > > > > > > > > > > > > > > > Yes, this is expected. It is due > to the > > OVSDB probe > > > between ovn- > > > > northd > > > > > and NB/SB OVSDB servers, which is > used to > > detect > > > the OVSDB > > > > connection > > > > > failure. > > > > > Usually this is not a concern > (unlike the > > probe > > > with a large > > > > number of > > > > > ovn-controller clients), because > ovn- > > northd is a > > > centralized > > > > component > > > > > and the CPU cost when there is no > > configuration > > > change doesn't > > > > matter > > > > > that much. However, if it is a > concern, > > the probe > > > interval > > > > (default 5 > > > > > sec) can be changed. > > > > > If you change, remember to change > on both > > server > > > side and client > > > > side. > > > > > For client side (ovn-northd), it > is > > configured in > > > the NB DB's > > > > NB_Global > > > > > table's > options:northd_probe_interval. > > See man page > > > of ovn-nb(5). > > > > > For server side (NB and SB), it > is > > configured in > > > the NB and SB > > > > DB's > > > > > Connection table's > inactivity_probe > > column. > > > > > > > > > > Thanks, > > > > > Han > > > > > > > > > > > > > > > > > > > > ======== > > > > > 2020-07- > > > 31T23:08:09.511Z|04267|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 8 (10.6.20.84:44358 > <http://10.6.20.84:44358> > > <http://10.6.20.84:44358> <http://10.6.20.84:44358> > > > <http://10.6.20.84:44358> > > > > <http://10.6.20.84:44358> <- > >10.6.20.84:6641 <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> ) at > lib/stream- > > fd.c:157 > > > (68% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:09.512Z|04268|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:09.512Z|04269|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:12.777Z|04270|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 9 (10.6.20.84:49158 > <http://10.6.20.84:49158> > > <http://10.6.20.84:49158> <http://10.6.20.84:49158> > > > <http://10.6.20.84:49158> > > > > <http://10.6.20.84:49158> <- > >10.6.20.85:6642 <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> ) at > lib/stream- > > fd.c:157 > > > (34% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:12.777Z|04271|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : idle > 5002 ms, > > sending > > > inactivity probe > > > > > 2020-07- > > > > > > 31T23:08:12.777Z|04272|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > entering IDLE > > > > > 2020-07- > > > > > > 31T23:08:12.777Z|04273|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : send > request, > > > method="echo", params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:12.777Z|04274|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:12.777Z|04275|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > entering > > ACTIVE > > > > > 2020-07- > > > > > > 31T23:08:12.777Z|04276|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:13.635Z|04277|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 9 (10.6.20.84:49158 > <http://10.6.20.84:49158> > > <http://10.6.20.84:49158> <http://10.6.20.84:49158> > > > <http://10.6.20.84:49158> > > > > <http://10.6.20.84:49158> <- > >10.6.20.85:6642 <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> ) at > lib/stream- > > fd.c:157 > > > (34% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:13.635Z|04278|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > received reply, > > > result=[], id="echo" > > > > > 2020-07- > > > 31T23:08:14.480Z|04279|hmap|DBG|Dropped 129 log > > > > messages in > > > > > last 5 seconds (most recently, 0 > seconds > > ago) due > > > to excessive > > > > rate > > > > > 2020-07- > > > 31T23:08:14.480Z|04280|hmap|DBG|lib/shash.c:112: 2 > > > > buckets > > > > > with 6+ nodes, including 2 > buckets with 6 > > nodes (32 > > > nodes total > > > > across > > > > > 32 buckets) > > > > > 2020-07- > > > 31T23:08:14.513Z|04281|poll_loop|DBG|wakeup due to > > > > 27-ms > > > > > timeout at lib/reconnect.c:643 > (34% CPU > > usage) > > > > > 2020-07- > > > > > > 31T23:08:14.513Z|04282|reconnect|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : idle > 5001 ms, > > sending > > > inactivity probe > > > > > 2020-07- > > > > > > 31T23:08:14.513Z|04283|reconnect|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > entering IDLE > > > > > 2020-07- > > > > > > 31T23:08:14.513Z|04284|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : send > request, > > > method="echo", params=[], > > > > > id="echo" > > > > > 2020-07- > > > 31T23:08:15.370Z|04285|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 8 (10.6.20.84:44358 > <http://10.6.20.84:44358> > > <http://10.6.20.84:44358> <http://10.6.20.84:44358> > > > <http://10.6.20.84:44358> > > > > <http://10.6.20.84:44358> <- > >10.6.20.84:6641 <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> ) at > lib/stream- > > fd.c:157 > > > (34% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:15.370Z|04286|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:15.370Z|04287|reconnect|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > entering > > ACTIVE > > > > > 2020-07- > > > > > > 31T23:08:15.370Z|04288|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:16.236Z|04289|poll_loop|DBG|wakeup due to > > > > 0-ms > > > > > timeout at tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> (100% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:16.236Z|04290|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > received reply, > > > result=[], id="echo" > > > > > 2020-07- > > > 31T23:08:17.778Z|04291|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 9 (10.6.20.84:49158 > <http://10.6.20.84:49158> > > <http://10.6.20.84:49158> <http://10.6.20.84:49158> > > > <http://10.6.20.84:49158> > > > > <http://10.6.20.84:49158> <- > >10.6.20.85:6642 <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> ) at > lib/stream- > > fd.c:157 > > > (100% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:17.778Z|04292|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:17.778Z|04293|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:20.372Z|04294|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 8 (10.6.20.84:44358 > <http://10.6.20.84:44358> > > <http://10.6.20.84:44358> <http://10.6.20.84:44358> > > > <http://10.6.20.84:44358> > > > > <http://10.6.20.84:44358> <- > >10.6.20.84:6641 <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> ) at > lib/stream- > > fd.c:157 > > > (41% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:20.372Z|04295|reconnect|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : idle > 5002 ms, > > sending > > > inactivity probe > > > > > 2020-07- > > > > > > 31T23:08:20.372Z|04296|reconnect|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > entering IDLE > > > > > 2020-07- > > > > > > 31T23:08:20.372Z|04297|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : send > request, > > > method="echo", params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:20.372Z|04298|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:20.372Z|04299|reconnect|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > entering > > ACTIVE > > > > > 2020-07- > > > > > > 31T23:08:20.372Z|04300|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:20.376Z|04301|hmap|DBG|Dropped 181 log > > > > messages in > > > > > last 6 seconds (most recently, 1 > seconds > > ago) due > > > to excessive > > > > rate > > > > > 2020-07- > > > 31T23:08:20.376Z|04302|hmap|DBG|northd/ovn- > > > > northd.c:595: 2 > > > > > buckets with 6+ nodes, including > 2 > > buckets with 6 > > > nodes (256 > > > > nodes total > > > > > across 256 buckets) > > > > > 2020-07- > > > 31T23:08:21.222Z|04303|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 8 (10.6.20.84:44358 > <http://10.6.20.84:44358> > > <http://10.6.20.84:44358> <http://10.6.20.84:44358> > > > <http://10.6.20.84:44358> > > > > <http://10.6.20.84:44358> <- > >10.6.20.84:6641 <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> ) at > lib/stream- > > fd.c:157 > > > (41% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:21.223Z|04304|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > received reply, > > > result=[], id="echo" > > > > > 2020-07- > > > 31T23:08:22.779Z|04305|poll_loop|DBG|wakeup due to > > > > 706-ms > > > > > timeout at lib/reconnect.c:643 > (41% CPU > > usage) > > > > > 2020-07- > > > > > > 31T23:08:22.779Z|04306|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : idle > 5001 ms, > > sending > > > inactivity probe > > > > > 2020-07- > > > > > > 31T23:08:22.779Z|04307|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > entering IDLE > > > > > 2020-07- > > > > > > 31T23:08:22.779Z|04308|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : send > request, > > > method="echo", params=[], > > > > > id="echo" > > > > > 2020-07- > > > 31T23:08:23.632Z|04309|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 9 (10.6.20.84:49158 > <http://10.6.20.84:49158> > > <http://10.6.20.84:49158> <http://10.6.20.84:49158> > > > <http://10.6.20.84:49158> > > > > <http://10.6.20.84:49158> <- > >10.6.20.85:6642 <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> ) at > lib/stream- > > fd.c:157 > > > (79% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:23.632Z|04310|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:23.632Z|04311|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > entering > > ACTIVE > > > > > 2020-07- > > > > > > 31T23:08:23.632Z|04312|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:24.481Z|04313|poll_loop|DBG|wakeup due to > > > > 0-ms > > > > > timeout at tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> (79% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:24.481Z|04314|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > received reply, > > > result=[], id="echo" > > > > > 2020-07- > > > 31T23:08:25.372Z|04315|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 8 (10.6.20.84:44358 > <http://10.6.20.84:44358> > > <http://10.6.20.84:44358> <http://10.6.20.84:44358> > > > <http://10.6.20.84:44358> > > > > <http://10.6.20.84:44358> <- > >10.6.20.84:6641 <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> ) at > lib/stream- > > fd.c:157 > > > (79% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:25.372Z|04316|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:25.372Z|04317|jsonrpc|DBG|tcp:10.6.20.84:6641 > <http://10.6.20.84:6641> > > <http://10.6.20.84:6641> > > > <http://10.6.20.84:6641> > > > > <http://10.6.20.84:6641> > > > > > <http://10.6.20.84:6641> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:28.634Z|04318|poll_loop|DBG|wakeup due to > > > > [POLLIN] > > > > > on fd 9 (10.6.20.84:49158 > <http://10.6.20.84:49158> > > <http://10.6.20.84:49158> <http://10.6.20.84:49158> > > > <http://10.6.20.84:49158> > > > > <http://10.6.20.84:49158> <- > >10.6.20.85:6642 <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> ) at > lib/stream- > > fd.c:157 > > > (50% CPU usage) > > > > > 2020-07- > > > > > > 31T23:08:28.634Z|04319|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : idle > 5002 ms, > > sending > > > inactivity probe > > > > > 2020-07- > > > > > > 31T23:08:28.634Z|04320|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > entering IDLE > > > > > 2020-07- > > > > > > 31T23:08:28.634Z|04321|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : send > request, > > > method="echo", params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:28.634Z|04322|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > received > > request, > > > method="echo", > > > > params=[], > > > > > id="echo" > > > > > 2020-07- > > > > > > 31T23:08:28.634Z|04323|reconnect|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : > entering > > ACTIVE > > > > > 2020-07- > > > > > > 31T23:08:28.634Z|04324|jsonrpc|DBG|tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > <http://10.6.20.85:6642> > > > <http://10.6.20.85:6642> > > > > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> : send > reply, > > result=[], > > > id="echo" > > > > > 2020-07- > > > 31T23:08:28.638Z|04325|hmap|DBG|Dropped 266 log > > > > messages in > > > > > last 9 seconds (most recently, 3 > seconds > > ago) due > > > to excessive > > > > rate > > > > > 2020-07- > > > 31T23:08:28.638Z|04326|hmap|DBG|northd/ovn- > > > > northd.c:595: 2 > > > > > buckets with 6+ nodes, including > 2 > > buckets with 6 > > > nodes (256 > > > > nodes total > > > > > across 256 buckets) > > > > > ======== > > > > > > > > > > Thanks! > > > > > > > > > > Tony > > > > > > > > > > > > > _______________________________________________ > > > > > discuss mailing list > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > discuss mailing list > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs- > > discuss > > > > > > > > > > > > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
