Good to know, thanks! Tony > -----Original Message----- > From: Han Zhou <hz...@ovn.org> > Sent: Friday, August 7, 2020 12:36 PM > To: Tony Liu <tonyliu0...@hotmail.com> > Cc: Han Zhou <hz...@ovn.org>; Numan Siddique <num...@ovn.org>; ovs-dev > <ovs-dev@openvswitch.org>; ovs-discuss <ovs-disc...@openvswitch.org> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > The raft probe is disabled if you use the latest version of OVS, e.g. > 2.13.1. > > > On Fri, Aug 7, 2020 at 12:28 PM Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > wrote: > > > Another one here, there is inactivity probe on the raft cluster > port. > ======== > 2020-08-07T19:04:53.184Z|02735|reconnect|ERR|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : no response to inactivity probe after 5 > seconds, disconnecting > 2020-08-07T19:04:53.184Z|02736|reconnect|INFO|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : connection dropped > 2020-08-07T19:04:54.185Z|02737|reconnect|INFO|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : connecting... > 2020-08-07T19:04:54.185Z|02738|reconnect|INFO|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : connected > 2020-08-07T19:15:26.228Z|02739|reconnect|ERR|tcp:10.6.20.84:49440 > <http://10.6.20.84:49440> : no response to inactivity probe after 5 > seconds, disconnecting > 2020-08-07T19:15:26.769Z|02740|reconnect|ERR|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : no response to inactivity probe after 5 > seconds, disconnecting > 2020-08-07T19:15:26.769Z|02741|reconnect|INFO|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : connection dropped > 2020-08-07T19:15:27.771Z|02742|reconnect|INFO|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : connecting... > 2020-08-07T19:15:27.771Z|02743|reconnect|INFO|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : connected > ======== > Which configuration is for that probe interval? > > > Thanks! > > Tony > > > -----Original Message----- > > From: dev <ovs-dev-boun...@openvswitch.org <mailto:ovs-dev- > boun...@openvswitch.org> > On Behalf Of Tony Liu > > Sent: Thursday, August 6, 2020 7:45 PM > > To: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> >; Numan > Siddique <num...@ovn.org <mailto:num...@ovn.org> > > > Cc: ovs-dev <ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> >; ovs-discuss <ovs- > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to > inactivity > > probe > > > > Hi Han and Numan, > > > > I'd like to have a few more clarifications. > > > > For inactivity probe: > > From ovn-controller to ovn-sb-db: ovn-remote-probe-interval > > > > From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval > > > > From ovn-controller to local ovsdb: which interval? > > > > From local ovsdb to ovn-controller: which interval? > > > > From ovs-vswitchd to ovn-controller: which interval? > > > > > > Regarding to the connection between ovn-controller and local > ovsdb- > > server, I recall that UNIX socket is lighter than TCP socket and > UNIX > > socket is recommended for local communication. > > Is that right? > > > > > > Thanks! > > > > Tony > > > > > -----Original Message----- > > > From: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > > > > Sent: Thursday, August 6, 2020 12:42 PM > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > Cc: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> >; Numan > Siddique <num...@ovn.org <mailto:num...@ovn.org> >; ovs-dev > > > <ovs-dev@openvswitch.org <mailto:ovs-dev@openvswitch.org> >; > ovs-discuss <ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity > probe > > > > > > > > > > > > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > Inline... > > > > > > > > Thanks! > > > > > > > > Tony > > > > > -----Original Message----- > > > > > From: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > Sent: Thursday, August 6, 2020 11:37 AM > > > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > Cc: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > >; Numan > > > > > Siddique <num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > >; ovs-dev > > > > > <ovs-dev@openvswitch.org <mailto:ovs-dev@openvswitch.org> > <mailto:ovs-dev@openvswitch.org <mailto:ovs-dev@openvswitch.org> > >; > > > > > ovs-discuss <ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity > probe > > > > > > > > > > > > > > > > > > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > wrote: > > > > > > > > > > > > Inline... (please read with monospaced font:)) > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Tony > > > > > > > -----Original Message----- > > > > > > > From: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > > > Sent: Wednesday, August 5, 2020 11:48 PM > > > > > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > Cc: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > >; Numan > > > > > > > Siddique <num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > >; ovs-dev > > > > > > > <ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > > > >; ovs-discuss <ovs-disc...@openvswitch.org > <mailto:ovs-disc...@openvswitch.org> > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > > > > Subject: Re: [ovs-discuss] [OVN] no response to > inactivity > > > > > > > probe > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu > > > > > > > <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I set the connection > target="ptcp:6641:10.6.20.84" for > > > > > > > ovn-nb- > > > > > db > > > > > > > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is > the > > > > > > > first > > > > > node > > > > > > > of cluster. Also ovn-openflow-probe-interval=30 > on > > > > > > > compute > > > > > node. > > > > > > > It seems helping. Not that many > connect/drop/reconnect > > > > > > > in > > > > > logging. > > > > > > > That "commit failure" is also gone. > > > > > > > The issue I reported in another thread "packet > drop" > > > > > > > seems > > > > > gone. > > > > > > > And launching VM starts working. > > > > > > > > > > > > > > How should I set connection table for all ovn-nb- > db and > > > > > > > ovn- > > > > > sb-db > > > > > > > nodes in the cluster to set inactivity_probe? > > > > > > > One row with address 0.0.0.0 seems not working. > > > > > > > > > > > > > > You can simply use 0.0.0.0 in the connection table, but > don't > > > > > > > specify the same connection method on the command line > when > > > > > > > starting > > > > > > > ovsdb- server for NB/SB DB. Otherwise, these are > conflicting > > > > > > > and that's why you saw "Address already in use" error. > > > > > > > > > > > > Could you share a bit details how it works? > > > > > > I thought the row in connection table only tells nbdb and > sbdb > > > > > > the probe interval. Isn't that right? Does nbdb and sbdb > also > > > > > > create socket based on target column? > > > > > > > > > > > > > > > > > > > > > In --remote option of ovsdb-server, you can specify either > a > > > > > connection method directly, or specify the db,table,column > which > > > > > contains the connection information. > > > > > Please see manpage ovsdb-server(1). > > > > > > > > Here is how one of those 3 nbdb nodes invoked. > > > > ======== > > > > ovsdb-server -vconsole:off -vfile:info > > > > --log-file=/var/log/kolla/openvswitch/ovn-sb-db.log > > > > --remote=punix:/var/run/ovn/ovnsb_db.sock -- > > > pidfile=/run/ovn/ovnsb_db.pid -- > unixctl=/var/run/ovn/ovnsb_db.ctl -- > > > remote=db:OVN_Southbound,SB_Global,connections --private- > > > key=db:OVN_Southbound,SSL,private_key -- > > > certificate=db:OVN_Southbound,SSL,certificate --ca- > > > cert=db:OVN_Southbound,SSL,ca_cert --ssl- > > > protocols=db:OVN_Southbound,SSL,ssl_protocols --ssl- > > > ciphers=db:OVN_Southbound,SSL,ssl_ciphers > > > --remote=ptcp:6642:10.6.20.84 /var/lib/openvswitch/ovn-sb/ov > sb.db > > > ======== It creates UNIX and TCP sockets, and takes > configuration from > > DB. > > > > Does that look ok? > > > > Given that, what the target column should be for all nodes of > the > > > cluster? > > > > And whatever target is set, ovsdb-server will create socket, > right? > > > > Oh... Should I do "--remote=ptcp:6642:0.0.0.0"? Then I can > set the > > > > same in connection table, and it won't cause conflict? > > > > If --remote and connection target are the same, whoever comes > in > > > > later will be ignored, right? > > > > In coding, does ovsdb-server create a connection object for > each of > > > > --remote and connection target, or it's one single connection > object > > > > for both of them because method:port:address is the same? I'd > expect > > > > the single object. > > > > > > > > > > > > > --remote=ptcp:6642:10.6.20.84 should be removed from the > command. > > > > > > You already specifies -- > remote=db:OVN_Southbound,SB_Global,connections > > > which should contain ptcp:6642:0.0.0.0 If you set both, it will > result > > > in conflict when ovsdb-server tries to create both sockets with > same > > > port. > > > > > > > > > > > > > Is "external_ids:ovn-remote-probe-interval" in > ovsdb- > > > server on > > > > > > > compute node for ovn-controller to probe ovn-sb- > db? > > > > > > > > > > > > > > OVSDB probe is bidirectional, so you need to set this > value, > > > > > > > too, if you don't want too many probes handled by the > SB > > server. > > > > > > > (setting the connection table for SB only changes the > server > > > side). > > > > > > > > > > > > In that case, how do I set probe interval for ovn- > controller? > > > > > > My understanding is that, ovn-controller reads > configuration > > > > > > from ovsdb-server on the local compute node. Isn't that > right? > > > > > > > > > > > > > > > > > > > > > The configuration you mentioned "external_ids:ovn-remote- > probe- > > > interval" > > > > > is exactly the way to set the ovn-controller -> SB probe > interval. > > > > > (SB -> ovn-controller probe is set in the connection table > of SB) > > > > > > > > > > > > > > > You are right that ovn-controller reads configuration from > the > > > > > local ovsdb-server. This setting is in local ovsdb-server. > > > > > > > > > > > > > > > > > Is "external_ids:ovn-openflow-probe-interval" in > > > > > > > ovsdb-server > > > > > on > > > > > > > compute node for ovn-controller to probe ovsdb- > server? > > > > > > > > > > > > > > It is for the OpenFlow connection between ovn- > controller and > > > > > > > ovs- vswitchd, which is part of the OpenFlow protocol. > > > > > > > > > > > > > > What's probe interval for ovsdb-server to probe > ovn- > > > controller? > > > > > > > > > > > > > > The local ovsdb connection uses unix socket, which > doesn't > > > > > > > send probe by default (if I remember correctly). > > > > > > > > > > > > Here is how ovsdb-server and ovn-controller is invoked on > > > > > > compute > > > node. > > > > > > ======== > > > > > > root 41129 0.0 0.0 157556 20532 ? S Jul30 > 1:51 > > > > > /usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db - > vconsole:emer > > > > > - vsyslog:err -vfile:info -- > remote=punix:/run/openvswitch/db.sock > > > > > -- > > > > > remote=ptcp:6640:127.0.0.1 -- > > > > > remote=db:Open_vSwitch,Open_vSwitch,manager_options --log- > > > > > file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile > > > > > > > > > > > > root 63775 55.9 0.4 1477796 1224324 ? Sl Aug04 > > 1360:55 > > > > > /usr/bin/ovn-controller --pidfile=/run/ovn/ovn- > controller.pid > > > > > --log- file=/var/log/kolla/openvswitch/ovn-controller.log > > > > > tcp:127.0.0.1:6640 <http://127.0.0.1:6640> > <http://127.0.0.1:6640> <http://127.0.0.1:6640> > > > > > > ======== > > > > > > Is that OK? Or UNIX socket method is recommended for > > > > > > ovn-controller > > > > > > > > > > > to connect to ovsdb-server? > > > > > > > > > > If using TCP, by default it is 5s probe interval. I think > it is > > > > > better to use unix socket. (but maybe it doesn't matter > that much) > > > > > > > > > > > > > > > > > > > > > > Here is the configuration in open_vswitch table in ovsdb- > server. > > > > > > ======== > > > > > > external_ids : {ovn-encap-ip="10.6.30.22", ovn- > encap- > > > > > type=geneve, ovn-openflow-probe-interval="30", ovn- > > > > > remote="tcp:10.6.20.84:6642 <http://10.6.20.84:6642> > <http://10.6.20.84:6642> > > > > > <http://10.6.20.84:6642> ,tcp:10.6.20.85:6642 > <http://10.6.20.85:6642> > > > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642> > > > > > ,tcp:10.6.20.86:6642 <http://10.6.20.86:6642> > <http://10.6.20.86:6642> > > > > > <http://10.6.20.86:6642> ", ovn-remote-probe- > interval="60000", > > > > > system-id="compute-3"} > > > > > > ======== > > > > > > ovn-controller connects to ovsdb-server and reads this > > > > > > configuration, so it knows how to connect to all sbdb > nodes, > > right? > > > > > > > > > > > > > > > > Yes, you are right. > > > > > > > > > > > If it's TCP between ovn-controller and ovsdb-server, is > that > > > > > > probe interval setting will also apply to the probe from > > > > > > ovn-controller to ovsdb-server? > > > > > > > > > > > > > > > > No. They are not related. > > > > > > > > > > > > > > > > ovn-controller connects to ovs-vswitchd by UNIX socket to > > > > > > program open-flow. ovs-vswitchd and ovsdb-server are > connected > > by UNIX too. > > > > > > So, is that ovn-openflow-probe-interval for the probe > from > > > > > > ovn-controller to ovs-vswitchd via UNIX? > > > > > > > > > > > > As a summary for the probe setting, > > > > > > > > > > > > +--------------+ driver configuration > > > > > > | ovn-driver | > > > > > > +--------------+ > > > > > > ^ | > > > > > > | v > > > > > > +--------------+ inactivity_probe in table "Connection" > > > > > > | ovn-nb-db | > > > > > > +--------------+ > > > > > > ^ | > > > > > > | v > > > > > > +--------------+ options:northd_probe_interval in table > > > "NB_Global" > > > > > > | ovn-northd | in nbdb. > > > > > > +--------------+ > > > > > > ^ | > > > > > > | v > > > > > > +--------------+ inactivity_probe in table "Connection" > > > > > > | ovn-sb-db | > > > > > > +--------------+ > > > > > > ^ | > > > > > > | v > > > > > > +--------------------------------+ in table > "Open_vSwitch" in > > > > > > +--------------------------------+ ovsdb-server > > > > > > | ovn-controller | ovn-remote-probe- > interval > > for > > > TCP > > > > > > +--------------------------------+ probe to ovsdb-server, > > > > > > ^ | ^ | ovn-openflow-probe- > interval > > > for > > > > > UNIX > > > > > > | v TCP | v UNIX probe to ovs-vswitchd > > > > > > +--------------+ +--------------+ > > > > > > | ovsdb-server | | ovs-vswitchd | > > > > > > +--------------+ +--------------+ > > > > > > > > > > > > > > > > > Is that correct? > > > > > > > > > > Correct. Except that you don't have to use TCP between > > > > > ovn-controller and the local ovsdb-server. Use UNIX and > then you > > > > > don't need to worry about the probe between them. > > > > > > > > > > > > > > > > > > > > > > > For ovn-controller, since it is implemented with > > > > > > > incremental-processing, even if there are probes from > openflow > > > > > > > or > > > > > local ovsdb, it doesn't matter. > > > > > > > If there is no configuration change, ovn-controller > simply > > > > > > > replies the probe and there is no extra cost. > > > > > > > > > > > > I did see errors about connecting to br-mgmt (UNIX > connection to > > > > > > ovs-vswitchd) in ovn-controller logging. After setting > proble > > > > > > internal, those errors are gone. It seems that it does > matter. > > > > > > > > > > > > > > > > What's the error log? (I suspect maybe it is just some log > but not > > > > > error) > > > > > > > > This is from ovn-controller. > > > > ======== > > > > 2020-08- > 06T03:53:08.663Z|128534|rconn|ERR|unix:/var/run/openvswitch/ > > > > br > > > > -int.mgmt: no response to inactivity probe after 10 seconds, > > > > disconnecting ======== It seems that there is probe on UNIX > socket. > > > > > > > > > > OK, if you see this it seems ovs-vswitchd is busy. If it > happens > > > occasionally due to the scale then disabling the probe using > ovn- > > > openflow-probe-interval should help. > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Tony > > > > > > > > -----Original Message----- > > > > > > > > From: discuss <ovs-discuss- > boun...@openvswitch.org <mailto:ovs-discuss-boun...@openvswitch.org> > > > > > > > <mailto:ovs-discuss-boun...@openvswitch.org > <mailto:ovs-discuss-boun...@openvswitch.org> > > > > > > > > <mailto:ovs-discuss-boun...@openvswitch.org > <mailto:ovs-discuss-boun...@openvswitch.org> > > > > > > > <mailto:ovs-discuss-boun...@openvswitch.org > <mailto:ovs-discuss-boun...@openvswitch.org> > > <mailto:ovs- > <mailto:ovs-> > > > > > > > <mailto:ovs- <mailto:ovs-> > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > discuss-boun...@openvswitch.org <mailto:discuss-boun...@openvswitch.org> > > > > > <mailto:discuss-boun...@openvswitch.org <mailto:discuss- > boun...@openvswitch.org> > <mailto:discuss- <mailto:discuss-> > > > > > <mailto:discuss- <mailto:discuss-> > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > On Behalf Of Tony > > > > > > > > Liu > > > > > > > > Sent: Wednesday, August 5, 2020 4:29 PM > > > > > > > > To: Han Zhou <hz...@ovn.org > <mailto:hz...@ovn.org> <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > > > > > Cc: ovs-dev <ovs-dev@openvswitch.org > <mailto:ovs-dev@openvswitch.org> > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- > <mailto:ovs-> > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs- > > > > > > > > > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > >; > > > > > ovs-discuss <ovs- > > > > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > > > > > > > > Subject: Re: [ovs-discuss] [OVN] no response to > > > > > > > inactivity > > > > > probe > > > > > > > > > > > > > > > > Hi Han, > > > > > > > > > > > > > > > > After setting connection > target="ptcp:6642:0.0.0.0" > > > > > > > for ovn-sb-db, I see > > > > > > > > this error. > > > > > > > > ======== > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0. > > > > > > > 0 > > > > > > > <http://0.0.0.0> : > > > > > > > > listen failed: Address already in use ======== > > > > > > > Anything I am missing > > > > > > > > here? > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > Tony > > > > > > > > > -----Original Message----- > > > > > > > > > From: Han Zhou <hz...@ovn.org > <mailto:hz...@ovn.org> <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > > > > > > Sent: Tuesday, August 4, 2020 4:44 PM > > > > > > > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > > > > > > > > > > > > Cc: Numan Siddique <num...@ovn.org > <mailto:num...@ovn.org> > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > >; > Han Zhou > > > > > <hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > <mailto:hz...@ovn.org > <mailto:hz...@ovn.org> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > >; ovs- > > > > > > > > > discuss <ovs-disc...@openvswitch.org > <mailto:ovs-disc...@openvswitch.org> > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > <mailto:ovs- <mailto:ovs-> > > > > > > > <mailto:ovs- <mailto:ovs-> > <mailto:ovs- <mailto:ovs-> > <mailto:ovs- <mailto:ovs-> > > > > > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > >; ovs-dev > > > > > > > > > <ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > > > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> <mailto:ovs-dev@openvswitch.org <mailto:ovs- > d...@openvswitch.org> > > > > > > > > > > > > > > > > > > > > > > > Subject: Re: [ovs-discuss] [OVN] no response > to > > > > > > > inactivity probe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu > > > > > > > <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > Since I have 3 OVN DB nodes, should I add > 3 rows > > > in > > > > > > > connection > > > > > > > > table > > > > > > > > > for the inactivity_probe? Or put 3 > addresses > > into > > > one > > > > > row? > > > > > > > > > > > > > > > > > > "set-connection" set one row only, and > there is > > no > > > > > "add- > > > > > > > connection". > > > > > > > > > How should I add 3 rows into the table > > connection? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You only need to set one row. Try this > command: > > > > > > > > > > > > > > > > > > ovn-nbctl -- --id=@conn_uuid create > Connection > > > > > > > > > target="ptcp\:6641\:0.0.0.0" > inactivity_probe=0 -- > > > > > > > set NB_Global . > > > > > > > > > connections=@conn_uuid > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > Tony > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Numan Siddique <num...@ovn.org > <mailto:num...@ovn.org> > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > <mailto:num...@ovn.org <mailto:num...@ovn.org> > > > > > > > > > > > > > > > Sent: Tuesday, August 4, 2020 12:36 AM > > > > > > > > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > Cc: ovs-discuss <ovs- > disc...@openvswitch.org <mailto:ovs-disc...@openvswitch.org> > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- > <mailto:ovs-> > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > >; ovs-dev <ovs- > > > > > > > > > > d...@openvswitch.org > <mailto:d...@openvswitch.org> > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > > > > > > Subject: Re: [ovs-discuss] [OVN] no > response > > to > > > > > > > inactivity probe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu > > > > > > > <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > > > > > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In my deployment, on each Neutron > server, > > > > > there are > > > > > > > 13 > > > > > > > > > Neutron > > > > > > > > > > server processes. > > > > > > > > > > I see 12 of them (monitor, > maintenance, > > > RPC, > > > > > API) > > > > > > > connect > > > > > > > > > to both > > > > > > > > > > ovn-nb-db > > > > > > > > > > and ovn-sb-db. With 3 Neutron > server > > nodes, > > > > > that's > > > > > > > 36 OVSDB > > > > > > > > > clients. > > > > > > > > > > Is so many clients OK? > > > > > > > > > > > > > > > > > > > > Any suggestions how to figure out > which > > > side > > > > > > > doesn't > > > > > > > > > respond the > > > > > > > > > > probe, > > > > > > > > > > if it's bi-directional? I don't > see any > > > > > activities > > > > > > > from > > > > > > > > > logging, > > > > > > > > > > other than > > > > > > > > > > connect/drop and reconnect... > > > > > > > > > > > > > > > > > > > > BTW, please let me know if this > is not > > the > > > > > right > > > > > > > place to > > > > > > > > > discuss > > > > > > > > > > Neutron OVN > > > > > > > > > > ML2 driver. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > Tony > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: dev <ovs-dev- > > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > <mailto:ovs-dev-boun...@openvswitch.org <mailto:ovs-dev- > boun...@openvswitch.org> > > > > > > <mailto:ovs-dev-boun...@openvswitch.org <mailto:ovs-dev- > boun...@openvswitch.org> > > > > > <mailto:ovs-dev-boun...@openvswitch.org <mailto:ovs-dev- > boun...@openvswitch.org> > > > > > > > > > <mailto:ovs-dev-boun...@openvswitch.org <mailto:ovs- > dev-boun...@openvswitch.org> > > > > > > > <mailto:ovs-dev-boun...@openvswitch.org <mailto:ovs- > dev-boun...@openvswitch.org> > <mailto:ovs-dev- <mailto:ovs-dev-> > > > > > > > <mailto:ovs-dev- <mailto:ovs-dev-> > > > > > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > <mailto:boun...@openvswitch.org <mailto:boun...@openvswitch.org> > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > > > > > > dev-boun...@openvswitch.org <mailto:dev- > boun...@openvswitch.org> > > > > > > > <mailto:dev-boun...@openvswitch.org <mailto:dev- > boun...@openvswitch.org> > > > > > > > > <mailto:dev-boun...@openvswitch.org <mailto:dev- > boun...@openvswitch.org> > > > > > > > <mailto:dev-boun...@openvswitch.org <mailto:dev- > boun...@openvswitch.org> > > <mailto:dev- <mailto:dev-> > > > > > > > <mailto:dev- <mailto:dev-> > <mailto:dev- <mailto:dev-> > <mailto:dev- <mailto:dev-> > > > > > > > > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > <mailto:boun...@openvswitch.org <mailto:boun...@openvswitch.org> > > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > > > > > > > > > > > > <mailto:ovs-dev- <mailto:ovs-dev-> <mailto:ovs-dev- > <mailto:ovs-dev-> > <mailto:ovs-dev- <mailto:ovs-dev-> > > > > > > > <mailto:ovs-dev- <mailto:ovs-dev-> > > <mailto:ovs- > dev- <mailto:ovs-dev-> <mailto:ovs-dev- <mailto:ovs-dev-> > > > > > > > > <mailto:ovs-dev- <mailto:ovs-dev-> <mailto:ovs-dev- > <mailto:ovs-dev-> > > > > > > > > > > <mailto:ovs-dev- <mailto:ovs-dev-> <mailto:ovs-dev- > <mailto:ovs-dev-> > <mailto:ovs-dev- <mailto:ovs-dev-> > > > > > > > <mailto:ovs-dev- <mailto:ovs-dev-> > > <mailto:ovs- > dev- <mailto:ovs-dev-> <mailto:ovs-dev- <mailto:ovs-dev-> > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > dev-> > > > > > > > > > > > > boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > > > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> <mailto:boun...@openvswitch.org > <mailto:boun...@openvswitch.org> > > > > > > > > > > > > > > > > On > > > > > > > > > Behalf Of Tony Liu > > > > > > > > > > > Sent: Monday, August 3, 2020 > 7:45 PM > > > > > > > > > > > To: ovs-discuss <ovs- > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > > > > > > > <mailto:ovs-disc...@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> > <mailto:ovs- <mailto:ovs-> > > > > > > > <mailto:ovs- <mailto:ovs-> > > > > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > > > > > > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > >; > > > > > > > > > ovs-dev <ovs- > > > > > > > > > > > d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > > > > > > > > Subject: [ovs-dev] [OVN] no > response > > to > > > > > > > inactivity probe > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > Neutron OVN ML2 driver was > > disconnected > > > by > > > > > ovn- > > > > > > > nb-db. > > > > > > > > > There are > > > > > > > > > > many > > > > > > > > > > > error messages from ovn-nb-db > leader. > > > > > > > > > > > ======== > > > > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620 > <http://10.6.20.81:58620> > > > <http://10.6.20.81:58620> <http://10.6.20.81:58620> > > > <http://10.6.20.81:58620> > > > > > > > > > <http://10.6.20.81:58620> > > > > > > > > > > <http://10.6.20.81:58620> : no > > > > > > > > > > > response to inactivity probe > after 5 > > > seconds, > > > > > > > > > disconnecting > > > > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300 > <http://10.6.20.81:58300> > > > <http://10.6.20.81:58300> <http://10.6.20.81:58300> > > > <http://10.6.20.81:58300> > > > > > > > > > <http://10.6.20.81:58300> > > > > > > > > > > <http://10.6.20.81:58300> : no > > > > > > > > > > > response to inactivity probe > after 5 > > > seconds, > > > > > > > > > disconnecting > > > > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582 > <http://10.6.20.81:59582> > > > <http://10.6.20.81:59582> <http://10.6.20.81:59582> > > > <http://10.6.20.81:59582> > > > > > > > > > <http://10.6.20.81:59582> > > > > > > > > > > <http://10.6.20.81:59582> : no > > > > > > > > > > > response to inactivity probe > after 5 > > > seconds, > > > > > > > > > disconnecting > > > > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626 > <http://10.6.20.83:42626> > > > <http://10.6.20.83:42626> <http://10.6.20.83:42626> > > > <http://10.6.20.83:42626> > > > > > > > > > <http://10.6.20.83:42626> > > > > > > > > > > <http://10.6.20.83:42626> : no > > > > > > > > > > > response to inactivity probe > after 5 > > > seconds, > > > > > > > > > disconnecting > > > > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412 > <http://10.6.20.82:45412> > > > <http://10.6.20.82:45412> <http://10.6.20.82:45412> > > > <http://10.6.20.82:45412> > > > > > > > > > <http://10.6.20.82:45412> > > > > > > > > > > <http://10.6.20.82:45412> : no > > > > > > > > > > > response to inactivity probe > after 5 > > > seconds, > > > > > > > > > disconnecting > > > > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416 > <http://10.6.20.81:59416> > > > <http://10.6.20.81:59416> <http://10.6.20.81:59416> > > > <http://10.6.20.81:59416> > > > > > > > > > <http://10.6.20.81:59416> > > > > > > > > > > <http://10.6.20.81:59416> : no > > > > > > > > > > > response to inactivity probe > after 5 > > > seconds, > > > > > > > > > disconnecting > > > > > > > > > > > 2020-08- > > > > > > > > > > > > > > > > > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004 > <http://10.6.20.81:60004> > > > <http://10.6.20.81:60004> <http://10.6.20.81:60004> > > > <http://10.6.20.81:60004> > > > > > > > > > <http://10.6.20.81:60004> > > > > > > > > > > <http://10.6.20.81:60004> : no > > > > > > > > > > > response to inactivity probe > after 5 > > > seconds, > > > > > > > > > disconnecting > > > > > > > > > > ======== > > > > > > > > > > > > > > > > > > > > > > Could anyone share a bit > details how > > > this > > > > > > > inactivity > > > > > > > > > probe works? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The inactivity probe is sent by both > the > > server > > > and > > > > > > > clients > > > > > > > > > > independently. > > > > > > > > > > Meaning ovsdb-server will send an > inactivity > > > probe > > > > > every > > > > > > > 'x' > > > > > > > > > configured > > > > > > > > > > seconds to all its connected clients > and if it > > > > > doesn't > > > > > > > get a > > > > > > > > reply > > > > > > > > > from > > > > > > > > > > the client within some time, it > disconnects > > the > > > > > > > connection. > > > > > > > > > > > > > > > > > > > > The inactivity probe from the server > side can > > be > > > > > > > configured. Run > > > > > > > > > "ovn- > > > > > > > > > > nbctl list connection" > > > > > > > > > > and you will see inactivity_probe > column. You > > > can > > > > > set > > > > > > > this column > > > > > > > > > to > > > > > > > > > > desired value like - ovn-nbctl set > connection . > > > > > > > > > inactivity_probe=30000 > > > > > > > > > > (for 30 seconds) > > > > > > > > > > > > > > > > > > > > The same thing for SB ovsdb-server. > > > > > > > > > > > > > > > > > > > > Similarly each client (ovn-northd, ovn- > > > controller, > > > > > > > neutron server) > > > > > > > > > sends > > > > > > > > > > inactivity probe every 'y' seconds and > if the > > > client > > > > > > > doesn't get > > > > > > > > > any > > > > > > > > > > reply from ovsdb-server it will > disconnect the > > > > > connection > > > > > > > and > > > > > > > > > reconnect > > > > > > > > > > again. > > > > > > > > > > > > > > > > > > > > For ovn-northd you can configured this > as - > > ovn- > > > > > nbctl set > > > > > > > > NB_Global > > > > > > > > > . > > > > > > > > > > options:northd_probe_interval=30000 > > > > > > > > > > > > > > > > > > > > For ovn-controllers - ovs-vsctl set > open . > > > > > > > external_ids:ovn- > > > > > > > > > remote- > > > > > > > > > > probe-interval=30000 > > > > > > > > > > > > > > > > > > > > There is also a probe interval for > openflow > > > > > connection > > > > > > > from ovn- > > > > > > > > > > controller to ovs-vswitchd which you > can > > > configure > > > > > as > > > > > > > ovs-vsctl > > > > > > > > set > > > > > > > > > > open . external_ids:ovn-openflow-probe- > > > interval=30 > > > > > (this > > > > > > > is in > > > > > > > > > seconds) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the neutron server I think it > is set > > > to 60 > > > > > > > seconds. > > > > > > > > > Please see > > > > > > > > > > this - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/openstack/neutron/blob/master/neutron/conf/plug > > > > > in > > > > > s/ml > > > > > > > > > > 2/drivers/ovn/ovn_conf.py#L80 > > > > > > > > > > > > > > > > > > > > From the logs you shared, it looks like > ovsdb- > > > server > > > > > is > > > > > > > not > > > > > > > > getting > > > > > > > > > the > > > > > > > > > > probe reply from neutron server after 5 > > seconds > > > and > > > > > hence > > > > > > > it is > > > > > > > > > > disconnecting. Not sure what's > happening > > though. > > > > > > > > > > > > > > > > > > > > You can try increasing the inactivity > probe > > > interval > > > > > on > > > > > > > the > > > > > > > > > ovsdb-server > > > > > > > > > > side with the first command I shared. > > > > > > > > > > Note: If "ovn-nbctl list connection" > returns > > > empty, > > > > > you > > > > > > > need to > > > > > > > > > create a > > > > > > > > > > connection row like - ovn-nbctl set- > connection > > > > > > > ptcp:6641:<IP> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > Numan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From OVN ML2 driver log, I see > it > > > connected > > > > > to > > > > > > > the leader, > > > > > > > > > then > > > > > > > > > > the > > > > > > > > > > > connection was closed by leader > after > > 5 > > > or 6 > > > > > > > seconds. Is > > > > > > > > > this > > > > > > > > > > probe one- > > > > > > > > > > > way or two-ways? > > > > > > > > > > > Both sides are not busy, not > taking > > much > > > CPU > > > > > > > cycles. Not > > > > > > > > > sure how > > > > > > > > > > this > > > > > > > > > > > could happen. Any thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > Tony > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > dev mailing list > > > > > > > > > > > d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org > <mailto:d...@openvswitch.org> > > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > > > > > > > > > > > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > discuss mailing list > > > > > > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs- > > > > > > > discuss > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > discuss mailing list > > > > > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > > > > > > > > > > > > > > > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs- > discuss > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > dev mailing list > > d...@openvswitch.org <mailto:d...@openvswitch.org> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
_______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
