Inline... (please read with monospaced font:)) Thanks!
Tony > -----Original Message----- > From: Han Zhou <hz...@ovn.org> > Sent: Wednesday, August 5, 2020 11:48 PM > To: Tony Liu <tonyliu0...@hotmail.com> > Cc: Han Zhou <hz...@ovn.org>; Numan Siddique <num...@ovn.org>; ovs-dev > <ovs-...@openvswitch.org>; ovs-discuss <ovs-discuss@openvswitch.org> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > wrote: > > > I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node > of cluster. Also ovn-openflow-probe-interval=30 on compute node. > It seems helping. Not that many connect/drop/reconnect in logging. > That "commit failure" is also gone. > The issue I reported in another thread "packet drop" seems gone. > And launching VM starts working. > > How should I set connection table for all ovn-nb-db and ovn-sb-db > nodes in the cluster to set inactivity_probe? > One row with address 0.0.0.0 seems not working. > > You can simply use 0.0.0.0 in the connection table, but don't specify > the same connection method on the command line when starting ovsdb- > server for NB/SB DB. Otherwise, these are conflicting and that's why you > saw "Address already in use" error. Could you share a bit details how it works? I thought the row in connection table only tells nbdb and sbdb the probe interval. Isn't that right? Does nbdb and sbdb also create socket based on target column? > > Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on > compute node for ovn-controller to probe ovn-sb-db? > > OVSDB probe is bidirectional, so you need to set this value, too, if you > don't want too many probes handled by the SB server. (setting the > connection table for SB only changes the server side). In that case, how do I set probe interval for ovn-controller? My understanding is that, ovn-controller reads configuration from ovsdb-server on the local compute node. Isn't that right? > Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on > compute node for ovn-controller to probe ovsdb-server? > > It is for the OpenFlow connection between ovn-controller and ovs- > vswitchd, which is part of the OpenFlow protocol. > > What's probe interval for ovsdb-server to probe ovn-controller? > > The local ovsdb connection uses unix socket, which doesn't send probe by > default (if I remember correctly). Here is how ovsdb-server and ovn-controller is invoked on compute node. ======== root 41129 0.0 0.0 157556 20532 ? S Jul30 1:51 /usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --remote=ptcp:6640:127.0.0.1 --remote=db:Open_vSwitch,Open_vSwitch,manager_options --log-file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile root 63775 55.9 0.4 1477796 1224324 ? Sl Aug04 1360:55 /usr/bin/ovn-controller --pidfile=/run/ovn/ovn-controller.pid --log-file=/var/log/kolla/openvswitch/ovn-controller.log tcp:127.0.0.1:6640 ======== Is that OK? Or UNIX socket method is recommended for ovn-controller to connect to ovsdb-server? Here is the configuration in open_vswitch table in ovsdb-server. ======== external_ids : {ovn-encap-ip="10.6.30.22", ovn-encap-type=geneve, ovn-openflow-probe-interval="30", ovn-remote="tcp:10.6.20.84:6642,tcp:10.6.20.85:6642,tcp:10.6.20.86:6642", ovn-remote-probe-interval="60000", system-id="compute-3"} ======== ovn-controller connects to ovsdb-server and reads this configuration, so it knows how to connect to all sbdb nodes, right? If it's TCP between ovn-controller and ovsdb-server, is that probe interval setting will also apply to the probe from ovn-controller to ovsdb-server? ovn-controller connects to ovs-vswitchd by UNIX socket to program open-flow. ovs-vswitchd and ovsdb-server are connected by UNIX too. So, is that ovn-openflow-probe-interval for the probe from ovn-controller to ovs-vswitchd via UNIX? As a summary for the probe setting, +--------------+ driver configuration | ovn-driver | +--------------+ ^ | | v +--------------+ inactivity_probe in table "Connection" | ovn-nb-db | +--------------+ ^ | | v +--------------+ options:northd_probe_interval in table "NB_Global" | ovn-northd | in nbdb. +--------------+ ^ | | v +--------------+ inactivity_probe in table "Connection" | ovn-sb-db | +--------------+ ^ | | v +--------------------------------+ in table "Open_vSwitch" in ovsdb-server | ovn-controller | ovn-remote-probe-interval for TCP +--------------------------------+ probe to ovsdb-server, ^ | ^ | ovn-openflow-probe-interval for UNIX | v TCP | v UNIX probe to ovs-vswitchd +--------------+ +--------------+ | ovsdb-server | | ovs-vswitchd | +--------------+ +--------------+ Is that correct? > For ovn-controller, since it is implemented with incremental-processing, > even if there are probes from openflow or local ovsdb, it doesn't matter. > If there is no configuration change, ovn-controller simply replies the > probe and there is no extra cost. I did see errors about connecting to br-mgmt (UNIX connection to ovs-vswitchd) in ovn-controller logging. After setting proble internal, those errors are gone. It seems that it does matter. > > > > > Thanks! > > Tony > > -----Original Message----- > > From: discuss <ovs-discuss-boun...@openvswitch.org <mailto:ovs- > discuss-boun...@openvswitch.org> > On Behalf Of Tony > > Liu > > Sent: Wednesday, August 5, 2020 4:29 PM > > To: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > > > Cc: ovs-dev <ovs-...@openvswitch.org <mailto:ovs- > d...@openvswitch.org> >; ovs-discuss <ovs- > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > Hi Han, > > > > After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db, > I see > > this error. > > ======== > > 2020-08- > > 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0 > <http://0.0.0.0> : > > listen failed: Address already in use ======== Anything I am > missing > > here? > > > > > > Thanks! > > > > Tony > > > -----Original Message----- > > > From: Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> > > > > Sent: Tuesday, August 4, 2020 4:44 PM > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > Cc: Numan Siddique <num...@ovn.org <mailto:num...@ovn.org> >; > Han Zhou <hz...@ovn.org <mailto:hz...@ovn.org> >; ovs- > > > discuss <ovs-discuss@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> >; ovs-dev > > > <ovs-...@openvswitch.org <mailto:ovs-...@openvswitch.org> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity > probe > > > > > > > > > > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > > Hi, > > > > > > Since I have 3 OVN DB nodes, should I add 3 rows in > connection > > table > > > for the inactivity_probe? Or put 3 addresses into one row? > > > > > > "set-connection" set one row only, and there is no "add- > connection". > > > How should I add 3 rows into the table connection? > > > > > > > > > > > > > > > You only need to set one row. Try this command: > > > > > > ovn-nbctl -- --id=@conn_uuid create Connection > > > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set > NB_Global . > > > connections=@conn_uuid > > > > > > > > > > > > Thanks! > > > > > > Tony > > > > > > > -----Original Message----- > > > > From: Numan Siddique <num...@ovn.org > <mailto:num...@ovn.org> <mailto:num...@ovn.org > <mailto:num...@ovn.org> > > > > > > Sent: Tuesday, August 4, 2020 12:36 AM > > > > To: Tony Liu <tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > > Cc: ovs-discuss <ovs-discuss@openvswitch.org <mailto:ovs- > disc...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > >; > ovs-dev <ovs- > > > > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > Subject: Re: [ovs-discuss] [OVN] no response to > inactivity probe > > > > > > > > > > > > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu > <tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > wrote: > > > > > > > > > > > > In my deployment, on each Neutron server, there are > 13 > > > Neutron > > > > server processes. > > > > I see 12 of them (monitor, maintenance, RPC, API) > connect > > > to both > > > > ovn-nb-db > > > > and ovn-sb-db. With 3 Neutron server nodes, that's > 36 OVSDB > > > clients. > > > > Is so many clients OK? > > > > > > > > Any suggestions how to figure out which side > doesn't > > > respond the > > > > probe, > > > > if it's bi-directional? I don't see any activities > from > > > logging, > > > > other than > > > > connect/drop and reconnect... > > > > > > > > BTW, please let me know if this is not the right > place to > > > discuss > > > > Neutron OVN > > > > ML2 driver. > > > > > > > > > > > > Thanks! > > > > > > > > Tony > > > > > > > > > -----Original Message----- > > > > > From: dev <ovs-dev-boun...@openvswitch.org > <mailto:ovs-dev-boun...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > > dev-boun...@openvswitch.org <mailto:dev- > boun...@openvswitch.org> > <mailto:ovs-dev- <mailto:ovs-dev-> > <mailto:ovs-dev- <mailto:ovs-dev-> > > > > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > <mailto:boun...@openvswitch.org <mailto:boun...@openvswitch.org> > > > > On > > > Behalf Of Tony Liu > > > > > Sent: Monday, August 3, 2020 7:45 PM > > > > > To: ovs-discuss <ovs-discuss@openvswitch.org > <mailto:ovs-discuss@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > > > > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > >; > > > ovs-dev <ovs- > > > > > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > Subject: [ovs-dev] [OVN] no response to > inactivity probe > > > > > > > > > > Hi, > > > > > > > > > > Neutron OVN ML2 driver was disconnected by ovn- > nb-db. > > > There are > > > > many > > > > > error messages from ovn-nb-db leader. > > > > > ======== > > > > > 2020-08- > > > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620 > <http://10.6.20.81:58620> > > > <http://10.6.20.81:58620> > > > > <http://10.6.20.81:58620> : no > > > > > response to inactivity probe after 5 seconds, > > > disconnecting > > > > > 2020-08- > > > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300 > <http://10.6.20.81:58300> > > > <http://10.6.20.81:58300> > > > > <http://10.6.20.81:58300> : no > > > > > response to inactivity probe after 5 seconds, > > > disconnecting > > > > > 2020-08- > > > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582 > <http://10.6.20.81:59582> > > > <http://10.6.20.81:59582> > > > > <http://10.6.20.81:59582> : no > > > > > response to inactivity probe after 5 seconds, > > > disconnecting > > > > > 2020-08- > > > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626 > <http://10.6.20.83:42626> > > > <http://10.6.20.83:42626> > > > > <http://10.6.20.83:42626> : no > > > > > response to inactivity probe after 5 seconds, > > > disconnecting > > > > > 2020-08- > > > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412 > <http://10.6.20.82:45412> > > > <http://10.6.20.82:45412> > > > > <http://10.6.20.82:45412> : no > > > > > response to inactivity probe after 5 seconds, > > > disconnecting > > > > > 2020-08- > > > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416 > <http://10.6.20.81:59416> > > > <http://10.6.20.81:59416> > > > > <http://10.6.20.81:59416> : no > > > > > response to inactivity probe after 5 seconds, > > > disconnecting > > > > > 2020-08- > > > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004 > <http://10.6.20.81:60004> > > > <http://10.6.20.81:60004> > > > > <http://10.6.20.81:60004> : no > > > > > response to inactivity probe after 5 seconds, > > > disconnecting > > > > ======== > > > > > > > > > > Could anyone share a bit details how this > inactivity > > > probe works? > > > > > > > > > > > > > > > > The inactivity probe is sent by both the server and > clients > > > > independently. > > > > Meaning ovsdb-server will send an inactivity probe every > 'x' > > > configured > > > > seconds to all its connected clients and if it doesn't > get a > > reply > > > from > > > > the client within some time, it disconnects the > connection. > > > > > > > > The inactivity probe from the server side can be > configured. Run > > > "ovn- > > > > nbctl list connection" > > > > and you will see inactivity_probe column. You can set > this column > > > to > > > > desired value like - ovn-nbctl set connection . > > > inactivity_probe=30000 > > > > (for 30 seconds) > > > > > > > > The same thing for SB ovsdb-server. > > > > > > > > Similarly each client (ovn-northd, ovn-controller, > neutron server) > > > sends > > > > inactivity probe every 'y' seconds and if the client > doesn't get > > > any > > > > reply from ovsdb-server it will disconnect the connection > and > > > reconnect > > > > again. > > > > > > > > For ovn-northd you can configured this as - ovn-nbctl set > > NB_Global > > > . > > > > options:northd_probe_interval=30000 > > > > > > > > For ovn-controllers - ovs-vsctl set open . > external_ids:ovn- > > > remote- > > > > probe-interval=30000 > > > > > > > > There is also a probe interval for openflow connection > from ovn- > > > > controller to ovs-vswitchd which you can configure as > ovs-vsctl > > set > > > > open . external_ids:ovn-openflow-probe-interval=30 (this > is in > > > seconds) > > > > > > > > > > > > Regarding the neutron server I think it is set to 60 > seconds. > > > Please see > > > > this - > > > > > > > > > > https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml > > > > 2/drivers/ovn/ovn_conf.py#L80 > > > > > > > > From the logs you shared, it looks like ovsdb-server is > not > > getting > > > the > > > > probe reply from neutron server after 5 seconds and hence > it is > > > > disconnecting. Not sure what's happening though. > > > > > > > > You can try increasing the inactivity probe interval on > the > > > ovsdb-server > > > > side with the first command I shared. > > > > Note: If "ovn-nbctl list connection" returns empty, you > need to > > > create a > > > > connection row like - ovn-nbctl set-connection > ptcp:6641:<IP> > > > > > > > > > > > > Thanks > > > > Numan > > > > > > > > > > > > > > > > > > > > > From OVN ML2 driver log, I see it connected to > the leader, > > > then > > > > the > > > > > connection was closed by leader after 5 or 6 > seconds. Is > > > this > > > > probe one- > > > > > way or two-ways? > > > > > Both sides are not busy, not taking much CPU > cycles. Not > > > sure how > > > > this > > > > > could happen. Any thoughts? > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > Tony > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > dev mailing list > > > > > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > _______________________________________________ > > > > discuss mailing list > > > > disc...@openvswitch.org > <mailto:disc...@openvswitch.org> <mailto:disc...@openvswitch.org > <mailto:disc...@openvswitch.org> > > > > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > <mailto:disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs- > discuss > > > > > > > > > > > > > > > > > > _______________________________________________ > > discuss mailing list > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
