Re: [ovs-discuss] [OVN] no response to inactivity probe

[Tony Liu]

I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db
and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node
of cluster. Also ovn-openflow-probe-interval=30 on compute node.
It seems helping. Not that many connect/drop/reconnect in logging.
That "commit failure" is also gone.
The issue I reported in another thread "packet drop" seems gone.
And launching VM starts working.

How should I set connection table for all ovn-nb-db and ovn-sb-db
nodes in the cluster to set inactivity_probe?
One row with address 0.0.0.0 seems not working.

Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
compute node for ovn-controller to probe ovn-sb-db?

Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on
compute node for ovn-controller to probe ovsdb-server?

What's probe interval for ovsdb-server to probe ovn-controller?


Thanks!

Tony
> -Original Message-
> From: discuss  On Behalf Of Tony
> Liu
> Sent: Wednesday, August 5, 2020 4:29 PM
> To: Han Zhou 
> Cc: ovs-dev ; ovs-discuss  disc...@openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 
> Hi Han,
> 
> After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db, I see
> this error.
> 
> 2020-08-
> 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0:
> listen failed: Address already in use  Anything I am missing
> here?
> 
> 
> Thanks!
> 
> Tony
> > -Original Message-
> > From: Han Zhou 
> > Sent: Tuesday, August 4, 2020 4:44 PM
> > To: Tony Liu 
> > Cc: Numan Siddique ; Han Zhou ; ovs-
> > discuss ; ovs-dev
> > 
> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> >
> >
> >
> > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu  >  > wrote:
> >
> >
> > Hi,
> >
> > Since I have 3 OVN DB nodes, should I add 3 rows in connection
> table
> > for the inactivity_probe? Or put 3 addresses into one row?
> >
> > "set-connection" set one row only, and there is no "add-connection".
> > How should I add 3 rows into the table connection?
> >
> >
> >
> >
> > You only need to set one row. Try this command:
> >
> > ovn-nbctl -- --id=@conn_uuid create Connection
> > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global .
> > connections=@conn_uuid
> >
> >
> >
> > Thanks!
> >
> > Tony
> >
> > > -Original Message-
> > > From: Numan Siddique mailto:num...@ovn.org> >
> > > Sent: Tuesday, August 4, 2020 12:36 AM
> > > To: Tony Liu  >  >
> > > Cc: ovs-discuss mailto:ovs-
> > disc...@openvswitch.org> >; ovs-dev  > > d...@openvswitch.org  >
> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > >
> > >
> > >
> > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu  > 
> > >  >  > > wrote:
> > >
> > >
> > >   In my deployment, on each Neutron server, there are 13
> > Neutron
> > > server processes.
> > >   I see 12 of them (monitor, maintenance, RPC, API) connect
> > to both
> > > ovn-nb-db
> > >   and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB
> > clients.
> > >   Is so many clients OK?
> > >
> > >   Any suggestions how to figure out which side doesn't
> > respond the
> > > probe,
> > >   if it's bi-directional? I don't see any activities from
> > logging,
> > > other than
> > >   connect/drop and reconnect...
> > >
> > >   BTW, please let me know if this is not the right place to
> > discuss
> > > Neutron OVN
> > >   ML2 driver.
> > >
> > >
> > >   Thanks!
> > >
> > >   Tony
> > >
> > >   > -Original Message-
> > >   > From: dev mailto:ovs-
> > dev-boun...@openvswitch.org>  
> > > boun...@openvswitch.org  > > On
> > Behalf Of Tony Liu
> > >   > Sent: Monday, August 3, 2020 7:45 PM
> > >   > To: ovs-discuss mailto:ovs-
> > disc...@openvswitch.org>  
> > > disc...@openvswitch.org  > >;
> > ovs-dev  > >   > d...@openvswitch.org 
> >  > >
> > >   > Subject: [ovs-dev] [OVN] no response to inactivity probe
> > >   >
> > >   > Hi,
> > >   >
> > >   > Neutron OVN ML2 driver was disconnected by ovn-nb-db.
> > There are
> > > many
> > >   > error messages from ovn-nb-db leader.
> > >   > 
> > >   > 2020-08-
> > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> > 
> > >  : no
> > >   > response to inactivity probe after 5 

Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Girish Moodalbail]

On Wed, Aug 5, 2020 at 5:23 PM Han Zhou  wrote:

>
>
> On Wed, Aug 5, 2020 at 4:35 PM Girish Moodalbail 
> wrote:
>
>>
>>
>> On Wed, Aug 5, 2020 at 3:05 PM Han Zhou  wrote:
>>
>>>
>>>
>>> On Wed, Aug 5, 2020 at 12:51 PM Winson Wang 
>>> wrote:
>>>
 Hello OVN Experts:

 With large scale ovn-k8s cluster,  there are several conditions that
 would make ovn-controller clients connect SB central from a balanced state
 to an unbalanced state.
 Is there an ongoing project to address this problem?
 If not,  I have one proposal not sure if it is doable.
 Please share your thoughts.

 The issue:

 OVN SB RAFT 3 node cluster,  at first all the ovn-controller clients
 will connect all the 3 nodes in a balanced state.

 The following conditions will make the connections become unbalanced.

-

One RAFT node restart,  all the ovn-controller clients to reconnect
to the two remaining cluster nodes.


-

Ovn-k8s,  after SB raft pods rolling upgrade, the last raft pod has
no client connections.


 RAFT clients in an unbalanced state would trigger more stress to the
 raft cluster,  which makes the raft unstable under stress compared to a
 balanced state.
 The proposal solution:

 Ovn-controller adds next unix commands “reconnect” with argument of
 preferred SB node IP.

 When unbalanced state happens,  the UNIX command can trigger
 ovn-controller reconnect

 To new SB raft node with fast sync which doesn’t trigger the whole DB
 downloading process.


>>> Thanks Winson. The proposal sounds good to me. Will you implement it?
>>>
>>
>> Han/Winson,
>>
>> The fast re-sync is for ovsdb-server restart and it will not apply for
>> ovn-controller restart, right?
>>
>>
> Right, but the proposal is to provide a command just to reconnect, without
> restarting. In that case fast-resync should work.
>
>
>> If the ovsdb-client (ovn-controller) restarts, then it would have lost
>> all its state and when it starts again it will still need to download
>> logical_flows, port_bindings , and other tables it cares about. So, fast
>> re-sync may not apply to this case.
>>
>> Also, the ovn-controller should stash the IP address of the SB server to
>> which it is connected to in Open_vSwitch table's external_id column. It
>> updates this field whenever it re-connects to a different SB server
>> (because that ovsdb-server instance failed or restarted). When
>> ovn-controller itself restarts it could check for the value in this field
>> and try to connect to it first and on failure fallback to connect to
>> default connection approach.
>>
>
> The imbalance is usually caused by failover on server side. When one
> server is down, all clients are expected to connect to the rest of the
> servers, and when the server is back, there is no motivation for the
> clients to reconnect again (unless you purposely restart the clients, which
> would bring 1/3 of the restarted clients back to the old server). So I
> don't understand how "stash the IP address" would work in this scenario.
>
> The proposal above by Winson is to purposely trigger a reconnection
> towards the desired server without restarting the clients, which I think
> solves this problem directly.
>

Right. This is what we discussed internally, however when I read this email
on the list I got confused with the other thread (rolling update of
ovn-controller in K8s cluster which involves restart of ovn-controller).
Sorry, for the noise.

Regards,
~Girish
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Girish Moodalbail]

On Wed, Aug 5, 2020 at 5:36 PM Han Zhou  wrote:

>
>
> On Wed, Aug 5, 2020 at 4:21 PM Girish Moodalbail 
> wrote:
>
>>
>>
>> On Wed, Aug 5, 2020 at 3:35 PM Han Zhou  wrote:
>>
>>>
>>>
>>> On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
>>> wrote:
>>>
 Hello OVN Experts,

 With ovn-k8s,  we need to keep the flows always on br-int which needed
 by running pods on the k8s node.
 Is there an ongoing project to address this problem?
 If not,  I have one proposal not sure if it is doable.
 Please share your thoughts.
 The issue:

 In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
 every K8s node.  When we restart ovn-controller for upgrade using
 `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
 works fine since br-int with flows still be Installed.

 However, when a new ovn-controller starts it will connect OVS IDL and
 do an engine init run,  clearing all OpenFlow flows and install flows based
 on SB DB.

 With open flows count above 200K+,  it took more than 15 seconds to get
 all the flows installed br-int bridge again.

 Proposal solution for the issue:

 When the ovn-controller gets “exit --start”,  it will write a
 “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
 external-ids column. When new ovn-controller starts, it will check if the
 “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
 OVS IDL to decide if it will force a recomputing process?


>>> Hi Winson,
>>>
>>> Thanks for the proposal. Yes, the connection break during upgrading is a
>>> real issue in a large scale environment. However, the proposal doesn't
>>> work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
>>> which is a completely different connection from the ovs-vswitchd open-flow
>>> connection.
>>> To avoid clearing the open-flow table during ovn-controller startup, we
>>> can find a way to postpone clearing the OVS flows after the recomputing in
>>> ovn-controller is completed, right before ovn-controller replacing with the
>>> new flows. This should largely reduce the time of connection broken during
>>> upgrading. Some changes in the ofctrl module's state machine are required,
>>> but I am not 100% sure if this approach is applicable. Need to check more
>>> details.
>>>
>>>
>> Thanks Han. Yes, postponing clearing of OpenFlow flows until all of the
>> logical flows have been translated to OpenFlows will reduce the connection
>> downtime. The question though is that can we use 'replace-flows' or
>> 'mod-flows equivalent where-in the non-modified flows remain intact and all
>> the sessions related to those flows will not face any downtime?
>>
>> I am not sure about the "replace-flows". However, I think these are
> independent optimizations. I think postponing the clearing would solve the
> major part of the problem. I believe currently > 90% of the time is spent
> on waiting for computing to finish while the OVS flows are already cleared,
> instead of on the one time flow installation. But yes, that could be a
> further optimization.
>

Agree.

>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Winson Wang]

Hi Han,


On Wed, Aug 5, 2020 at 3:05 PM Han Zhou  wrote:

>
>
> On Wed, Aug 5, 2020 at 12:51 PM Winson Wang 
> wrote:
>
>> Hello OVN Experts:
>>
>> With large scale ovn-k8s cluster,  there are several conditions that
>> would make ovn-controller clients connect SB central from a balanced state
>> to an unbalanced state.
>> Is there an ongoing project to address this problem?
>> If not,  I have one proposal not sure if it is doable.
>> Please share your thoughts.
>>
>> The issue:
>>
>> OVN SB RAFT 3 node cluster,  at first all the ovn-controller clients will
>> connect all the 3 nodes in a balanced state.
>>
>> The following conditions will make the connections become unbalanced.
>>
>>-
>>
>>One RAFT node restart,  all the ovn-controller clients to reconnect
>>to the two remaining cluster nodes.
>>
>>
>>-
>>
>>Ovn-k8s,  after SB raft pods rolling upgrade, the last raft pod has
>>no client connections.
>>
>>
>> RAFT clients in an unbalanced state would trigger more stress to the raft
>> cluster,  which makes the raft unstable under stress compared to a balanced
>> state.
>> The proposal solution:
>>
>> Ovn-controller adds next unix commands “reconnect” with argument of
>> preferred SB node IP.
>>
>> When unbalanced state happens,  the UNIX command can trigger
>> ovn-controller reconnect
>>
>> To new SB raft node with fast sync which doesn’t trigger the whole DB
>> downloading process.
>>
>>
> Thanks Winson. The proposal sounds good to me. Will you implement it?
>

Thanks for reviewing my proposal solution.
I am hoping someone from the OVN team who more familiar with the
ovn-controller code to deliver the feature if possible:).

Regards,
Winson


> Han
>
>
>
>>
>> --
>> Winson
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ovn-kubernetes" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to ovn-kubernetes+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com
>> 
>> .
>>
>

-- 
Winson
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Han Zhou]

On Wed, Aug 5, 2020 at 4:21 PM Girish Moodalbail 
wrote:

>
>
> On Wed, Aug 5, 2020 at 3:35 PM Han Zhou  wrote:
>
>>
>>
>> On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
>> wrote:
>>
>>> Hello OVN Experts,
>>>
>>> With ovn-k8s,  we need to keep the flows always on br-int which needed
>>> by running pods on the k8s node.
>>> Is there an ongoing project to address this problem?
>>> If not,  I have one proposal not sure if it is doable.
>>> Please share your thoughts.
>>> The issue:
>>>
>>> In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
>>> every K8s node.  When we restart ovn-controller for upgrade using
>>> `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
>>> works fine since br-int with flows still be Installed.
>>>
>>> However, when a new ovn-controller starts it will connect OVS IDL and do
>>> an engine init run,  clearing all OpenFlow flows and install flows based on
>>> SB DB.
>>>
>>> With open flows count above 200K+,  it took more than 15 seconds to get
>>> all the flows installed br-int bridge again.
>>>
>>> Proposal solution for the issue:
>>>
>>> When the ovn-controller gets “exit --start”,  it will write a
>>> “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
>>> external-ids column. When new ovn-controller starts, it will check if the
>>> “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
>>> OVS IDL to decide if it will force a recomputing process?
>>>
>>>
>> Hi Winson,
>>
>> Thanks for the proposal. Yes, the connection break during upgrading is a
>> real issue in a large scale environment. However, the proposal doesn't
>> work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
>> which is a completely different connection from the ovs-vswitchd open-flow
>> connection.
>> To avoid clearing the open-flow table during ovn-controller startup, we
>> can find a way to postpone clearing the OVS flows after the recomputing in
>> ovn-controller is completed, right before ovn-controller replacing with the
>> new flows. This should largely reduce the time of connection broken during
>> upgrading. Some changes in the ofctrl module's state machine are required,
>> but I am not 100% sure if this approach is applicable. Need to check more
>> details.
>>
>>
> Thanks Han. Yes, postponing clearing of OpenFlow flows until all of the
> logical flows have been translated to OpenFlows will reduce the connection
> downtime. The question though is that can we use 'replace-flows' or
> 'mod-flows equivalent where-in the non-modified flows remain intact and all
> the sessions related to those flows will not face any downtime?
>
> I am not sure about the "replace-flows". However, I think these are
independent optimizations. I think postponing the clearing would solve the
major part of the problem. I believe currently > 90% of the time is spent
on waiting for computing to finish while the OVS flows are already cleared,
instead of on the one time flow installation. But yes, that could be a
further optimization.


> Regards,
> ~Girish
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Han Zhou]

On Wed, Aug 5, 2020 at 3:59 PM Tony Liu  wrote:

> Sorry for hijacking this thread, I'd like to get some clarifications.
>
> How is the initial balanced state established, say 100 ovn-controllers
> connecting to 3 ovn-sb-db?
>
> The ovn-controller by default randomly connects to any servers specified
in the connection method, e.g. tcp::6642,tcp:6642,tcp::6643.
(Please see ovsdb(7) for details on "Connection Method".)

So initially it is balanced.


> The ovn-controller doesn't have to connect to the leader of ovn-sb-db,
> does it? In case it connects to the follower, the write request still
> needs to be forwarded to the leader, right?
>
> These logs keep showing up.
> 
> 2020-08-05T22:48:33.141Z|103607|reconnect|INFO|tcp:10.6.20.84:6642:
> connecting...
> 2020-08-05T22:48:33.151Z|103608|reconnect|INFO|tcp:127.0.0.1:6640:
> connected
> 2020-08-05T22:48:33.151Z|103609|reconnect|INFO|tcp:10.6.20.84:6642:
> connected
> 2020-08-05T22:48:33.159Z|103610|main|INFO|OVNSB commit failed, force
> recompute next time.
> 2020-08-05T22:48:33.161Z|103611|ovsdb_idl|INFO|tcp:10.6.20.84:6642:
> clustered database server is disconnected from cluster; trying another
> server
> 2020-08-05T22:48:33.161Z|103612|reconnect|INFO|tcp:10.6.20.84:6642:
> connection attempt timed out
> 2020-08-05T22:48:33.161Z|103613|reconnect|INFO|tcp:10.6.20.84:6642:
> waiting 2 seconds before reconnect
> 
> What's that "clustered database server is disconnected from cluster" mean?
>
> It means the server is part of a cluster, but it is disconnected from the
cluster, e.g. due to network partitioning, or overloaded and lost
heartbeat, or the cluster lost quorum and there is no leader elected.
If you use a clustered DB, it's better to set the connect method to all
servers (or you can use a LB VIP that points to all servers), instead of
only specifying a single server, which doesn't provide desired HA.


>
> Thanks!
>
> Tony
>
>
> > -Original Message-
> > From: discuss  On Behalf Of Han
> > Zhou
> > Sent: Wednesday, August 5, 2020 3:05 PM
> > To: Winson Wang 
> > Cc: winson wang ; ovn-kuberne...@googlegroups.com;
> > ovs-discuss@openvswitch.org
> > Subject: Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster
> > clients to balanced state again
> >
> >
> >
> > On Wed, Aug 5, 2020 at 12:51 PM Winson Wang  >  > wrote:
> >
> >
> >   Hello OVN Experts:
> >
> >   With large scale ovn-k8s cluster,  there are several conditions
> > that would make ovn-controller clients connect SB central from a
> > balanced state to an unbalanced state.
> >
> >   Is there an ongoing project to address this problem?
> >   If not,  I have one proposal not sure if it is doable.
> >   Please share your thoughts.
> >
> >   The issue:
> >
> >   OVN SB RAFT 3 node cluster,  at first all the ovn-controller
> > clients will connect all the 3 nodes in a balanced state.
> >
> >   The following conditions will make the connections become
> > unbalanced.
> >
> >   *   One RAFT node restart,  all the ovn-controller clients to
> > reconnect to the two remaining cluster nodes.
> >
> >   *   Ovn-k8s,  after SB raft pods rolling upgrade, the last raft
> > pod has no client connections.
> >
> >
> >   RAFT clients in an unbalanced state would trigger more stress to
> > the raft cluster,  which makes the raft unstable under stress compared
> > to a balanced state.
> >
> >
> >   The proposal solution:
> >
> >
> >
> >   Ovn-controller adds next unix commands “reconnect” with argument of
> > preferred SB node IP.
> >
> >   When unbalanced state happens,  the UNIX command can trigger ovn-
> > controller reconnect
> >
> >   To new SB raft node with fast sync which doesn’t trigger the whole
> > DB downloading process.
> >
> >
> >
> > Thanks Winson. The proposal sounds good to me. Will you implement it?
> >
> > Han
> >
> >
> >
> >
> >
> >   --
> >
> >   Winson
> >
> >
> >
> >   --
> >   You received this message because you are subscribed to the Google
> > Groups "ovn-kubernetes" group.
> >   To unsubscribe from this group and stop receiving emails from it,
> > send an email to ovn-kubernetes+unsubscr...@googlegroups.com
> >  .
> >   To view this discussion on the web visit
> > https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--
> > iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com
> >  > iOW0LxxtkOhJpRT49E-
> > 9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com?utm_medium=email_source=foo
> > ter> .
> >
>
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Han Zhou]

On Wed, Aug 5, 2020 at 4:35 PM Girish Moodalbail 
wrote:

>
>
> On Wed, Aug 5, 2020 at 3:05 PM Han Zhou  wrote:
>
>>
>>
>> On Wed, Aug 5, 2020 at 12:51 PM Winson Wang 
>> wrote:
>>
>>> Hello OVN Experts:
>>>
>>> With large scale ovn-k8s cluster,  there are several conditions that
>>> would make ovn-controller clients connect SB central from a balanced state
>>> to an unbalanced state.
>>> Is there an ongoing project to address this problem?
>>> If not,  I have one proposal not sure if it is doable.
>>> Please share your thoughts.
>>>
>>> The issue:
>>>
>>> OVN SB RAFT 3 node cluster,  at first all the ovn-controller clients
>>> will connect all the 3 nodes in a balanced state.
>>>
>>> The following conditions will make the connections become unbalanced.
>>>
>>>-
>>>
>>>One RAFT node restart,  all the ovn-controller clients to reconnect
>>>to the two remaining cluster nodes.
>>>
>>>
>>>-
>>>
>>>Ovn-k8s,  after SB raft pods rolling upgrade, the last raft pod has
>>>no client connections.
>>>
>>>
>>> RAFT clients in an unbalanced state would trigger more stress to the
>>> raft cluster,  which makes the raft unstable under stress compared to a
>>> balanced state.
>>> The proposal solution:
>>>
>>> Ovn-controller adds next unix commands “reconnect” with argument of
>>> preferred SB node IP.
>>>
>>> When unbalanced state happens,  the UNIX command can trigger
>>> ovn-controller reconnect
>>>
>>> To new SB raft node with fast sync which doesn’t trigger the whole DB
>>> downloading process.
>>>
>>>
>> Thanks Winson. The proposal sounds good to me. Will you implement it?
>>
>
> Han/Winson,
>
> The fast re-sync is for ovsdb-server restart and it will not apply for
> ovn-controller restart, right?
>
>
Right, but the proposal is to provide a command just to reconnect, without
restarting. In that case fast-resync should work.


> If the ovsdb-client (ovn-controller) restarts, then it would have lost all
> its state and when it starts again it will still need to download
> logical_flows, port_bindings , and other tables it cares about. So, fast
> re-sync may not apply to this case.
>
> Also, the ovn-controller should stash the IP address of the SB server to
> which it is connected to in Open_vSwitch table's external_id column. It
> updates this field whenever it re-connects to a different SB server
> (because that ovsdb-server instance failed or restarted). When
> ovn-controller itself restarts it could check for the value in this field
> and try to connect to it first and on failure fallback to connect to
> default connection approach.
>

The imbalance is usually caused by failover on server side. When one server
is down, all clients are expected to connect to the rest of the servers,
and when the server is back, there is no motivation for the clients to
reconnect again (unless you purposely restart the clients, which would
bring 1/3 of the restarted clients back to the old server). So I don't
understand how "stash the IP address" would work in this scenario.

The proposal above by Winson is to purposely trigger a reconnection towards
the desired server without restarting the clients, which I think solves
this problem directly.

Thanks,
Han


>
> Regards,
> ~Girish
>
>
>
>
>>
>> Han
>>
>>
>>
>>>
>>> --
>>> Winson
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "ovn-kubernetes" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to ovn-kubernetes+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com
>>> 
>>> .
>>>
>> ___
>> discuss mailing list
>> disc...@openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>
> --
> You received this message because you are subscribed to the Google Groups
> "ovn-kubernetes" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ovn-kubernetes+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTrZb%2BNo8%2B3%3DOJcMqd6T_1sS5bm-xnF6v_P4%2B2uqKtZAQ%40mail.gmail.com
> 
> .
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Girish Moodalbail]

On Wed, Aug 5, 2020 at 3:05 PM Han Zhou  wrote:

>
>
> On Wed, Aug 5, 2020 at 12:51 PM Winson Wang 
> wrote:
>
>> Hello OVN Experts:
>>
>> With large scale ovn-k8s cluster,  there are several conditions that
>> would make ovn-controller clients connect SB central from a balanced state
>> to an unbalanced state.
>> Is there an ongoing project to address this problem?
>> If not,  I have one proposal not sure if it is doable.
>> Please share your thoughts.
>>
>> The issue:
>>
>> OVN SB RAFT 3 node cluster,  at first all the ovn-controller clients will
>> connect all the 3 nodes in a balanced state.
>>
>> The following conditions will make the connections become unbalanced.
>>
>>-
>>
>>One RAFT node restart,  all the ovn-controller clients to reconnect
>>to the two remaining cluster nodes.
>>
>>
>>-
>>
>>Ovn-k8s,  after SB raft pods rolling upgrade, the last raft pod has
>>no client connections.
>>
>>
>> RAFT clients in an unbalanced state would trigger more stress to the raft
>> cluster,  which makes the raft unstable under stress compared to a balanced
>> state.
>> The proposal solution:
>>
>> Ovn-controller adds next unix commands “reconnect” with argument of
>> preferred SB node IP.
>>
>> When unbalanced state happens,  the UNIX command can trigger
>> ovn-controller reconnect
>>
>> To new SB raft node with fast sync which doesn’t trigger the whole DB
>> downloading process.
>>
>>
> Thanks Winson. The proposal sounds good to me. Will you implement it?
>

Han/Winson,

The fast re-sync is for ovsdb-server restart and it will not apply for
ovn-controller restart, right?

If the ovsdb-client (ovn-controller) restarts, then it would have lost all
its state and when it starts again it will still need to download
logical_flows, port_bindings , and other tables it cares about. So, fast
re-sync may not apply to this case.

Also, the ovn-controller should stash the IP address of the SB server to
which it is connected to in Open_vSwitch table's external_id column. It
updates this field whenever it re-connects to a different SB server
(because that ovsdb-server instance failed or restarted). When
ovn-controller itself restarts it could check for the value in this field
and try to connect to it first and on failure fallback to connect to
default connection approach.

Regards,
~Girish




>
> Han
>
>
>
>>
>> --
>> Winson
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ovn-kubernetes" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to ovn-kubernetes+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com
>> 
>> .
>>
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [OVN] no response to inactivity probe

[Tony Liu]

Hi Han,

After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db,
I see this error.

2020-08-05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0: 
listen failed: Address already in use

Anything I am missing here?


Thanks!

Tony
> -Original Message-
> From: Han Zhou 
> Sent: Tuesday, August 4, 2020 4:44 PM
> To: Tony Liu 
> Cc: Numan Siddique ; Han Zhou ; ovs-
> discuss ; ovs-dev 
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 
> 
> 
> On Tue, Aug 4, 2020 at 2:50 PM Tony Liu   > wrote:
> 
> 
>   Hi,
> 
>   Since I have 3 OVN DB nodes, should I add 3 rows in connection
> table
>   for the inactivity_probe? Or put 3 addresses into one row?
> 
>   "set-connection" set one row only, and there is no "add-connection".
>   How should I add 3 rows into the table connection?
> 
> 
> 
> 
> You only need to set one row. Try this command:
> 
> ovn-nbctl -- --id=@conn_uuid create Connection
> target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global .
> connections=@conn_uuid
> 
> 
> 
>   Thanks!
> 
>   Tony
> 
>   > -Original Message-
>   > From: Numan Siddique mailto:num...@ovn.org> >
>   > Sent: Tuesday, August 4, 2020 12:36 AM
>   > To: Tony Liu   >
>   > Cc: ovs-discuss mailto:ovs-
> disc...@openvswitch.org> >; ovs-dev> d...@openvswitch.org  >
>   > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
>   >
>   >
>   >
>   > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu  
>   >   > > wrote:
>   >
>   >
>   >   In my deployment, on each Neutron server, there are 13
> Neutron
>   > server processes.
>   >   I see 12 of them (monitor, maintenance, RPC, API) connect
> to both
>   > ovn-nb-db
>   >   and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB
> clients.
>   >   Is so many clients OK?
>   >
>   >   Any suggestions how to figure out which side doesn't
> respond the
>   > probe,
>   >   if it's bi-directional? I don't see any activities from
> logging,
>   > other than
>   >   connect/drop and reconnect...
>   >
>   >   BTW, please let me know if this is not the right place to
> discuss
>   > Neutron OVN
>   >   ML2 driver.
>   >
>   >
>   >   Thanks!
>   >
>   >   Tony
>   >
>   >   > -Original Message-
>   >   > From: dev mailto:ovs-
> dev-boun...@openvswitch.org>  
>   > boun...@openvswitch.org  > > On
> Behalf Of Tony Liu
>   >   > Sent: Monday, August 3, 2020 7:45 PM
>   >   > To: ovs-discuss mailto:ovs-
> disc...@openvswitch.org>  
>   > disc...@openvswitch.org  > >;
> ovs-dev>   > d...@openvswitch.org 
>  > >
>   >   > Subject: [ovs-dev] [OVN] no response to inactivity probe
>   >   >
>   >   > Hi,
>   >   >
>   >   > Neutron OVN ML2 driver was disconnected by ovn-nb-db.
> There are
>   > many
>   >   > error messages from ovn-nb-db leader.
>   >   > 
>   >   > 2020-08-
> 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> 
>   >  : no
>   >   > response to inactivity probe after 5 seconds,
> disconnecting
>   >   > 2020-08-
> 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> 
>   >  : no
>   >   > response to inactivity probe after 5 seconds,
> disconnecting
>   >   > 2020-08-
> 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> 
>   >  : no
>   >   > response to inactivity probe after 5 seconds,
> disconnecting
>   >   > 2020-08-
> 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> 
>   >  : no
>   >   > response to inactivity probe after 5 seconds,
> disconnecting
>   >   > 2020-08-
> 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> 
>   >  : no
>   >   > response to inactivity probe after 5 seconds,
> disconnecting
>   >   > 2020-08-
> 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> 
>   >  : no
>   >   > response to inactivity probe after 5 seconds,
> disconnecting
>   >   > 2020-08-
> 

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Girish Moodalbail]

On Wed, Aug 5, 2020 at 3:35 PM Han Zhou  wrote:

>
>
> On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
> wrote:
>
>> Hello OVN Experts,
>>
>> With ovn-k8s,  we need to keep the flows always on br-int which needed by
>> running pods on the k8s node.
>> Is there an ongoing project to address this problem?
>> If not,  I have one proposal not sure if it is doable.
>> Please share your thoughts.
>> The issue:
>>
>> In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
>> every K8s node.  When we restart ovn-controller for upgrade using
>> `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
>> works fine since br-int with flows still be Installed.
>>
>> However, when a new ovn-controller starts it will connect OVS IDL and do
>> an engine init run,  clearing all OpenFlow flows and install flows based on
>> SB DB.
>>
>> With open flows count above 200K+,  it took more than 15 seconds to get
>> all the flows installed br-int bridge again.
>>
>> Proposal solution for the issue:
>>
>> When the ovn-controller gets “exit --start”,  it will write a
>> “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
>> external-ids column. When new ovn-controller starts, it will check if the
>> “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
>> OVS IDL to decide if it will force a recomputing process?
>>
>>
> Hi Winson,
>
> Thanks for the proposal. Yes, the connection break during upgrading is a
> real issue in a large scale environment. However, the proposal doesn't
> work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
> which is a completely different connection from the ovs-vswitchd open-flow
> connection.
> To avoid clearing the open-flow table during ovn-controller startup, we
> can find a way to postpone clearing the OVS flows after the recomputing in
> ovn-controller is completed, right before ovn-controller replacing with the
> new flows. This should largely reduce the time of connection broken during
> upgrading. Some changes in the ofctrl module's state machine are required,
> but I am not 100% sure if this approach is applicable. Need to check more
> details.
>
>
Thanks Han. Yes, postponing clearing of OpenFlow flows until all of the
logical flows have been translated to OpenFlows will reduce the connection
downtime. The question though is that can we use 'replace-flows' or
'mod-flows equivalent where-in the non-modified flows remain intact and all
the sessions related to those flows will not face any downtime?

Regards,
~Girish
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Tony Liu]

Sorry for hijacking this thread, I'd like to get some clarifications.

How is the initial balanced state established, say 100 ovn-controllers
connecting to 3 ovn-sb-db?

The ovn-controller doesn't have to connect to the leader of ovn-sb-db,
does it? In case it connects to the follower, the write request still
needs to be forwarded to the leader, right?

These logs keep showing up.

2020-08-05T22:48:33.141Z|103607|reconnect|INFO|tcp:10.6.20.84:6642: 
connecting...
2020-08-05T22:48:33.151Z|103608|reconnect|INFO|tcp:127.0.0.1:6640: connected
2020-08-05T22:48:33.151Z|103609|reconnect|INFO|tcp:10.6.20.84:6642: connected
2020-08-05T22:48:33.159Z|103610|main|INFO|OVNSB commit failed, force recompute 
next time.
2020-08-05T22:48:33.161Z|103611|ovsdb_idl|INFO|tcp:10.6.20.84:6642: clustered 
database server is disconnected from cluster; trying another server
2020-08-05T22:48:33.161Z|103612|reconnect|INFO|tcp:10.6.20.84:6642: connection 
attempt timed out
2020-08-05T22:48:33.161Z|103613|reconnect|INFO|tcp:10.6.20.84:6642: waiting 2 
seconds before reconnect

What's that "clustered database server is disconnected from cluster" mean?


Thanks!

Tony


> -Original Message-
> From: discuss  On Behalf Of Han
> Zhou
> Sent: Wednesday, August 5, 2020 3:05 PM
> To: Winson Wang 
> Cc: winson wang ; ovn-kuberne...@googlegroups.com;
> ovs-discuss@openvswitch.org
> Subject: Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster
> clients to balanced state again
> 
> 
> 
> On Wed, Aug 5, 2020 at 12:51 PM Winson Wang   > wrote:
> 
> 
>   Hello OVN Experts:
> 
>   With large scale ovn-k8s cluster,  there are several conditions
> that would make ovn-controller clients connect SB central from a
> balanced state to an unbalanced state.
> 
>   Is there an ongoing project to address this problem?
>   If not,  I have one proposal not sure if it is doable.
>   Please share your thoughts.
> 
>   The issue:
> 
>   OVN SB RAFT 3 node cluster,  at first all the ovn-controller
> clients will connect all the 3 nodes in a balanced state.
> 
>   The following conditions will make the connections become
> unbalanced.
> 
>   *   One RAFT node restart,  all the ovn-controller clients to
> reconnect to the two remaining cluster nodes.
> 
>   *   Ovn-k8s,  after SB raft pods rolling upgrade, the last raft
> pod has no client connections.
> 
> 
>   RAFT clients in an unbalanced state would trigger more stress to
> the raft cluster,  which makes the raft unstable under stress compared
> to a balanced state.
> 
> 
>   The proposal solution:
> 
> 
> 
>   Ovn-controller adds next unix commands “reconnect” with argument of
> preferred SB node IP.
> 
>   When unbalanced state happens,  the UNIX command can trigger ovn-
> controller reconnect
> 
>   To new SB raft node with fast sync which doesn’t trigger the whole
> DB downloading process.
> 
> 
> 
> Thanks Winson. The proposal sounds good to me. Will you implement it?
> 
> Han
> 
> 
> 
> 
> 
>   --
> 
>   Winson
> 
> 
> 
>   --
>   You received this message because you are subscribed to the Google
> Groups "ovn-kubernetes" group.
>   To unsubscribe from this group and stop receiving emails from it,
> send an email to ovn-kubernetes+unsubscr...@googlegroups.com
>  .
>   To view this discussion on the web visit
> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--
> iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com
>  iOW0LxxtkOhJpRT49E-
> 9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com?utm_medium=email_source=foo
> ter> .
> 

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Han Zhou]

On Wed, Aug 5, 2020 at 12:58 PM Winson Wang  wrote:

> Hello OVN Experts,
>
> With ovn-k8s,  we need to keep the flows always on br-int which needed by
> running pods on the k8s node.
> Is there an ongoing project to address this problem?
> If not,  I have one proposal not sure if it is doable.
> Please share your thoughts.
> The issue:
>
> In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
> every K8s node.  When we restart ovn-controller for upgrade using
> `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
> works fine since br-int with flows still be Installed.
>
> However, when a new ovn-controller starts it will connect OVS IDL and do
> an engine init run,  clearing all OpenFlow flows and install flows based on
> SB DB.
>
> With open flows count above 200K+,  it took more than 15 seconds to get
> all the flows installed br-int bridge again.
>
> Proposal solution for the issue:
>
> When the ovn-controller gets “exit --start”,  it will write a
> “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
> external-ids column. When new ovn-controller starts, it will check if the
> “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
> OVS IDL to decide if it will force a recomputing process?
>
>
Hi Winson,

Thanks for the proposal. Yes, the connection break during upgrading is a
real issue in a large scale environment. However, the proposal doesn't
work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
which is a completely different connection from the ovs-vswitchd open-flow
connection.
To avoid clearing the open-flow table during ovn-controller startup, we can
find a way to postpone clearing the OVS flows after the recomputing in
ovn-controller is completed, right before ovn-controller replacing with the
new flows. This should largely reduce the time of connection broken during
upgrading. Some changes in the ofctrl module's state machine are required,
but I am not 100% sure if this approach is applicable. Need to check more
details.

Thanks,
Han
Test log:

> Check flow cnt on br-int every second:
>
> packet_count=0 byte_count=0 flow_count=0
>
> packet_count=0 byte_count=0 flow_count=0
>
> packet_count=0 byte_count=0 flow_count=0
>
> packet_count=0 byte_count=0 flow_count=0
>
> packet_count=0 byte_count=0 flow_count=0
>
> packet_count=0 byte_count=0 flow_count=0
>
> packet_count=0 byte_count=0 flow_count=10322
>
> packet_count=0 byte_count=0 flow_count=34220
>
> packet_count=0 byte_count=0 flow_count=60425
>
> packet_count=0 byte_count=0 flow_count=82506
>
> packet_count=0 byte_count=0 flow_count=106771
>
> packet_count=0 byte_count=0 flow_count=131648
>
> packet_count=2 byte_count=120 flow_count=158303
>
> packet_count=29 byte_count=1693 flow_count=185999
>
> packet_count=188 byte_count=12455 flow_count=212764
>
>
>
> --
> Winson
>
> --
> You received this message because you are subscribed to the Google Groups
> "ovn-kubernetes" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ovn-kubernetes+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS8eC2EtMJbqBccGD0hyvLFBkzkeJ9sXOsT_TVF3Ltm2hA%40mail.gmail.com
> 
> .
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] packet drop

[Tony Liu]

The drop is caused by flow change.

When packet is dropped.

recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,geneve({class=0x102,type=0x80,len=4,0x20003/0x7fff}),flags(-df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type=8/0xf8),
 packets:14, bytes:1372, used:0.846s, actions:drop
recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df:1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no),icmp(type=0),
 packets:6, bytes:588, used:8.983s, actions:drop


When packet goes through.

recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,geneve({class=0x102,type=0x80,len=4,0x20003/0x7fff}),flags(-df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type=8/0xf8),
 packets:3, bytes:294, used:0.104s, actions:12
recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df:1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no),icmp(type=0),
 packets:3, bytes:294, used:0.103s, 
actions:ct_clear,set(tunnel(tun_id=0x1a8ee,dst=10.6.30.92,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x1000b}),flags(df|csum|key))),set(eth(src=fa:16:3e:75:b7:e5,dst=52:54:00:0c:ef:b9)),set(ipv4(ttl=63)),3


Is that flow programmed by ovn-controller via ovs-vswitchd?


Thanks!

Tony

> -Original Message-
> From: discuss  On Behalf Of Tony
> Liu
> Sent: Wednesday, August 5, 2020 2:48 PM
> To: ovs-discuss@openvswitch.org; ovs-...@openvswitch.org
> Subject: [ovs-discuss] packet drop
> 
> Hi,
> 
> I am running ping from external to VM via OVN gateway.
> On the compute node, ICMP request packet is consistently coming into
> interface "ovn-gatewa-1". But there is about 10 out of 25 packet loss on
> tap interface. It's like the switch pauses 10s after every 15s.
> 
> Has anyone experiences such issue?
> Any advice how to look into it?
> 
> 
> 21fed09f-909e-4efc-b117-f5d5fcb636c9
> Bridge br-int
> fail_mode: secure
> datapath_type: system
> Port "ovn-gatewa-0"
> Interface "ovn-gatewa-0"
> type: geneve
> options: {csum="true", key=flow, remote_ip="10.6.30.91"}
> bfd_status: {diagnostic="No Diagnostic", flap_count="1",
> forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up,
> state=up}
> Port "tap2588bb4e-35"
> Interface "tap2588bb4e-35"
> Port "ovn-gatewa-1"
> Interface "ovn-gatewa-1"
> type: geneve
> options: {csum="true", key=flow, remote_ip="10.6.30.92"}
> bfd_status: {diagnostic="No Diagnostic", flap_count="1",
> forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up,
> state=up}
> Port "tap37f6b2d7-cc"
> Interface "tap37f6b2d7-cc"
> Port "tap2c4b3b0f-8b"
> Interface "tap2c4b3b0f-8b"
> Port "tap23245491-a4"
> Interface "tap23245491-a4"
> Port "tap51660269-2c"
> Interface "tap51660269-2c"
> Port "tap276cd1ef-e1"
> Interface "tap276cd1ef-e1"
> Port "tap138526d3-b3"
> Interface "tap138526d3-b3"
> Port "tapd1ae48a1-2d"
> Interface "tapd1ae48a1-2d"
> Port br-int
> Interface br-int
> type: internal
> Port "tapdd08f476-94"
> Interface "tapdd08f476-94"
> 
> 
> 
> Thanks!
> 
> Tony
> 
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Han Zhou]

On Wed, Aug 5, 2020 at 12:51 PM Winson Wang  wrote:

> Hello OVN Experts:
>
> With large scale ovn-k8s cluster,  there are several conditions that would
> make ovn-controller clients connect SB central from a balanced state to
> an unbalanced state.
> Is there an ongoing project to address this problem?
> If not,  I have one proposal not sure if it is doable.
> Please share your thoughts.
>
> The issue:
>
> OVN SB RAFT 3 node cluster,  at first all the ovn-controller clients will
> connect all the 3 nodes in a balanced state.
>
> The following conditions will make the connections become unbalanced.
>
>-
>
>One RAFT node restart,  all the ovn-controller clients to reconnect to
>the two remaining cluster nodes.
>
>
>-
>
>Ovn-k8s,  after SB raft pods rolling upgrade, the last raft pod has no
>client connections.
>
>
> RAFT clients in an unbalanced state would trigger more stress to the raft
> cluster,  which makes the raft unstable under stress compared to a balanced
> state.
> The proposal solution:
>
> Ovn-controller adds next unix commands “reconnect” with argument of
> preferred SB node IP.
>
> When unbalanced state happens,  the UNIX command can trigger
> ovn-controller reconnect
>
> To new SB raft node with fast sync which doesn’t trigger the whole DB
> downloading process.
>
>
Thanks Winson. The proposal sounds good to me. Will you implement it?

Han



>
> --
> Winson
>
> --
> You received this message because you are subscribed to the Google Groups
> "ovn-kubernetes" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ovn-kubernetes+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com
> 
> .
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] packet drop

[Tony Liu]

Hi,

I am running ping from external to VM via OVN gateway.
On the compute node, ICMP request packet is consistently coming
into interface "ovn-gatewa-1". But there is about 10 out of 25
packet loss on tap interface. It's like the switch pauses 10s
after every 15s.

Has anyone experiences such issue?
Any advice how to look into it?


21fed09f-909e-4efc-b117-f5d5fcb636c9
Bridge br-int
fail_mode: secure
datapath_type: system
Port "ovn-gatewa-0"
Interface "ovn-gatewa-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.6.30.91"}
bfd_status: {diagnostic="No Diagnostic", flap_count="1", 
forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
Port "tap2588bb4e-35"
Interface "tap2588bb4e-35"
Port "ovn-gatewa-1"
Interface "ovn-gatewa-1"
type: geneve
options: {csum="true", key=flow, remote_ip="10.6.30.92"}
bfd_status: {diagnostic="No Diagnostic", flap_count="1", 
forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
Port "tap37f6b2d7-cc"
Interface "tap37f6b2d7-cc"
Port "tap2c4b3b0f-8b"
Interface "tap2c4b3b0f-8b"
Port "tap23245491-a4"
Interface "tap23245491-a4"
Port "tap51660269-2c"
Interface "tap51660269-2c"
Port "tap276cd1ef-e1"
Interface "tap276cd1ef-e1"
Port "tap138526d3-b3"
Interface "tap138526d3-b3"
Port "tapd1ae48a1-2d"
Interface "tapd1ae48a1-2d"
Port br-int
Interface br-int
type: internal
Port "tapdd08f476-94"
Interface "tapdd08f476-94"



Thanks!

Tony

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Winson Wang]

Hello OVN Experts,

With ovn-k8s,  we need to keep the flows always on br-int which needed by
running pods on the k8s node.
Is there an ongoing project to address this problem?
If not,  I have one proposal not sure if it is doable.
Please share your thoughts.
The issue:

In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
every K8s node.  When we restart ovn-controller for upgrade using
`ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
works fine since br-int with flows still be Installed.

However, when a new ovn-controller starts it will connect OVS IDL and do an
engine init run,  clearing all OpenFlow flows and install flows based on SB
DB.

With open flows count above 200K+,  it took more than 15 seconds to get all
the flows installed br-int bridge again.

Proposal solution for the issue:

When the ovn-controller gets “exit --start”,  it will write a
“ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
external-ids column. When new ovn-controller starts, it will check if the
“ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
OVS IDL to decide if it will force a recomputing process?

Test log:

Check flow cnt on br-int every second:

packet_count=0 byte_count=0 flow_count=0

packet_count=0 byte_count=0 flow_count=0

packet_count=0 byte_count=0 flow_count=0

packet_count=0 byte_count=0 flow_count=0

packet_count=0 byte_count=0 flow_count=0

packet_count=0 byte_count=0 flow_count=0

packet_count=0 byte_count=0 flow_count=10322

packet_count=0 byte_count=0 flow_count=34220

packet_count=0 byte_count=0 flow_count=60425

packet_count=0 byte_count=0 flow_count=82506

packet_count=0 byte_count=0 flow_count=106771

packet_count=0 byte_count=0 flow_count=131648

packet_count=2 byte_count=120 flow_count=158303

packet_count=29 byte_count=1693 flow_count=185999

packet_count=188 byte_count=12455 flow_count=212764



-- 
Winson
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] SB flows not being created in OVN K8 Stateful set

[Dumitru Ceara]

On 8/5/20 5:14 PM, Brendan Doyle wrote:
> Folks,
> 
> I'm stumped here, I have the k8 ovnkube-db-raft Stateful set up and
> running.
> But when I create a simple network, no SB flows are generated.
> 
> ovn-nbctl show shows my network. ovn-sbctl show shows the physicals
> systems in my network.
> But I can't ping between any hosts because ovn-sbctl lflow-list is
> empty, and there are no
> errors or warnings in the logs. The ovn cluster says it is up and healthy.
> 
> Anybody got any ideas why this might be?
> 

Hi Brendan,

Maybe I missed it but is ovn-northd running anywhere? If so, could you
please share its logs?

Thanks,
Dumitru

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

[Winson Wang]

Hello OVN Experts:

With large scale ovn-k8s cluster,  there are several conditions that would
make ovn-controller clients connect SB central from a balanced state to
an unbalanced state.
Is there an ongoing project to address this problem?
If not,  I have one proposal not sure if it is doable.
Please share your thoughts.

The issue:

OVN SB RAFT 3 node cluster,  at first all the ovn-controller clients will
connect all the 3 nodes in a balanced state.

The following conditions will make the connections become unbalanced.

   -

   One RAFT node restart,  all the ovn-controller clients to reconnect to
   the two remaining cluster nodes.


   -

   Ovn-k8s,  after SB raft pods rolling upgrade, the last raft pod has no
   client connections.


RAFT clients in an unbalanced state would trigger more stress to the raft
cluster,  which makes the raft unstable under stress compared to a balanced
state.
The proposal solution:

Ovn-controller adds next unix commands “reconnect” with argument of
preferred SB node IP.

When unbalanced state happens,  the UNIX command can trigger ovn-controller
reconnect

To new SB raft node with fast sync which doesn’t trigger the whole DB
downloading process.


-- 
Winson
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] SB flows not being created in OVN K8 Stateful set

[Brendan Doyle]

Folks,

I'm stumped here, I have the k8 ovnkube-db-raft Stateful set up and running.
But when I create a simple network, no SB flows are generated.

ovn-nbctl show shows my network. ovn-sbctl show shows the physicals 
systems in my network.
But I can't ping between any hosts because ovn-sbctl lflow-list is 
empty, and there are no

errors or warnings in the logs. The ovn cluster says it is up and healthy.

Anybody got any ideas why this might be?


ovn-nbctl 
--db="tcp:253.255.0.33:6641,tcp:253.255.0.34:6641,tcp:253.255.0.35:6641" 
show

--
switch 5eb7330c-bb58-454b-9b0f-78f33e477f9a (ls_vcn1)
    port ls_vcn1_net2-lr_vcn1_net2
    type: router
    addresses: ["40:44:00:00:00:40"]
    router-port: lr_vcn1_net2-ls_vcn1_net2
    port ls_vcn1_net1-lr_vcn1_net1
    type: router
    addresses: ["40:44:00:00:00:30"]
    router-port: lr_vcn1_net1-ls_vcn1_net1
    port 00bff7c0-2e2d-41ba-9485-3b5fa9801365
    addresses: ["52:54:00:e6:4f:46 192.16.1.5"]
    port 47433b54-ac10-42f1-ae84-cc6fbb580297
    addresses: ["52:54:00:be:06:16 192.16.1.6"]
    port 1cb7d760-90b0-4201-9517-88cb2de31c79
    addresses: ["52:54:00:80:d0:c8 192.16.2.5"]
switch c03b6ad3-be18-47a6-951a-65bd1d64d912 (ls_vcn1_backbone)
    port lsb_vcn1_net1-lr_vcn1_net1
    type: router
    router-port: lr_vcn1_net1-lsb_vcn1_net1
    port lsb_vcn1_net2-lr_vcn1_net2
    type: router
    router-port: lr_vcn1_net2-lsb_vcn1_net2
router c931466d-2768-4a4d-9528-6bb3ac530084 (lr_vcn1_net2)
    port lr_vcn1_net2-ls_vcn1_net2
    mac: "40:44:00:00:00:40"
    networks: ["192.16.2.1/24"]
    port lr_vcn1_net2-lsb_vcn1_net2
    mac: "40:44:00:00:00:60"
    networks: ["253.255.130.2/23"]
router 76b42f83-f508-43b7-b1a1-887b3b900740 (lr_vcn1_net1)
    port lr_vcn1_net1-ls_vcn1_net1
    mac: "40:44:00:00:00:30"
    networks: ["192.16.1.1/24"]
    port lr_vcn1_net1-lsb_vcn1_net1
    mac: "40:44:00:00:00:50"
    networks: ["253.255.130.1/23"]

ovn-sbctl 
--db="tcp:253.255.0.33:6642,tcp:253.255.0.34:6642,tcp:253.255.0.35:6642" 
show

---
Chassis ca-rain05
    hostname: ca-rain05.us.oracle.com
    Encap geneve
    ip: "253.255.1.1"
    options: {csum="true"}
Chassis ca-rain17
    hostname: ca-rain17.us.oracle.com
    Encap geneve
    ip: "253.255.3.1"
    options: {csum="true"}
Chassis ca-rain03
    hostname: ca-rain03.us.oracle.com
    Encap geneve
    ip: "253.255.0.35"
    options: {csum="true"}
Chassis ca-rain06
    hostname: ca-rain06.us.oracle.com
    Encap geneve
    ip: "253.255.2.1"
    options: {csum="true"}
Chassis ca-rain02
    hostname: ca-rain02.us.oracle.com
    Encap geneve
    ip: "253.255.0.34"
    options: {csum="true"}
Chassis ca-rain01
    hostname: ca-rain01.us.oracle.com
    Encap geneve
    ip: "253.255.0.33"
    options: {csum="true"}

ovn-sbctl 
--db="tcp:253.255.0.33:6642,tcp:253.255.0.34:6642,tcp:253.255.0.35:6642" 
lflow-list

--
empty

k8 stuff
--
kubectl get services -o wide --namespace=ovn-kubernetes
NAME TYPE    CLUSTER-IP   EXTERNAL-IP PORT(S) 
AGE   SELECTOR
ovnkube-db   ClusterIP   None  6641/TCP,6642/TCP   33m   



 kubectl get StatefulSet -o wide --namespace=ovn-kubernetes
NAME READY   AGE   CONTAINERS  IMAGES
ovnkube-db   3/3 33m   nb-ovsdb,sb-ovsdb 
ovn-daemonset:latest,ovn-daemonset:latest


kubectl get pods -o wide --namespace=ovn-kubernetes
NAME   READY   STATUS    RESTARTS   AGE   IP NODE    
NOMINATED NODE   READINESS GATES
ovnkube-db-0   2/2 Running   0  33m   253.255.0.33 
ca-rain01      
ovnkube-db-1   2/2 Running   0  33m   253.255.0.35 
ca-rain03      
ovnkube-db-2   2/2 Running   0  33m   253.255.0.34 
ca-rain02      


kubectl describe StatefulSet --namespace=ovn-kubernetes
Name:   ovnkube-db
Namespace:  ovn-kubernetes
CreationTimestamp:  Wed, 05 Aug 2020 10:22:50 -0400
Selector:   name=ovnkube-db
Labels: 
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{"kubernetes.io/description":"This 
statefulset launches the OVN Nor...
    kubernetes.io/description: This statefulset 
launches the OVN Northbound/Southbound Database raft clusters.

Replicas:   3 desired | 3 total
Update Strategy:    RollingUpdate
  Partition:    824644587688
Pods Status:    3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:   

[ovs-discuss] 答复: [ovs-dev] there are many error logs when processing igmp packet

[王培辉]

Thanks for your response, ben, 
I found these error logs will occur with hw-offload=true, This error will 
disappear when I disable hardware offload.
It seems there is a bug processing igmp packet when hardware offload enabled.
I'm not familiar with this code,  Could you provide any clue how to debug this 
issue? 

Thanks.

-邮件原件-
发件人: Ben Pfaff [mailto:b...@ovn.org] 
发送时间: 2020年8月5日 0:06
收件人: Frank Wang(王培辉) 
抄送: ovs-discuss@openvswitch.org; ovs-...@openvswitch.org
主题: Re: [ovs-dev] there are many error logs when processing igmp packet

On Tue, Aug 04, 2020 at 11:55:19AM +, Frank Wang(王培辉) wrote:
> Hello,
> 
> I found there are many error logs as follows in ovs-vswitchd.log:
> Aug  4 18:48:03 node-170 ovs-vswitchd:
> ovs|00100|odp_util(handler171)|ERR|internal error parsing flow key
> recirc_id(0),dp_hash(0),skb_priority(0),in_port(5),skb_mark(0),ct_stat
> e(0),c t_zone(0),ct_label(0),eth(src=6c:92:bf:14:e9:f8,
> dst=01:00:5e:00:00:16),eth_type(0x0800),ipv4(src=100.7.40.41,dst=224.0
> .0.22,
> proto=2,tos=0,ttl=1,frag=no)
> 
>It seems parse_l2_5_onward function return ODP_FIT_TOO_LITTLE when 
> it processing igmp packet after digging,

ODP_FIT_TOO_LITTLE is expected for IGMP, because the kernel doesn't parse the 
IGMP header, whereas userspace does.

>I'm wondering the reason to do this ,how to avoid this?

The log message shows that there is definitely a bug but it does not sound like 
it is in parse_l2_5_onward().


smime.p7s
Description: S/MIME cryptographic signature
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss