Re: [ovs-discuss] [ovs-dev] [OVN] stale data complained by ovn-controller after db restore

[Han Zhou]

On Tue, Aug 4, 2020 at 11:31 AM Tony Liu  wrote:

> Is there any difference to restore DB on existing cluster vs. fresh
> cluster,
> in terms of performance?
>
> If I don't have to restore on fresh cluster, which is recommended?
>
> I would suggest to directly restore on top of existing cluster instead of
creating a fresh cluster.


> For now, since ovn-northd always recomputes the whole DB, I guess not much
> difference?
>
> With incremental-process, would restoring to a fresh cluster be better?
>
> No.


> Is it necessary to stop or restart ovn-northd during DB restore?
>
> No.


>
> Thanks!
>
> Tony
>
> > -Original Message-
> > From: Han Zhou 
> > Sent: Tuesday, August 4, 2020 11:13 AM
> > To: Tony Liu 
> > Cc: ovs-discuss ; ovs-dev  > d...@openvswitch.org>
> > Subject: Re: [ovs-dev] [OVN] stale data complained by ovn-controller
> > after db restore
> >
> >
> >
> > On Tue, Aug 4, 2020 at 10:30 AM Tony Liu  >  > wrote:
> >
> >
> >   Hi,
> >
> >   Here is how I restore OVN DB.
> >   * Stop all ovn-nb-db, ovn-sb-db and ovn-northd services.
> >   * Clean up all DB files.
> >   * Start all DB services. Fresh ovn-nb-db and ovn-sb-db clusters are
> > up and
> > running.
> >   * Set DB election timer to 10s.
> >   * Restore DB to ovn-nb-db by ovsdb-client.
> >   * Start all ovn-northd services.
> >
> >   A few minutes after, ovn-sb-db is fully synced with ovn-nb-db.
> >
> >   Now, the client of ovn-sb-db, ovn-controller and nova-compute
> > complaint about
> >   "stale data". The chassis node is not getting updated.
> >   
> >   2020-08-04 09:07:45.892 26 INFO ovsdbapp.backend.ovs_idl.vlog [-]
> > tcp:10.6.20.84:6642  : connected
> >   2020-08-04 09:07:45.895 26 WARNING ovsdbapp.backend.ovs_idl.vlog
> [-]
> > tcp:10.6.20.84:6642  : clustered database server
> > has stale data; trying another server
> >   
> >
> >   Restarting ovn-controller and nova-compute resolve the issue.
> >
> >   Is this expected? As part of the DB restore process, should I
> > restart
> >   ovn-controller and nova-compute on all chassis node?
> >
> >
> >
> >
> > Yes, this is expected if you freshly start a new cluster. (It wouldn't
> > happen if you simply restore the old data on the existing cluster.
> > However, I understand that the scenario of restoring data on a freshly
> > created cluster is a valid use case).
> > For this case, you could either restart ovn-controller, or trigger a
> > client side raft index reset by:
> > ovn-appctl -t ovn-controller sb-cluster-state-reset
> >
> > Similarly for ovn-northd:
> > ovn-appctl -t ovn-northd nb-cluster-state-reset
> > ovn-appctl -t ovn-northd sb-cluster-state-reset
> >
> > To use this command, you will need at least 20.06 of OVN and OVS master.
> >
> >
> > Thanks,
> > Han
> >
> >
> >
> >
> >   Thanks!
> >
> >   Tony
> >
> >   ___
> >   dev mailing list
> >   d...@openvswitch.org 
> >   https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
>
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [ovs-dev] [OVN] stale data complained by ovn-controller after db restore

[Tony Liu]

Is there any difference to restore DB on existing cluster vs. fresh cluster,
in terms of performance?

If I don't have to restore on fresh cluster, which is recommended?

For now, since ovn-northd always recomputes the whole DB, I guess not much
difference?

With incremental-process, would restoring to a fresh cluster be better?

Is it necessary to stop or restart ovn-northd during DB restore?


Thanks!

Tony

> -Original Message-
> From: Han Zhou 
> Sent: Tuesday, August 4, 2020 11:13 AM
> To: Tony Liu 
> Cc: ovs-discuss ; ovs-dev  d...@openvswitch.org>
> Subject: Re: [ovs-dev] [OVN] stale data complained by ovn-controller
> after db restore
> 
> 
> 
> On Tue, Aug 4, 2020 at 10:30 AM Tony Liu   > wrote:
> 
> 
>   Hi,
> 
>   Here is how I restore OVN DB.
>   * Stop all ovn-nb-db, ovn-sb-db and ovn-northd services.
>   * Clean up all DB files.
>   * Start all DB services. Fresh ovn-nb-db and ovn-sb-db clusters are
> up and
> running.
>   * Set DB election timer to 10s.
>   * Restore DB to ovn-nb-db by ovsdb-client.
>   * Start all ovn-northd services.
> 
>   A few minutes after, ovn-sb-db is fully synced with ovn-nb-db.
> 
>   Now, the client of ovn-sb-db, ovn-controller and nova-compute
> complaint about
>   "stale data". The chassis node is not getting updated.
>   
>   2020-08-04 09:07:45.892 26 INFO ovsdbapp.backend.ovs_idl.vlog [-]
> tcp:10.6.20.84:6642  : connected
>   2020-08-04 09:07:45.895 26 WARNING ovsdbapp.backend.ovs_idl.vlog [-]
> tcp:10.6.20.84:6642  : clustered database server
> has stale data; trying another server
>   
> 
>   Restarting ovn-controller and nova-compute resolve the issue.
> 
>   Is this expected? As part of the DB restore process, should I
> restart
>   ovn-controller and nova-compute on all chassis node?
> 
> 
> 
> 
> Yes, this is expected if you freshly start a new cluster. (It wouldn't
> happen if you simply restore the old data on the existing cluster.
> However, I understand that the scenario of restoring data on a freshly
> created cluster is a valid use case).
> For this case, you could either restart ovn-controller, or trigger a
> client side raft index reset by:
> ovn-appctl -t ovn-controller sb-cluster-state-reset
> 
> Similarly for ovn-northd:
> ovn-appctl -t ovn-northd nb-cluster-state-reset
> ovn-appctl -t ovn-northd sb-cluster-state-reset
> 
> To use this command, you will need at least 20.06 of OVN and OVS master.
> 
> 
> Thanks,
> Han
> 
> 
> 
> 
>   Thanks!
> 
>   Tony
> 
>   ___
>   dev mailing list
>   d...@openvswitch.org 
>   https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [ovs-dev] [OVN] stale data complained by ovn-controller after db restore

[Han Zhou]

On Tue, Aug 4, 2020 at 10:30 AM Tony Liu  wrote:

> Hi,
>
> Here is how I restore OVN DB.
> * Stop all ovn-nb-db, ovn-sb-db and ovn-northd services.
> * Clean up all DB files.
> * Start all DB services. Fresh ovn-nb-db and ovn-sb-db clusters are up and
>   running.
> * Set DB election timer to 10s.
> * Restore DB to ovn-nb-db by ovsdb-client.
> * Start all ovn-northd services.
>
> A few minutes after, ovn-sb-db is fully synced with ovn-nb-db.
>
> Now, the client of ovn-sb-db, ovn-controller and nova-compute complaint
> about
> "stale data". The chassis node is not getting updated.
> 
> 2020-08-04 09:07:45.892 26 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:
> 10.6.20.84:6642: connected
> 2020-08-04 09:07:45.895 26 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:
> 10.6.20.84:6642: clustered database server has stale data; trying another
> server
> 
>
> Restarting ovn-controller and nova-compute resolve the issue.
>
> Is this expected? As part of the DB restore process, should I restart
> ovn-controller and nova-compute on all chassis node?
>
>
Yes, this is expected if you freshly start a new cluster. (It wouldn't
happen if you simply restore the old data on the existing cluster. However,
I understand that the scenario of restoring data on a freshly created
cluster is a valid use case).
For this case, you could either restart ovn-controller, or trigger a client
side raft index reset by:
ovn-appctl -t ovn-controller sb-cluster-state-reset

Similarly for ovn-northd:
ovn-appctl -t ovn-northd nb-cluster-state-reset
ovn-appctl -t ovn-northd sb-cluster-state-reset

To use this command, you will need at least 20.06 of OVN and OVS master.

Thanks,
Han



> Thanks!
>
> Tony
>
> ___
> dev mailing list
> d...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss