On Fri, Apr 8, 2022 at 4:55 PM Frode Nordahl <frode.nord...@canonical.com> wrote: > > > > fre. 8. apr. 2022, 14:41 skrev Christian Stelter <refu...@last-refuge.net>: >> >> Hi! >> >> We are experiencing high latencies from time to time in one of our >> openstack environments. While looking into the issue we found that >> the southdb has frequent leader changes. >> >> We are running the Xena release and ovsdb-server (Open vSwitch) 2.16.0 >> deployed via kolla-ansible. >> >> >> We also noticed that the leader changes are caused by snapshots >> (database compaction?). > > > Since this commit [0] introduced in OVS 2.16, transfer of leadership before > snapshot is the intended behaviour. Before the change the leader would stop > processing requests until the snapshot completed. > >> Do these changes/snapshots have an impact on the performance of the southdb? >> >> Is there anything to improve the performance in case of leadership changes? > > > Until this patch [1], Neutron would require a connection to the leader for > all threads talking to the SB DB. That lead to unesecary reconnection on > leader change, which may be the source of the issues you see. Try it out and > see if it helps.
It should also be noted that until OVS 2.17, python-ovs didn't support monitor-cond-since. So when these leadership changes happen, the disconnection will cause all registered tables/columns to be dumped to the client. On large DBs, this will absolutely destroy performance. We should at least consider backporting the python RAFT support and the related fixes, as RAFT support in the C library had been there for a while and it seems like an oversight for it not to have been supported in the Python IDL--especially considering just how terrible performance will be without it and with no obvious reason. d29491eeb python: idl: Set cond_changed to true if condition change requested. e3de0bd82 python: idl: Set cond_changed to false if last id is zero. 6de8868d1 reconnect: Fix broken inactivity probe if there is no other reason to wake up. 5202710a7 python: idl: Clear last_id on reconnect if condition changes in-flight. 718dc8fca python: idl: Resend requested but not acked conditions when reconnecting. 46d44cf3b python: idl: Add monitor_cond_since support. and the as-yet unmerged https://patchwork.ozlabs.org/project/openvswitch/patch/20220413135226.593135-1-twil...@redhat.com/ Terry > 0: > https://github.com/openvswitch/ovs/commit/3c2d6274bceecb65ec8f2f93f2aac26897a7ddfe > 1: https://review.opendev.org/q/I7442170d015f195a5430e71567fbc7d67b81d385 > > -- > Frode Nordahl > > >> Kind regards, >> >> Christian Stelter >> _______________________________________________ >> discuss mailing list >> disc...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > _______________________________________________ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
