Performance testing of distributed load balancer route advertisement found
two avoidable sources of Southbound database and controller work:
- A persistent Service_Monitor IDL index is maintained on every monitor
update, even when a chassis has no health-gated advertised routes.
- Publishing one Advertised_Route_Status row per route and chassis adds
write and reconnect costs, although the table has no current consumer.
This series addresses them as follows:
- Patch 1 replaces the persistent index with a temporary hash built only
while active route gates are evaluated. It then scans the monitored
Service_Monitor table once.
- Patch 2 removes Advertised_Route_Status while its consumers and
scalability requirements are reconsidered. No later SB schema changes
had landed, so it restores the pre-status version 21.11.0.
ovn-heater runs compared the persistent index with the temporary hash on an
Ubuntu 24.04 GitHub runner. Each job used a four-vCPU VM, two controller
chassis, 1000 load balancers and 4000 Service_Monitor rows.
The table below shows aggregate ovn-controller CPU over each workload
where a build with a baseline (index present) and candidate (index
dropped in favor of hashing) are compared.
Baseline:
https://github.com/dshcherb/ovn/commit/d79e6cdc54a02d10d44d87d01b884b9b39dd0cec
https://github.com/ovn-org/ovn/compare/main...dshcherb:ovn:benchmark/service-monitor-index
Candidate:
https://github.com/dshcherb/ovn/commit/8007a5e3a37dbeef46a081e17c65d78e9152ccbe
https://github.com/ovn-org/ovn/compare/main...dshcherb:ovn:benchmark/service-monitor-hash
Test run results
(https://github.com/dshcherb/ovn-heater/actions/runs/32301315896):
Feature disabled 40.690 -> 33.125 (-18.59%)
Distributed 98.785 -> 80.180 (-18.83%)
Sparse gates 59.995 -> 49.920 (-16.79%)
The feature-disabled case creates no advertised routes. The distributed
case creates 8000 Advertised_Route rows. The sparse case keeps 4000
monitors but advertises only 10 load balancers.
The temporary hash reduced controller CPU in all comparisons.
A separate run measured Advertised_Route_Status publication. The 8000-row
workload compared the implementation with status writes disabled and
enabled:
ovn-controller CPU +10.52%
SB database CPU +50.92%
ovn-controller peak RSS +6.25%
SB database peak RSS +14.90%
SB reconnect convergence +51.28%
Test results: https://github.com/dshcherb/ovn-heater/actions/runs/32299608680
A 40000-row run measured a 35.83% controller CPU increase and a 51.14% SB
database CPU increase. Controller peak RSS increased by 1.65% and SB
database peak RSS decreased by 1.62%. Distributed transition and SB
reconnect convergence increased by 19.26% and 22.92% respectively.
Test results: https://github.com/dshcherb/ovn-heater/actions/runs/32295995380
A discussion about Advertised_Route_Status can be found here:
https://mail.openvswitch.org/pipermail/ovs-dev/2026-August/435124.html
To reproduce the comparisons, use the Advertised Route Performance workflow
from the advertised-route-perf-gha branch of dshcherb/ovn-heater. Enable
the 40000-row input for the large status comparison.
Dmitrii Shcherbakov (2):
controller: Avoid persistent Service_Monitor index.
controller: Remove Advertised_Route_Status table.
NEWS | 3 -
controller/chassis.c | 27 +--
controller/chassis.h | 4 +-
controller/ovn-controller.c | 297 ++-----------------------------
controller/route-exchange.c | 38 +---
controller/route-exchange.h | 5 +-
controller/route.c | 190 ++++++++++----------
controller/route.h | 23 +--
northd/ovn-northd.c | 17 --
ovn-sb.ovsschema | 34 +---
ovn-sb.xml | 92 ----------
tests/automake.mk | 6 +-
tests/ovn-controller.at | 59 ------
tests/ovn-inc-proc-graph-dump.at | 2 -
tests/ovn-northd.at | 11 --
tests/ovn-sbctl.at | 6 -
tests/system-ovn.at | 184 -------------------
utilities/ovn-sbctl.8.xml | 5 +-
utilities/ovn-sbctl.c | 17 --
19 files changed, 123 insertions(+), 897 deletions(-)
--
2.53.0
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
