I tested the package 22.03.8-0ubuntu0.22.04.2 from jammy-proposed and the service stop path works as expected. Upon executing 'systemclt stop ovn-controller' the corresponding chassis is removed from the 'chassis_private' and 'chassis' registers in the OVN SB db. After the subsequent restart, the chassis registers itself in the SB db properly. The NB db, 'lrp-get-gateway-chassis' shows it with the lowest priority, which is expected.
Regarding the regression test, I conducted an extensive set of testing on Jammy, Noble, and Resolute multi-node environments. The test scenario was the same for all three releases: - ping with 100 ms interval running from bastion to VM1 (north-south traffic) - ping with 100 ms interval running from VM1 to VM2 (each VM on a separate compute node) (east-west traffic) - at least five consecutive ovn-host and ovn-common package upgrade downgrades within the release Tested package versions: Jammy: 22.03.8-0ubuntu0.22.04.1 (pre-patch), 22.03.8-0ubuntu0.22.04.2 (with patch) Noble: 24.03.2-0ubuntu0.24.04.2, 24.03.8-0ubuntu0.24.04.1 Resolute: 26.03.0-1 (from my PPA [3]), 26.03.0-2 During the tests on both Jammy and Noble I observed an individual ping packets loss during the ovn-controller restart sequence after the ovn-host package upgrade/downgrade. After changing the external_ids:ovn-ofctrl-wait-before-clear to 5000 milliseconds (the default is 0 [2]), and repeating the same set of tests (multiple repetitions), I haven't observed a single packet loss during the ovn-host package upgrade/downgrade. I haven't reproduce the data plane outage during the apt operations on Resolute running OVN 26.03. Considering the fact that the problem on Jammy and Noble was mitigated by setting high enough 'external_ids:ovn-ofctrl-wait-before-clear' and the fact that OVN 25.09 completely removed the support for this flag (now the OpenFlow removal wait time is dynamic and coordinated) [4], meaning Resolute using OVN 26.03 bas 'buit-in' capability to delay crearing the flows upon controller restart, it is important to calculate the flow clear dealy properly (as documented in [2]). Reading the flag documentation [2] its sole purpose was to mitigate data plane outage during the ovn-controller restarts or upgrades. So it is expected that the flows are recalculated and reapplied during the controller startup. Finally, I was able to reproduce the same data plane packet loss on Jammy package version 22.03.8-0ubuntu0.22.04.1, which does not contain the fix by forcing package reinstall: sudo apt install ovn-host=22.03.8-0ubuntu0.22.04.1 ovn- common=22.03.8-0ubuntu0.22.04.1 ovn-central=22.03.8-0ubuntu0.22.04.1 --reinstall On controller startup/reconnection, OVN recomputes its desired flows and installs the flow/group changes through ofctrl_put(). The initial deletion of existing flows and installation of replacement flows are included in the same OpenFlow bundle, using OFPBF_ORDERED | OFPBF_ATOMIC, and applied on bundle commit. Therefore, observing flow recreation inevitable but does not automatically mean the table flow is empty. Atomic installation does not, however, guarantee that the initially computed replacement flow set is complete. This is where for OVN releases pre 25.09 the 'external_ids:ovn-ofctrl-wait-before-clear' flag comes into play. The code links supporting the statement above: Atomic bundle creation [5] Initial flow/group deletion [6] Replacement groups and flows [7] Bundle commit [8] Conclusions: My testing found no evidence that commit [1] introduces a dataplane regression. Individual single-packet loss was observed during package operations both before and after the change on Jammy. It was also reproduced in a multi-node Noble deployment. The losses correlate with ovn-controller startup and are consistent with the documented behavior of 'ovn-ofctrl-wait-before-clear' (default 0). Configuring a 5000 ms delay avoided observed loss in repeated tests on both releases. Repeated multi-node Resolute tests with OVN 26.03, which includes the automatic startup coordination introduced in OVN 25.09, showed no packet loss. [1] https://git.launchpad.net/ubuntu/+source/ovn/commit/?h=ubuntu/jammy-proposed&id=72825ae75e969d042f83aa907794fe24337b24ec [2] https://manpages.ubuntu.com/manpages/noble/man8/ovn-controller.8.html#external_ids:ovn~17 [3] https://launchpad.net/~wilkmarcin/+archive/ubuntu/lp2150130 [4] https://www.ovn.org/en/releases/25.09/ [5] https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2766 [6] https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2779 [7] https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2802 [8] https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2884 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2150130 Title: --restart in ovn-controller leaves gateway ports active To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2150130/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
