We still hit this with 7a70217d78 applied, through another path, and can
reproduce it on master.

7a70217d78 keeps the floating IP's revision row when the deleting worker cannot 
see the NAT, so that
maintenance retries the delete. But the retry can run on a worker that cannot 
see the NAT either; it
then drops the row, and the NAT stays in the OVN Northbound database for good.

This happens whenever the maintenance worker holding the lock works from stale 
data. For example, the
NB raft leader freezes long enough for the followers to elect a new leader, 
while the lock holder's
inactivity probe has not fired yet: it stays attached to the old leader, keeps 
the lock and runs
maintenance from its frozen copy of the NB, while the other units move to the 
new leader. A floating
IP associated and deleted in that window has a NAT the stale worker cannot see, 
so it drops the
revision row. Nothing brings the NAT back to maintenance, and a later floating 
IP with the same
address on the same router loses N/S traffic, as described in this bug.

The attached functional test 
(test_floating_ip_delete_stale_view_keeps_revision, test_maintenance.py)
deletes a floating IP without its NAT, runs one maintenance pass that cannot 
see the NAT, and checks
that the row is kept and that a normal pass then deletes the NAT. It fails on 
master (37519b260d).

We also reproduced it end to end in a lab, on master and on 2026.1 with 
7a70217d78 backported:
3-member NB cluster, three API units, leader frozen with SIGSTOP (the lock 
holder had a longer
ovsdb_probe_interval so it is the last to notice; with equal settings it 
depends on timing). The NAT
was left behind, and a new floating IP with the same address on the same router 
passed 0 of 10 pings;
removing only the leftover NAT restored it.

Proposed fix: keep a deleted resource's revision row as a tombstone for a 
configurable period
(default 2 hours, a margin over how long a worker can act on a stale view) 
instead of dropping it when
the OVN object is not seen; the first pass with an up-to-date view then cleans 
up. Done in one place
(a session hook records the deletion of any standard-attribute resource; 
delete_revision() keeps those
rows), it covers every resource type. A prototype makes the attached test pass; 
six existing
functional tests that expect the row gone right after a delete would need 
updating.

Happy to propose the test and the fix on Gerrit if this direction looks
acceptable.


** Patch added: "Functional test: maintenance task drops revision row from 
stale data"
   
https://bugs.launchpad.net/neutron/+bug/1987530/+attachment/6016570/+files/0001-OVN-Add-test-for-maintenance-with-a-stale-NB-view.patch

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1987530

Title:
  Duplicate external_ip in NAT table lead to loss of N/S connectivity

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1987530/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to