Public bug reported: [Impact]
On OVN compute nodes, ovs-vswitchd enters a state where a handler thread spins forever in userspace and never reaches an RCU quiescent point. The main thread is then blocked permanently in ovsrcu_synchronize(). The datapath keeps forwarding existing traffic, but the node stops applying any new network configuration: new instances, migrations, security group and port changes are all silently ignored. ovs-appctl becomes unusable and the OpenFlow connections from ovn-controller are never drained. The only external symptom is a failing healthcheck, so this can go unnoticed for a long time. Recovery requires stopping ovn-controller and restarting ovs-vswitchd; the condition returns as soon as the offending action list is programmed again. The trigger is an ordinary OVN configuration, a logical router with differing port MTUs (which makes OVN emit check_pkt_len for path-MTU handling) plus a packet-in rate limit (which puts a meter in front of it). No unusual setup is required. [Affected package] openvswitch-switch 3.5.1-0ubuntu0.1~cloud0 Ubuntu Noble + Ubuntu Cloud Archive (OpenStack 2025.1 / Epoxy) The offending code appears unchanged in current upstream master, so other series are likely affected as well; we have only verified 3.5.1. [Upstream] Reported to [email protected], currently awaiting moderator approval. Link to the archived thread to follow. [Root cause] dpif_execute_helper_cb() in lib/dpif.c (~lines 1206-1223 in v3.5.1) collects meter actions seen earlier in the action list: {{{ if (aux->meter_action) { const struct nlattr *a = aux->meter_action; do { ofpbuf_put(&execute_actions, a, NLA_ALIGN(a->nla_len)); /* Find next meter action before 'action', if any. */ do { a = nl_attr_next(a); } while (a != action && nl_attr_type(a) != OVS_ACTION_ATTR_METER); } while (a != action); } }}} Both loops terminate only on exact pointer equality with 'action'. This assumes 'aux->meter_action' and 'action' are siblings in the same attribute list. That assumption does not hold: odp_execute_actions() recurses into nested containers -- clone (odp-execute.c:735), sample (753/757) and check_pkt_len (802) -- passing the nested action list but reusing the same 'aux'. So 'aux->meter_action' can be a top-level attribute while 'action' points inside a container. When walking forward from the meter action, nl_attr_next() skips over the whole container attribute, and therefore over the nested 'action' itself. Pointer equality is never reached, the cursor runs past the end of the buffer, and the loop cannot terminate. The action list that triggered it here is plain OVN output: {{{ actions:3,meter(70),pop_vlan, check_pkt_len(size=1456, gt(userspace(pid=...,controller(reason=1,...))), le(drop)) }}} meter(70) is top level; userspace (OVS_ACTION_ATTR_USERSPACE, one of the types handled by dpif_execute_helper_cb) is nested inside check_pkt_len. [Evidence from the stuck process] Log, with the usual RCU warning but unbounded: {{{ 2026-09-07T15:41:29.491Z|00001|ovs_rcu(urcu7)|WARN|blocked 1000 ms waiting for handler14 to quiesce ... 2026-09-07T15:50:00.493Z|00131|ovs_rcu|WARN|blocked 512000 ms waiting for handler14 to quiesce }}} The handler thread is not blocked in a syscall. Its kernel stack is empty, while every other handler shows the expected poll frames: {{{ === handler14 <-- empty: running in userspace === handler15 [<0>] do_poll.constprop.0+0x319/0x3c0 [<0>] do_sys_poll+0x1ef/0x290 [<0>] do_restart_poll+0x59/0xb0 [<0>] __do_sys_restart_syscall+0x25/0x30 [<0>] do_syscall_64+0x87/0x180 [<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80 }}} perf record over 10 s: 40047 samples (~19.6 G cycles) on three adjacent addresses spanning 15 bytes: {{{ 78.31% handler14 ovs-vswitchd [.] 0x00000000000e4840 13.23% handler14 ovs-vswitchd [.] 0x00000000000e4849 5.70% handler14 ovs-vswitchd [.] 0x00000000000e484e }}} Disassembly of that range (offsets verified against the stock package openvswitch-switch 3.5.1-0ubuntu0.1~cloud0, amd64; the shipped binary is byte-identical): {{{ e4820-e4838 ofpbuf_put(&execute_actions, a, NLA_ALIGN(a->nla_len)) e484e-e485b a = nl_attr_next(a) e485e-e4861 while (a != action) -> cmp %r12,%r15 ; jne e4840-e484c nl_attr_type(a) != OVS_ACTION_ATTR_METER (19 == 0x13) }}} So r15 holds 'action' and r12 holds 'a'. The enclosing function spans 0xe4470-0xe4a90 (loop at +0x3d0); its prologue matches the dpif_execute_helper_cb signature (nl_attr_type(action) from arg3, the batch-size-1 assert, the switch on action type), and the call target 0x14dd00 dereferences ofpbuf->data/->size, i.e. ofpbuf_put(). Register state on the stuck thread: {{{ r12 0x73d1037dd914 (a) r15 0x73d1037dd90c (action) x/8xh $r12: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 }}} 'a' is 8 bytes past 'action' -- it has overshot, exactly as the container skip predicts. It then landed on zeroed memory where nla_len is 0, so nl_attr_next() advances by NLA_ALIGN(0) = 0 and the cursor is pinned permanently. nl_attr_type() there is 0, not METER, so ofpbuf_put() is never reached and the buffer does not grow -- the thread just spins on three instructions, matching the perf profile. [Consequences while stuck] * The main thread never returns to its main loop; its poll set contains only two internal latch pipes with a multi-hour timeout. * The unixctl socket is in LISTEN but accept() is never called (strace: 22331 syscalls in 10 s, zero accept/accept4). Pending connections pile up until the 64-slot backlog is full: {{{ u_str LISTEN 65 64 /var/run/openvswitch/ovs-vswitchd.22.ctl }}} ovs-appctl becomes unusable and the container healthcheck fails forever. * OpenFlow connections from ovn-controller on br-int.mgmt are never drained (33916 and 26240 bytes stuck in Recv-Q). The node keeps forwarding existing traffic but applies no new flows. * ovs-vsctl, ovs-dpctl and BFD keep working, since they do not involve the main thread. From the outside this looks like nothing but an unhealthy container. [Test case] On the affected node it reproduces on every vswitchd start, as long as ovn-controller is running: 1. Restart ovs-vswitchd with ovn-controller up -> blocks within minutes, unixctl dead. 2. Stop ovn-controller, restart ovs-vswitchd -> comes up clean, ovs-appctl responds, backlog stays 0, datapath converges. 3. Start ovn-controller again -> it reconnects, reinstalls flows, and the block reappears (on a different handler thread, suggesting a race on which handler picks up the offending packet first). A synthetic reproducer should be an action list with a top-level meter followed by a nested container holding an action that requires datapath assistance, executed through dpif_execute_with_help(), e.g.: {{{ meter:N, check_pkt_len(gt(userspace(...)), le(drop)) meter:N, clone(output:M) }}} In OVN terms: a logical router with options:gateway_mtu set on one port (which emits check_pkt_len for path-MTU handling), plus a packet-in rate limit (which puts a meter in front). Then send oversized DF packets across the router. A quick check for whether a node is currently affected: {{{ ss -xlp | grep 'vswitchd.*ctl' }}} A Recv-Q value that stays above zero means the unixctl socket is no longer being accepted from. [Suggested fix] Two independent hardening steps seem warranted: 1. Terminate on a bound comparison (a >= action, or an explicit end pointer for the list actually being walked) rather than inequality, so an overshoot cannot loop forever. 2. Track meter actions per recursion level, or record the list bounds alongside aux->meter_action, so the walk is never performed across nesting levels. The surrounding comment already notes that this meter collection is an approximation; the nesting case appears not to have been considered. [Environment] {{{ Open vSwitch 3.5.1 (openvswitch-switch 3.5.1-0ubuntu0.1~cloud0), kernel datapath OVN 2025.1, ovn-controller on the same host Ubuntu Noble, Kolla-Ansible containers (OpenStack 2025.1) Dual socket, 2 NUMA nodes, 56 cores Per-CPU upcall dispatch: "Overriding n-handler-threads to 56, setting n-revalidator-threads to 4" -> 60 threads br-int: ~50 geneve tunnels, fail_mode: secure memory: handlers:56 ofconns:4 ports:58 revalidators:4 rules:3911 Kernel live patching is active (klp_x64_sys_call frames in stacks) }}} ** Affects: openvswitch (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2166751 Title: ovs-vswitchd 3.5.1 hangs: infinite loop in dpif_execute_helper_cb() silently freezes the control plane on OVN compute nodes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/2166751/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
