Hi Pim, I don't know if the scenario you described (process RTM_DELADDR for link-local address, NS arrives on interface & causes crash) is possible. As part of tearing down IPv6 on an interface, ip6-not-enabled should be enabled for that interface on ip6-unicast/ip6-multicast. As you noted, the operations to delete/add interface addresses are performed with workers waiting at a barrier. So it seems like an NS that arrives on that interface after a call to ip6_link_disable() should just be dropped because it will run into ip6-not-enabled. Maybe there are edge cases that I'm not considering though.
I'm skeptical that making the call to ip6_disable_link() conditional on (0 == lcp_router_count_interface_addresses (sw_if_index, AF_IP6)) will help. The call to ip6_link_disable() only initiates the teardown of IPv6 on an interface if the number of locks reaches 0 as part of the operation. So if the condition evaluates false and you omit the call to ip6_link_disable(), ip6_link_disable() would not have done anything that could have caused the crash in that case, it only would have decremented the lock count. If you want to share your core files, I can look and see if anything jumps out at me. -Matt On Sun, Jun 14, 2026 at 4:47 AM Pim van Pelt via lists.fd.io <pim= [email protected]> wrote: > Hoi, > > I've found that issuing a new link-local from Linux will emit a Netlink > address remove followed by an address add with the new address. > Linux CP will see this removal and disable linklocal processing on the > interface in src/plugins/linux-cp/lcp_router.c:686: > > if (ip6_address_is_link_local_unicast (&ip_addr_v6 (&nh))) > if (is_del) > ip6_link_disable (sw_if_index); > else > { > ip6_link_enable (sw_if_index, NULL); > ip6_link_set_local_address (sw_if_index, &ip_addr_v6 > (&nh)); > } > > The subsequent add will re-enable and set the link-local address. There is > no ip6_link_remove_local_address() so the code as written makes sense, but > there's a lot of book-keeping that occurs on link disable/enable, including > address listeners, MLD, VRRP, FIB changes and a few other bits and pieces. > I propose that we disable the link iff there are no addresses left, rather > than unconditionally on ip6ll changes from netlink. > > https://gerrit.fd.io/r/c/vpp/+/46038 has this change, but I'm not sure as > the original crash is related to having no ilt_fibs entry for the > sw_if_index, which is done by ip6_link_disable(), but should be re-enabled > with the subsequent add from ip6_link_enable+set_address. Considering these > calls are done under barrier lock, the only explanation I could come up > with, is that the address delete comes in on batch #1, leaving > ilt_fibs[sw_if_index] = ~0, while the address add come in on batch #2, and > the window of time between #1 and #2 is where an incoming NS can cause the > crash. > > Matt, what do you think? > > > On 10.06.2026 21:27, Pim van Pelt via lists.fd.io wrote: > > Hoi, > > I am running VPP on a few aarch64 machines and observed regular crashes > with a stacktrace that suggests IPv6 FIB lookup issue - > > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #0 0x0000fc1b8ac808f8 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from linux-vdso.so.1 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #1 0x0000fc1b8998ac3c > ip6_fib_table_lookup_exact_match + 0x3c > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from > /lib/aarch64-linux-gnu/libvnet.so.26.06 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #2 0x0000fc1b89a21d2c > proxy_arp_intfc_walk + 0x4030 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from > /lib/aarch64-linux-gnu/libvnet.so.26.06 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #3 0x0000fc1b89101658 > vlib_exit_with_status + 0x808 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from > /lib/aarch64-linux-gnu/libvlib.so.26.06 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #4 0x0000fc1b89103eb0 > vlib_exit_with_status + 0x3060 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from > /lib/aarch64-linux-gnu/libvlib.so.26.06 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #5 0x0000fc1b8912d88c > vlib_worker_thread_bootstrap_fn + 0x6c > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from > /lib/aarch64-linux-gnu/libvlib.so.26.06 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #6 0x0000fc1b88de595c > pthread_condattr_setpshared + 0x5bc > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from > /lib/aarch64-linux-gnu/libc.so.6 > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #7 0x0000fc1b88e4bb4c __clone + > 0x2cc > Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from > /lib/aarch64-linux-gnu/libc.so.6 > > I was lucky enough to get two coredumps out of it, which I fed to Claude > and it came back with an analysis that pointed to me having changed the > link-local address on an interface: > > The interface did have IPv6/ND enabled — otherwise the > icmp6_neighbor_solicitation node would never have run. What it transiently > lacked was its per-interface link-local FIB table (ilt_fibs[sw_if_index]), > which is a structure distinct from "IPv6 is enabled." That table is created > lazily when a link-local address is added (ip6_ll_fib_create) and freed, > with the slot reset to ~0, when the last FIB_SOURCE_IP6_ND entry is removed > (ip6_ll_table.c:150-154). Crucially, ip6_link_set_local_address() > (ip6_link.c:359-362) changes a link-local address by doing > delete-then-update: it removes the old LL prefix (which frees the FIB and > sets ilt_fibs = ~0), then re-adds the new one (which recreates it). The ND > node stays enabled across this whole sequence. So the NS did not arrive > anywhere unexpected — it hit a normally-configured interface during the > brief window in which its LL FIB had been torn down and not yet rebuilt. A > worker thread forwarding an ordinary link-local NS in that window read > ilt_fibs == ~0 and segfaulted; two crashes 31 minutes apart is consistent > with two separate LL-address-change events, matching your note that you'd > been changing link-local addresses. > > The debugging session made my head spin a little bit as I'm not very good > with gdb, but what the cores do prove: an unguarded ~0 from > ip6_ll_fib_get() in the link-local NS branch causes an out-of-bounds > pool_elt_at_index; the trigger is an NS for a link-local target arriving > while ilt_fibs[sw_if_index] == ~0. > > The fix is a simple check of ip6_ll_fib_get() before doing the FIB lookup, > just as happens a few lines further down in the same vnet/ip6-nd/ip6_nd.c > file. > A candidate fix is in https://gerrit.fd.io/r/c/vpp/+/46038 and I have not > observed crashes after applying it, although I'm not certain if returning > FIB_NODE_INDEX_INVALID and dropping the packet is the right call in this > case, or if we have to do something more? > > I have the coredumps and symbols here if somebody wants to take a closer > look. > > groet, > Pim > > -- > Pim van Pelt <[email protected]> <[email protected]> > PBVP1-RIPE https://ipng.ch/ > > > > -- > Pim van Pelt > PBVP1-RIPE - https://ipng.ch/ > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27073): https://lists.fd.io/g/vpp-dev/message/27073 Mute This Topic: https://lists.fd.io/mt/119745721/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
