Hi,
We're seeing an issue where IPv6 Neighbor Discovery is completely broken on
certain VPP sub-interfaces because the IPv6 mFIB specific entries
(ff02::1/128, ff02::2/128, ff02::16/128, ff02::1:ff00:0/104) created by the
ND subsystem under the "Special" source do not include all IPv6-enabled
interfaces in the VRF.
VPP version: v25.10-release built on SUSE
SETUP
-----
Two servers connected back-to-back via 10G Intel X710 NICs with SR-IOV. VPP
binds to a VF (iavf driver) on each server for the direct link. Each server
also has a PF (i40e driver) connected to an upstream router. Both the VF
sub-interface and PF sub-interface are in the same IPv6 VRF (table 2).
OSPFv3 runs on the VF sub-interface between the servers (PtP link).
Multiple servers deployed with identical VPP version and configuration.
Interfaces per server in VRF table 2:
- GigE-PF.200 — PF sub-interface (i40e), upstream router, IPv6
2001:db8:1::1/64
- TenG-VF.4091 — VF sub-interface (iavf), server-to-server direct, IPv6
fd00:ab:cd::1/125
- TenG-VF.4092 — VF sub-interface (iavf), additional link, IPv6
fd00:ab:cd::19/125
- host-internal.4091 — internal af_packet, IPv6 fd00:ab:cd::9/125
PROBLEM
-------
IPv6 ping between servers on TenG-VF.4091 fails with 100% loss. IPv4 on the
same interface works fine. OSPFv3 neighbor is Full (uses ff02::5 multicast
which is unaffected). The issue is that incoming Neighbor Solicitation
multicast packets on TenG-VF.4091 are dropped by VPP with "Multicast RPF
check failed" because TenG-VF.4091 is missing from the specific mFIB
entries.
The general ff00::/8 entry correctly includes all 4 interfaces. But the
specific entries only include GigE-PF.200. Since specific entries win via
longest-prefix-match, the RPF check fails for any interface not listed in
them.
This is not consistent across servers — with identical VPP version and
config, some servers have all 4 interfaces in the specific entries
(working), others have only 1 or 2 (broken). The issue is deterministic per
server and survives VPP restart.
WHAT WE TRIED (none fixed the mFIB)
------------------------------------
1. ip mroute add/del via CLI — silently ignored, cannot modify "Special"
source entries
2. ip mroute add with Accept-all-itf flag — added as CLI source but Special
source takes precedence
3. Removing and re-adding IPv6 addresses on affected interface — re-joins
ff00::/8 but not specific entries
4. Removing IPv6 from ALL interfaces and re-adding — specific entries
persist with empty lists, only one interface gets re-added
5. Interface admin down/up — no effect on mFIB
6. VPP restart — reproduces same broken state
7. Static IPv6 neighbors (bypassing ND) — unicast IPv6 works perfectly,
confirming data plane is fine
QUESTIONS
---------
1. Is there a known issue with the ND subsystem not adding interfaces to
existing specific mFIB entries when they join the same multicast group?
2. Is there a way to force the Special source to re-evaluate interface
membership at runtime?
3. Is there a startup config option that controls this behavior?
Detailed logs and mFIB dumps from both working and broken servers are in
the attached file.
Thanks for any guidance.
================================================================================
VPP IPv6 mFIB Bug — Detailed Logs
VPP version: v25.10-release built on SUSE
================================================================================
================================================================================
1. INTERFACE CONFIGURATION
================================================================================
--- Server-A (BROKEN) ---
# show interface addr TenG-VF.4091
TenG-VF.4091 (up):
L3 172.16.0.1/29 ip4 table-id 2 fib-idx 5
L3 fd00:ab:cd::1/125 ip6 table-id 2 fib-idx 2
# show interface addr GigE-PF.200
GigE-PF.200 (up):
L3 10.0.1.1/29 ip4 table-id 2 fib-idx 5
L3 2001:db8:1::1/64 ip6 table-id 2 fib-idx 2
# show hardware-interfaces GigE-PF
Intel X710/XL710 Family
flags: admin-up promisc maybe-multiseg subif tx-offload intel-phdr-cksum
rx-ip4-cksum int-supported int-unmaskable
promiscuous: unicast on all-multicast on
# show hardware-interfaces TenG-VF
Intel iAVF
flags: admin-up maybe-multiseg subif tx-offload intel-phdr-cksum
rx-ip4-cksum int-unmaskable
promiscuous: unicast off all-multicast on
--- Server-B (peer, also BROKEN) ---
# show interface addr TenG-VF.4091
TenG-VF.4091 (up):
L3 172.16.0.3/29 ip4 table-id 2 fib-idx 5
L3 fd00:ab:cd::3/125 ip6 table-id 2 fib-idx 2
================================================================================
2. IPv4 PING — WORKS
================================================================================
# ping 172.16.0.3 source TenG-VF.4091 repeat 3
116 bytes from 172.16.0.3: icmp_seq=1 ttl=64 time=3.1172 ms
116 bytes from 172.16.0.3: icmp_seq=2 ttl=64 time=5.8330 ms
116 bytes from 172.16.0.3: icmp_seq=3 ttl=64 time=.0552 ms
Statistics: 3 sent, 3 received, 0% packet loss
================================================================================
3. IPv6 PING — FAILS
================================================================================
# ping fd00:ab:cd::3 source TenG-VF.4091 repeat 5
Statistics: 5 sent, 0 received, 100% packet loss
# (from Linux) ip netns exec vpp-ns ip vrf exec iac2 ping6 -c 5 fd00:ab:cd::3
From fd00:ab:cd::1 icmp_seq=1 Destination unreachable: Address unreachable
From fd00:ab:cd::1 icmp_seq=2 Destination unreachable: Address unreachable
From fd00:ab:cd::1 icmp_seq=3 Destination unreachable: Address unreachable
From fd00:ab:cd::1 icmp_seq=4 Destination unreachable: Address unreachable
From fd00:ab:cd::1 icmp_seq=5 Destination unreachable: Address unreachable
================================================================================
4. IPv6 NEIGHBOR TABLE — EMPTY
================================================================================
# show ip6 neighbors TenG-VF.4091
(empty)
# (Linux) ip netns exec vpp-ns ip -6 neigh show dev TenG-VF.4091
fd00:ab:cd::3 FAILED
================================================================================
5. IPv6 ND INTERFACE CONFIG — ND IS ENABLED
================================================================================
--- Server-A ---
# show ip6 interface TenG-VF.4091
TenG-VF.4091 is admin up
Local unicast address(es):
fd00:ab:cd::1/125
Link-local address(es):
fe80::f1:c4ff:fe20:6909
Joined group address(es):
ff02::1
ff02::2
Neighbor Discovery: enabled
--- Server-B ---
# show ip6 interface TenG-VF.4091
TenG-VF.4091 is admin up
Local unicast address(es):
fd00:ab:cd::3/125
Link-local address(es):
fe80::87:deff:fef0:5dcc
Joined group address(es):
ff02::1
ff02::2
Neighbor Discovery: enabled
================================================================================
6. mFIB — BROKEN SERVER (Server-A)
================================================================================
# show ip6 mfib table 2
(*, ::/0): flags:Drop,
Interfaces:
(*, ff00::/8):
Interfaces:
TenG-VF.4092: Accept,
TenG-VF.4091: Accept, <-- PRESENT in general entry
host-internal.4091: Accept,
GigE-PF.200: Accept,
(*, ff02::1/128):
Interfaces:
GigE-PF.200: Accept, <-- TenG-VF.4091 MISSING
(*, ff02::2/128):
Interfaces:
GigE-PF.200: Accept, <-- TenG-VF.4091 MISSING
(*, ff02::16/128):
Interfaces:
GigE-PF.200: Accept, <-- TenG-VF.4091 MISSING
(*, ff02::1:ff00:0/104):
Interfaces:
GigE-PF.200: Accept, <-- TenG-VF.4091 MISSING
--- Detailed entry ---
# show ip6 mfib table 2 ff02::1:ff00:0/104
(*, ff02::1:ff00:0/104):
fib:2 index:29 locks:1
src:Special flags:none locks:1:
path-list:[135] locks:2 flags:no-uRPF, uPRF-list:1414 len:1 itfs:[229, ]
path:[153] pl-index:135 ip6 weight=1 pref=0 receive:
oper-flags:resolved, cfg-flags:local,
[@0]: dpo-receive
path:[2597] pl-index:135 ip6 weight=1 pref=0 attached:
oper-flags:resolved,
GigE-PF.200
Extensions:
path:153 flags:Forward,
path:2597 flags:Accept,
Interface-Forwarding:
GigE-PF.200: Accept,
Interfaces:
GigE-PF.200: Accept,
================================================================================
7. mFIB — WORKING SERVER (Server-C, identical VPP version and config)
================================================================================
# show ip6 mfib table 2
(*, ff00::/8):
Interfaces:
TenG-VF.4092: Accept,
TenG-VF.4091: Accept,
host-internal.4091: Accept,
GigE-PF.300: Accept,
(*, ff02::1/128):
Interfaces:
TenG-VF.4092: Accept,
TenG-VF.4091: Accept, <-- PRESENT
host-internal.4091: Accept,
GigE-PF.300: Accept,
(*, ff02::2/128):
Interfaces:
TenG-VF.4092: Accept,
TenG-VF.4091: Accept, <-- PRESENT
host-internal.4091: Accept,
GigE-PF.300: Accept,
(*, ff02::16/128):
Interfaces:
TenG-VF.4092: Accept,
TenG-VF.4091: Accept, <-- PRESENT
host-internal.4091: Accept,
GigE-PF.300: Accept,
(*, ff02::1:ff00:0/104):
Interfaces:
TenG-VF.4092: Accept,
TenG-VF.4091: Accept, <-- PRESENT
host-internal.4091: Accept,
GigE-PF.300: Accept,
--- Detailed entry ---
# show ip6 mfib table 2 ff02::1:ff00:0/104
(*, ff02::1:ff00:0/104):
fib:2 index:29 locks:1
src:Special flags:none locks:1:
path-list:[135] locks:2 flags:no-uRPF, uPRF-list:1724 len:4 itfs:[12, 14,
16, 231, ]
path:[153] pl-index:135 ip6 weight=1 pref=0 receive:
oper-flags:resolved, cfg-flags:local,
[@0]: dpo-receive
path:[2764] pl-index:135 ip6 weight=1 pref=0 attached: oper-flags:drop,
if_index:0
path:[2736] pl-index:135 ip6 weight=1 pref=0 attached:
oper-flags:resolved,
TenG-VF.4092
path:[2821] pl-index:135 ip6 weight=1 pref=0 attached:
oper-flags:resolved,
TenG-VF.4091
path:[2820] pl-index:135 ip6 weight=1 pref=0 attached:
oper-flags:resolved,
host-internal.4091
Extensions:
path:153 flags:Forward,
path:2821 flags:Accept,
path:2764 flags:Accept,
path:2736 flags:Accept,
path:2820 flags:Accept,
Interface-Forwarding:
TenG-VF.4092: Accept,
TenG-VF.4091: Accept,
host-internal.4091: Accept,
GigE-PF.300: Accept,
================================================================================
8. mFIB — ANOTHER BROKEN SERVER (Server-D, partial — 2 of 4 interfaces)
================================================================================
# show ip6 mfib table 2 (specific entries only)
(*, ff02::1:ff00:0/104):
Interfaces:
host-internal.4091: Accept,
GigE-PF.300: Accept, <-- 2 interfaces, TenG-VF.4091 still MISSING
================================================================================
9. PACKET TRACE — NS DROPPED BY RPF CHECK
================================================================================
# clear trace
# trace add dpdk-input 200
# (trigger ping from peer)
Packet:
dpdk-input
TenG-VF rx queue 0
buffer: current data 0, length 90
IP6: 02:aa:bb:cc:dd:ee -> 33:33:ff:20:69:09 802.1q vlan 4091
ICMP6: fe80::aabb:ccff:fedd:eeff -> ff02::1:ff20:6909
tos 0x00, flow label 0x0, hop limit 255, payload length 32
ICMP neighbor_solicitation checksum 0x6ad0
target address fe80::f1:c4ff:fe20:6909
ethernet-input
frame: flags 0x3, hw-if-index 3, sw-if-index 3
IP6: 02:aa:bb:cc:dd:ee -> 33:33:ff:20:69:09 802.1q vlan 4091
ip6-input
ICMP6: fe80::aabb:ccff:fedd:eeff -> ff02::1:ff20:6909
ip6-mfib-forward-lookup
fib 5 entry 29
ip6-mfib-forward-rpf
entry 29 itf -1 flags <-- itf -1 = no accepted interface
ip6-drop
fib:2 adj:29 flow:0x00000012
ICMP6: fe80::aabb:ccff:fedd:eeff -> ff02::1:ff20:6909
error-drop
rx:TenG-VF.4091
drop
ip6-input: Multicast RPF check failed <-- DROP
================================================================================
10. VPP ERROR COUNTERS
================================================================================
# show errors | grep -i "ip6.*multicast\|ip6.*neighbor\|mfib"
3 ip6-glean neighbor solicitations sent
13 ip6-icmp-input neighbor solicitations for unknown
6092 ip6-icmp-input neighbor discovery not configured
1089 ip6-input Multicast RPF check failed
================================================================================
11. STATIC NEIGHBOR WORKAROUND — CONFIRMS DATA PLANE IS FINE
================================================================================
# set ip neighbor TenG-VF.4091 fd00:ab:cd::3 02:aa:bb:cc:dd:ee
# (Linux) ip netns exec vpp-ns ip -6 neigh replace fd00:ab:cd::3 lladdr
02:aa:bb:cc:dd:ee dev TenG-VF.4091 nud permanent
# (same on peer server for reverse direction)
# ping fd00:ab:cd::3 source TenG-VF.4091 repeat 5
76 bytes from fd00:ab:cd::3: icmp_seq=1 ttl=64 time=1.1261 ms
76 bytes from fd00:ab:cd::3: icmp_seq=2 ttl=64 time=4.1163 ms
76 bytes from fd00:ab:cd::3: icmp_seq=3 ttl=64 time=8.1090 ms
76 bytes from fd00:ab:cd::3: icmp_seq=4 ttl=64 time=1.1252 ms
76 bytes from fd00:ab:cd::3: icmp_seq=5 ttl=64 time=5.1118 ms
Statistics: 5 sent, 5 received, 0% packet loss
# (Linux) ip netns exec vpp-ns ip vrf exec iac2 ping6 -c 5 fd00:ab:cd::3
64 bytes from fd00:ab:cd::3: icmp_seq=1 ttl=64 time=0.133 ms
64 bytes from fd00:ab:cd::3: icmp_seq=2 ttl=64 time=6.53 ms
64 bytes from fd00:ab:cd::3: icmp_seq=3 ttl=64 time=5.35 ms
64 bytes from fd00:ab:cd::3: icmp_seq=4 ttl=64 time=0.180 ms
64 bytes from fd00:ab:cd::3: icmp_seq=5 ttl=64 time=0.183 ms
5 packets transmitted, 5 received, 0% packet loss
================================================================================
12. REMEDIATION ATTEMPTS — ALL FAILED
================================================================================
--- Attempt 1: ip mroute add via CLI ---
# ip mroute add ff02::1:ff00:0/104 table 2 via TenG-VF.4091 Accept
(no error, but mFIB unchanged — CLI source doesn't modify Special source)
--- Attempt 2: Accept-all-itf flag ---
# ip mroute add ff02::1:ff00:0/104 table 2 Accept-all-itf
(added as CLI source with Accept-all-itf flag, but Special source takes
precedence)
# show ip6 mfib table 2 ff02::1:ff00:0/104
src:Special flags:none locks:1:
...
Interface-Forwarding:
GigE-PF.200: Accept,
src:CLI flags:none locks:1: flags:Accept-all-itf, <-- added but ignored
Extensions:
Interface-Forwarding:
Interfaces:
GigE-PF.200: Accept, <-- unchanged
--- Attempt 3: ip mroute del ---
# ip mroute del ff02::1:ff00:0/104 table 2 via GigE-PF.200 Accept
(no error, but entry unchanged — cannot delete Special source paths)
--- Attempt 4: Remove/re-add IPv6 on affected interface ---
# set interface ip address del TenG-VF.4091 fd00:ab:cd::1/125
# set interface ip address TenG-VF.4091 fd00:ab:cd::1/125
(interface re-added to ff00::/8 but NOT to specific entries)
--- Attempt 5: Remove IPv6 from ALL interfaces, re-add all ---
# (removed all 4 IPv6 addresses)
(specific entries persist with EMPTY interface lists)
# (re-added all 4 IPv6 addresses)
(only GigE-PF.200 re-appears in specific entries — same broken state)
--- Attempt 6: Interface admin bounce ---
# set interface state TenG-VF.4091 down
# set interface state TenG-VF.4091 up
(no effect on mFIB)
--- Attempt 7: VPP restart ---
# systemctl restart vpp
(mFIB reproduces same broken state — deterministic)
--- Attempt 8: Enable promiscuous on VF ---
# set interface promiscuous on TenG-VF
(promiscuous enabled, but mFIB unchanged after IPv6 address bounce)
================================================================================
13. OSPFv3 — WORKS (uses ff02::5, matches ff00::/8)
================================================================================
OSPFv3 neighbor on TenG-VF.4091: Full/PtP
OSPFv3 uses ff02::5 multicast which matches the general ff00::/8 entry
(all interfaces present in ff00::/8), so OSPFv3 is unaffected.
This confirms the physical link and VLAN tagging are correct.
Only the specific mFIB entries (ff02::1, ff02::2, ff02::16, ff02::1:ff00:0/104)
are broken.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#26977): https://lists.fd.io/g/vpp-dev/message/26977
Mute This Topic: https://lists.fd.io/mt/118969596/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
