https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240106
--- Comment #28 from kvs <overwa...@lab.kyngin.net> --- (In reply to Slawomir Wojciech Wojtczak from comment #27) I have some headway on my end, though I don't know how much it's related to the earlier bugs at this point. After further testing, vlans apparently aren't related to my problem. The problem occurs on lagg without vlan interfaces. When a jail+VNET (on bridge) sends an ARP request it traverses the bridge and exits both interfaces in the host lagg group. When the ARP reply comes back, it appears it will only ever enter the host bridge if it comes in on the primary lagg member. I'm not certain this is exclusive to vnets, also possibly this is normal operation for laggs using lacp? Lab test: lagg0 (ports cc0 + cc1), bridge2020 (members epair0a & lagg0) ping from jail+VNET to switch (10.20.20.254), using source epair0b (10.20.20.77) (epair0b -> epair0a -> bridge2020 -> lagg0 -> cc0/cc1 -> switch) tcpdump -i epair0a 10:00:17.981011 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 tcpdump -i bridge2020 10:00:17.981051 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 tcpdump -i lagg0 10:00:17.981030 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 10:00:17.981282 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 tcpdump -i cc0: 10:00:17.981050 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 42 tcpdump -i cc1: 10:00:17.981041 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 10:00:17.981282 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 Arp table is not populated on VM, as bridge2020 and epair0a/b never sees ARP reply come in over cc1. I believe in my case specifically the switch is seeing cc1 as the primary lagg member while the FreeBSD server sees cc0 as the primary lagg member. When ARP replies manage to come in over cc0, the ARP replies make it to the vnet interface and the jail populates its ARP table. I can force this event by downing cc1 or shutting down the cc1 switch port (in both cases it appears the switch then identifies cc0 as the primary lagg member over which it sends ARP replies). Alternatively, if both cc0 and cc1 are up, and the switch sends an ARP reply over cc0 (has happened randomly), the ARP reply does makes it through the bridge/epair and populates the ARP cache on the VM. Example after ifconfig cc1 down: tcpdump -i epair0a 10:48:18.949695 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 tcpdump -i bridge2020 10:48:18.949731 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 tcpdump -i lagg0 10:48:18.949711 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 tcpdump -i cc0 10:48:18.949722 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 ARP table on VM is now populated with switch address, and everything appears to work as normal over lagg0 (with cc0 up / cc1 down). In the mean time I've managed to get the switch configured to send L2/ARP over both lagg members which has fixed the immediate problem. Though I do think it's strange that FreeBSD populates the ARP table just fine on the host over cc1, but just wont send that ARP reply over the bridge interface unless it comes in on cc0. That *feels* like a bug, as it only seems to affect the second interface on a lagg that's in a bridge, and quite possibly only for layer 2 (L2/3 needs further testing - I've not lost packets once the arp table is populated, but it's possible the switch was handling layer 3 differently and always using the cc0 port, in which case FreeBSD would probably send over the bridge without trouble). Testing has been performed on 14-CURRENT and 13-STABLE with identical results. -- You are receiving this mail because: You are the assignee for the bug.
