Hello, all. I decided to reinstall OpenBSD to a newer version on my VMware ESXi cluster. So I deleted an old router and start the new one using the old configuration, except that I add lladdr parameter with the old MAC address to the external interface to avoid blocking traffic on the port of the Internet provider.
After applying the configuration and rebooting, I've got packet loss. The strange thing is that packets started passing when I turned on BPF on another terminal: terminal[1]: router1# uname -a OpenBSD router1 6.7 GENERIC.MP#182 amd64 router1# pfctl -d pfctl: pf not enabled router1# ping -c3 185.1.162.100 PING 185.1.162.100 (185.1.162.100): 56 data bytes --- 185.1.162.100 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss router1# arping -i vmx2 -c3 185.1.162.100 ARPING 185.1.162.100 Timeout Timeout Timeout --- 185.1.162.100 statistics --- 3 packets transmitted, 0 packets received, 100% unanswered (0 extra) Now, I run a tcpdump on another terminal and don't turn it off: terminal[2]: router1# tcpdump -ni vmx2 -e net 185.1.162.0/24 tcpdump: listening on vmx2, link-type EN10MB 07:55:47.123666 68:b5:99:c5:2a:c0 00:50:56:92:d1:18 0800 85: 185.1.162.100.55992 > 185.1.162.2.179: P 1989291079:1989291098(19) ack 284218081 win 229 <nop,nop,timestamp 448227401 881035409>: BGP (KEEPALIVE) (DF) [tos 0xc0] [ttl 1] 07:55:47.318977 00:50:56:92:d1:18 68:b5:99:c5:2a:c0 0800 66: 185.1.162.2.179 > 185.1.162.100.55992: . ack 19 win 543 <nop,nop,timestamp 881035415 448227401> [tos 0xc0] [ttl 1] ... Let's back on terminal[1] and run ping: router1# ping -c3 185.1.162.100 PING 185.1.162.100 (185.1.162.100): 56 data bytes 64 bytes from 185.1.162.100: icmp_seq=0 ttl=64 time=0.357 ms 64 bytes from 185.1.162.100: icmp_seq=1 ttl=64 time=0.376 ms 64 bytes from 185.1.162.100: icmp_seq=2 ttl=64 time=0.399 ms --- 185.1.162.100 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.357/0.377/0.399/0.017 ms router1# arping -i vmx2 -c3 185.1.162.100 ARPING 185.1.162.100 60 bytes from 68:b5:99:c5:2a:c0 (185.1.162.100): index=0 time=320.700 usec 60 bytes from 68:b5:99:c5:2a:c0 (185.1.162.100): index=1 time=368.795 usec 60 bytes from 68:b5:99:c5:2a:c0 (185.1.162.100): index=2 time=381.975 usec --- 185.1.162.100 statistics --- 3 packets transmitted, 3 packets received, 0% unanswered (0 extra) rtt min/avg/max/std-dev = 0.321/0.357/0.382/0.026 ms Now if I will stop tcpdump on terminal[2] I'll get packet loss again. This is a weird behavior. What could be wrong? This router is a virtual machine on VMware ESXi server with a network adapter type VMXNET3, which is mapped to the vmx2 interface in guest OS (OpenBSD 6.7): router1# grep vmx2 /var/run/dmesg.boot | tail -1 vmx2 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 4 int 16, address 00:50:56:92:02:51 router1# cat /etc/hostname.vmx2 inet 185.1.162.3 255.255.255.0 NONE description "Linx-IX" lladdr "00:50:56:92:c7:27" router1# ifconfig vmx2 vmx2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:50:56:92:c7:27 description: Linx-IX index 3 priority 0 llprio 3 media: Ethernet autoselect (10GbaseT) status: active inet 185.1.162.3 netmask 0xffffff00 broadcast 185.1.162.255 router1# The VMXNET3 interface belongs to vSwitch which contains allowable policy ruleset: [root@esxi5:~] esxcli system version get Product: VMware ESXi Version: 6.0.0 Build: Releasebuild-2494585 Update: 0 Patch: 0 [root@esxi5:~] esxcli network vswitch standard policy security get -v vSwitch3 Allow Promiscuous: true Allow MAC Address Change: true Allow Forged Transmits: true [root@esxi5:~] Don't know if it helps to understand what's going on. My dmesg.boot is here https://gist.github.com/dynax60/b1d7e89b0f1b592b2d7bd0fd99b0272b I removed the lladdr parameter from /etc/hostname.vmx2 and asked the ISP network engeneer to clear the old MAC address 00:50:56:92:c7:27 on port-security of the Cisco network switch. After that, I rebooted the router and it worked as it should. I think to avoid subsequent contacts with ISP with the request to clear MAC-address, the best way would be to use lladdr to keep an old MAC address on interface, because reinstalling the virtual machine on VMware ESXi server will entail a change in the mac address on the interface. The strange thing is that this does not work for me. I don't know why. P.S. offtopic: I turn to the developers: I tearfully ask you to add python to the base installation. This would allow the configuration to be rolled automatically using Ansible right after install an OS. It would simplify the task of configuring OpenBSD on remote hosts. -- Regards, Denis
