> Synopsis: PXE install of OpenBSD 7.3 (amd64) fails on Protectli
VP4650 & VP2420, with 'igc' Intel I225-V 2.5Gbps NICs
> Category: Network
> Environment:
System : OpenBSD 7.3
Details : OpenBSD 7.3 (GENERIC.MP) #3: Tue Jul 25 08:20:26 MDT 2023
Architecture: OpenBSD.amd64
Machine : amd64
> Description:
When attempting a PXE-boot install of OpenBSD 7.3 (or 7.2) on two
recent Protectli models, VP4650 and VP2420 (both with Intel I225-V
2.5Gbps NICs), the target machine loads the auto_install (BOOTX64.EFI)
file over tftp from the pxeboot server, and brings up the usual OpenBSD
boot> prompt, but then fails to load the OpenBSD install kernel via tftp
from the server:
probing: pc0 mem[636K 1883M 16K 7M 5M 76K 172K 700K 6M 5M 2060M]
disk: hd0* hd1* hd2* hd3* hd4*
>> OpenBSD/amd64 BOOTX64 3.63
boot>
cannot open tftp:/etc/random.seed: No such file or directory
booting tftp:/bsd: open tftp:/bsd: No such file or directory
failed(2). will try /bsd
boot> boot bsd.rd
cannot open tftp:/etc/random.seed: No such file or directory
booting tftp:bsd.rd: open tftp:bsd.rd: No such file or directory
failed(2). will try /bsd
boot>
The 'bsd' file it's looking for definitely does exist on my pxeboot
server, in the tftpd root dirirectory where it's supposed to be. I've
tried with 'bsd' symlinked to 'bsd.rd' on the server, and also with
'bsd' as an actual copy of bsd.rd (just in case symlinking was the
problem) and got the same result. I know the pxeboot/tftp server is
working, because the Protectlis are getting DHCP and loading the
auto_install (BOOTX64.EFI) file via tftp from the server. The server's
dhcpd.conf follows the example from
https://man.openbsd.org/man8/amd64/pxeboot.8 and I've done many
successful PXE installs of OpenBSD onto other machines using this same
server -- though most of my installs have been onto legacy BIOS machines
rather than UEFI, and so I had the server serving 'pxeboot' as the
auto_install file instead of 'BOOTX64.EFI' for those past installs.
My best guess is that OpenBSD's BOOTX64.EFI is lacking support for
the 'igc' Intel I225-V 2.5Gbps NICs used in these Protectli models...
I.e. the Protectlis' built-in iPXE software is able to get DHCP from the
pxeboot server, and pull the BOOTX64.EFI file over tftp, but then once
OpenBSD's BOOTX64.EFI takes over, it's seemingly unable to communicate
over the network to pull down the OpenBSD install kernel.
I ran a tcpdump on the pxeboot server, which showed the Protectli's
built-in iPXE fetching the 'auto_install' file (which is a symlink to
'BOOTX64.EFI' on the server), followed by some UDP packets, but then no
more activity whatsoever on the tcpdump after the point when the
Protectli brings up the BOOTX64.EFI boot> prompt. I'm not seeing the
request for the bsd kernel on the server's tcpdump at all:
root@pixie/tftpboot 57 # tcpdump -ni em1
tcpdump: listening on em1, link-type EN10MB
12:53:27.004863 0.0.0.0.68 > 255.255.255.255.67: xid:0x531e7443
secs:4 [|bootp]
12:53:27.004976 fe80::6662:66ff:fe22:1627 > ff02::2: icmp6: router
solicitation
12:53:27.005186 arp who-has 10.0.201.25 tell 10.0.201.1
12:53:27.130855 fe80::6662:66ff:fe22:1627 > ff02::2: icmp6: router
solicitation
12:53:27.381297 fe80::6662:66ff:fe22:1627 > ff02::2: icmp6: router
solicitation
12:53:27.882242 fe80::6662:66ff:fe22:1627 > ff02::2: icmp6: router
solicitation
12:53:28.002547 0.0.0.0.68 > 255.255.255.255.67: xid:0x531e7443
secs:8 [|bootp]
12:53:28.002741 10.0.201.1.67 > 10.0.201.25.68: xid:0x531e7443
secs:4 Y:10.0.201.25 S:10.0.201.1 [|bootp] [tos 0x10]
12:53:28.884155 fe80::6662:66ff:fe22:1627 > ff02::2: icmp6: router
solicitation
12:53:30.006332 0.0.0.0.68 > 255.255.255.255.67: xid:0x531e7443
secs:18 [|bootp]
12:53:30.008661 10.0.201.1.67 > 10.0.201.25.68: xid:0x531e7443
secs:18 Y:10.0.201.25 S:10.0.201.1 [|bootp] [tos 0x10]
12:53:30.008803 arp who-has 10.0.201.25 tell 10.0.201.25
12:53:30.008836 10.0.201.1 > 10.0.201.25: icmp: echo request
12:53:30.008926 arp who-has 10.0.201.1 tell 10.0.201.25
12:53:30.008945 arp reply 10.0.201.1 is-at 00:0d:b9:49:b1:81
12:53:30.009048 10.0.201.25 > 10.0.201.1: icmp: echo reply
12:53:30.962189 10.0.201.25.10880 > 10.0.201.1.69: 42 RRQ
"auto_install"
12:53:30.962907 10.0.201.1.12471 > 10.0.201.25.10880: udp 28
12:53:30.963049 10.0.201.25.10880 > 10.0.201.1.12471: udp 4
12:53:30.963297 10.0.201.1.12471 > 10.0.201.25.10880: udp 1436
12:53:30.963441 10.0.201.25.10880 > 10.0.201.1.12471: udp 4
12:53:30.963581 10.0.201.1.12471 > 10.0.201.25.10880: udp 1436
12:53:30.963729 10.0.201.25.10880 > 10.0.201.1.12471: udp 4
# ...followed by a few more udp lines like the ones above, and then
nothing more after the Protectli brings up the BOOTX64.EFI boot> prompt.
I also tested serving the OpenBSD 7.2 (instead of 7.3) install
kernel + BOOTX64.EFI files (BOOTX64 v3.62 instead of v3.63), and got the
same result.
And finally, I tested installing OpenBSD 7.3 from a USB disk onto
the Protectli VP4650, and everything worked fine (including network
connectivity while booted from the OpenBSD installer kernel on the USB
disk). So the problem only affects PXE-booting.
Here are dmesg lines about the 'igc' network interfaces on the
Protectli VP4650 (after doing the successful OpenBSD install from a USB
disk):
rex1# dmesg | grep igc
igc0 at pci1 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4
queues, address 64:62:66:xx:xx:xx
igc1 at pci2 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4
queues, address 64:62:66:xx:xx:xx
igc2 at pci3 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4
queues, address 64:62:66:xx:xx:xx
igc3 at pci4 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4
queues, address 64:62:66:xx:xx:xx
igc4 at pci5 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4
queues, address 64:62:66:xx:xx:xx
igc5 at pci6 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4
queues, address 64:62:66:xx:xx:xx
Please let me know if there's anything else I should try or other
info I can provide, or if my theory that BOOTX64.EFI lacks support for
this NIC is correct.
> How-To-Repeat:
Attempt PXE-boot install of OpenBSD 7.3 onto a Protectli VP4650,
VP2420, or other hardware with Intel I225-V (rev 3) 2.5Gbps NIC ('igc'
driver)
> Fix: Unknown
