Hi, I have a baremetal pc on which i'm trying to get the onboard 2 sfp+ ports to work. I tried with a freebsd 13.2 install and freebsd 14 livecd. Also tested with ubuntu, where everything works fine with the same setup of cables etc. With freebsd the sfp+ ports are recognized, but not sure what is going on, although cables are connected, no matter what i try, i get: status: no carrier. I had some help & suggestions on IRC, I was told it could be a driver bug. I'm familiar with Linux but new to *bsd, so tried to collect relevant information in the hope to get it working or correct my mistakes.
full dmesg output from freebsd 14 live usb medium boot: https://pastebin.com/6UGzidXq partial dmesg output on freebsd 14 for sfp+ nics: Code: ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0 ax0: SFP detected: ax0: vendor: OEM ax0: part number: SFP-H10GB-CU5M ax0: revision level: 09 ax0: serial number: CSC231202820016 ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x7 ax1: xgbe_phy_sfp_detect: mod absent ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x2 ax1: SFP detected: ax1: vendor: Walsun ax1: part number: HXSX-ATRI-1 ax1: revision level: 1.0 ax1: serial number: H230S014913 ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0 I can configure ax0 or ax1 fine and bring them also up but it will show 'no carrier' in ifconfig Code: ax0: flags=8867 metric 0 mtu 1500 description: OPT1 (opt1) options=4e0032b ether d0:63:b4:05:47:43 inet 10.133.120.125 netmask 0xffffff00 broadcast 10.133.120.255 media: Ethernet autoselect status: no carrier nd6 options=29 ax1: flags=8867 metric 0 mtu 1500 description: LANSFP (opt1) options=4e0032b ether d0:63:b4:05:47:44 media: Ethernet autoselect status: no carrier nd6 options=29 when I check with tcpdump on the baremetal pc and ping it from my other arch linux on the local lan i see this: Code: tcpdump --interface ax0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ax0, link-type EN10MB (Ethernet), capture size 262144 bytes 21:28:10.551283 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:11.287919 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:11.821977 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 21:28:12.287909 LLDP, length 69 21:28:13.284656 LLDP, length 69 21:28:14.281404 LLDP, length 69 21:28:15.278147 LLDP, length 69 21:28:15.586787 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 21:28:16.053341 IP 10.133.120.5.38313 > 239.255.255.250.ssdp: UDP, length 101 21:28:16.053354 IP 10.133.120.5.33435 > 239.255.255.250.ssdp: UDP, length 101 21:28:16.053363 IP 10.133.120.3.59149 > 239.255.255.250.ssdp: UDP, length 101 21:28:16.053382 IP 10.133.120.4.33015 > 239.255.255.250.ssdp: UDP, length 101 21:28:16053388 IP 10.133.120.3.51253 > 239.255.255.250.ssdp: UDP, length 101 21:28:16.053391 IP 10.133.120.4.44274 > 239.255.255.250.ssdp: UDP, length 101 21:28:20.901812 IP 10.133.120.3.59695 > 239.255.255.250.ssdp: UDP, length 129 21:28:20.901941 IP 10.133.120.4.48273 > 239.255.255.250.ssdp: UDP, length 129 21:28:20.902080 IP 10.133.120.5.39369 > 239.255.255.250.ssdp: UDP, length 129 21:28:21.102922 IP 10.133.120.3.59695 > 239.255.255.250.ssdp: UDP, length 129 21:28:21.103053 IP 10.133.120.4.48273 > 239.255.255.250.ssdp: UDP, length 129 21:28:21.103188 IP 10.133.120.5.39369 > 239.255.255.250.ssdp: UDP, length 129 21:28:21.302014 IP 10.133.120.3.59695 > 239.255.255.250.ssdp: UDP, length 129 21:28:21.302027 IP 10.133.120.4.48273 > 239.255.255.250.ssdp: UDP, length 129 21:28:21.302152 IP 10.133.120.5.39369 > 239.255.255.250.ssdp: UDP, length 129 21:28:23.113808 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:23.821144 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 21:28:24.037080 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:26.053695 IP 10.133.120.3.34175 > 239.255.255.250.ssdp: UDP, length 101 21:28:26.053703 IP 10.133.120.5.40299 > 239.255.255.250.ssdp: UDP, length 101 21:28:26.053713 IP 10.133.120.4.42403 > 239.255.255.250.ssdp: UDP, length 101 21:28:27.473666 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 21:28:34.769972 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 21:28:35.573080 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:36.053367 IP 10.133.120.3.48149 > 239.255255.250.ssdp: UDP, length 101 21:28:36.053383 IP 10.133.120.4.59157 > 239.255.255.250.ssdp: UDP, length 101 21:28:36.053391 IP 10.133.120.5.52090 > 239.255.255.250.ssdp: UDP, length 101 21:28:36.166390 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:36.820373 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 21:28:37.206600 LLDP, length 69 21:28:38203331 LLDP, length 69 21:28:39.200100 LLDP, length 69 21:28:40.196843 LLDP, length 69 21:28:40.943626 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 21:28:46.053949 IP 10.133.120.3.60867 > 239.255.255.250.ssdp: UDP, length 101 21:28:46.053966 IP 10.133.120.4.38949 > 239.255.255.250.ssdp: UDP, length 101 21:28:46.053970 IP 10.133.120.5.47332 > 239.255.255.250.ssdp: UDP, length 101 21:28:46.053973 IP 10.133.120.3.51494 > 239.255.255.250.ssdp: UDP, length 101 21:28:46.053976 IP 10.133.120.4.55277 > 239.255.255.250.ssdp: UDP, length 101 21:28:46.053984 IP 10.133.120.5.38689 > 239.255.255.250.ssdp: UDP, length 101 21:28:48.142246 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:48.592245 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88 21:28:48.819592 IP6 fe80::3efd:feff:fe9e:cb7c > ff02::2: ICMP6, router solicitation, length 16 ^C 51 packets captured 51 packets received by filter 0 packets dropped by kernel tested a few thing, below snippet includes ifconfig ouput, then I bring the sfp+ devices up, assign static IPs and try to ping the other machine to which it is connected and in between ifconfig outputs: https://pastebin.com/SgztQzjE Here some output from working ubuntu in case it helps in some way lspci output from working ubuntu Code: 0a:00.2 Ethernet controller: Advanced Micro Devices, Inc. [AMD] Device 1458 Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1458 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 16GT/s, Width x16 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=7 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00001000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Kernel driver in use: amd-xgbe Kernel modules: amd_xgbe 0a:00.3 Ethernet controller: Advanced Micro Devices, Inc. [AMD] Device 1458 Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1458 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 16GT/s, Width x16 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=7 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00001000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Kernel driver in use: amd-xgbe Kernel modules: amd_xgbe ethtool output on ubuntu Code: Settings for enp10s0f2: Supported ports: [ TP ] Supported link modes: 10000baseCR/Full Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 10000baseCR/Full Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Auto-negotiation: off Port: None PHYAD: 0 Transceiver: internal Current message level: 0x00000034 (52) link ifdown ifup Link detected: yes Settings for enp10s0f3: Supported ports: [ FIBRE ] Supported link modes: 10000baseSR/Full Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 10000baseSR/Full Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Auto-negotiation: off Port: None PHYAD: 0 Transceiver: internal Current message level: 0x00000034 (52) link ifdown ifup Link detected: yes ethtool -i enp10s0f2 driver: amd-xgbe version: 6.2.0-39-generic firmware-version: 19.118.48 expansion-rom-version: bus-info: 0000:0a:00.2 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ethtool -i enp10s0f3 driver: amd-xgbe version: 6.2.0-39-generic firmware-version: 19.118.48 expansion-rom-version: bus-info: 0000:0a:00.3 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ifconfig entries for sfp+ nic's on ubuntu, I setup a static network to test connectivity with another local arch linux server on local lan: Code: enp10s0f2: flags=4163 mtu 1500 inet 10.133.120.11 netmask 255.255.255.0 broadcast 10.133.120.255 inet6 fe80::d263:b4ff:fe05:4743 prefixlen 64 scopeid 0x20 ether d0:63:b4:05:47:43 txqueuelen 1000 (Ethernet) RX packets 2 bytes 128 (128.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 59 bytes 4448 (4.4 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 43 enp10s0f3: flags=4163 mtu 1500 inet 10.133.120.12 netmask 255.255.255.0 broadcast 10.133.120.255 inet6 fe80::d263:b4ff:fe05:4744 prefixlen 64 scopeid 0x20 ether d0:63:b4:05:47:44 txqueuelen 1000 (Ethernet) RX packets 1352 bytes 92127 (92.1 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 47 bytes 3975 (3.9 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 58 ubuntu lshw network devices info: Code: lshw -class network *-network description: Wireless interface product: Wi-Fi 6 AX210/AX211/AX411 160MHz vendor: Intel Corporation physical id: 0 bus info: pci@0000:03:00.0 logical name: wlp3s0 version: 1a serial: 2c:33:58:f3:cc:4e width: 64 bits clock: 33MHz capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical wireless configuration: broadcast=yes driver=iwlwifi driverversion=6.2.0-39-generic firmware=72.a764baac.0 ty-a0-gf-a0-72.uc latency=0 link=no multicast=yes wireless=IEEE 802.11 resources: irq:113 memory:fcd00000-fcd03fff *-network description: Ethernet interface product: Intel Corporation vendor: Intel Corporation physical id: 0 bus info: pci@0000:04:00.0 logical name: enp4s0 version: 04 serial: d0:63:b4:05:47:3f capacity: 1Gbit/s width: 32 bits clock: 33MHz capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=igc driverversion=6.2.0-39-generic firmware=2022:889d latency=0 link=no multicast=yes port=twisted pair resources: irq:37 memory:fc900000-fc9fffff memory:fca00000-fca03fff memory:fc800000-fc8fffff *-network description: Ethernet interface product: Intel Corporation vendor: Intel Corporation physical id: 0 bus info: pci@0000:05:00.0 logical name: enp5s0 version: 04 serial: d0:63:b4:05:47:40 capacity: 1Gbit/s width: 32 bits clock: 33MHz capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=igc driverversion=6.2.0-39-generic firmware=2022:889d latency=0 link=no multicast=yes port=twisted pair resources: irq:37 memory:fc600000-fc6fffff memory:fc700000-fc703fff memory:fc500000-fc5fffff *-network description: Ethernet interface product: Intel Corporation vendor: Intel Corporation physical id: 0 bus info: pci@0000:06:00.0 logical name: enp6s0 version: 04 serial: d0:63:b4:05:47:41 capacity: 1Gbit/s width: 32 bits clock: 33MHz capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=igc driverversion=6.2.0-39-generic firmware=2022:889d latency=0 link=no multicast=yes port=twisted pair resources: irq:78 memory:fc300000-fc3fffff memory:fc400000-fc403fff memory:fc200000-fc2fffff *-network description: Ethernet interface product: Intel Corporation vendor: Intel Corporation physical id: 0 bus info: pci@0000:07:00.0 logical name: enp7s0 version: 04 serial: d0:63:b4:05:47:42 capacity: 1Gbit/s width: 32 bits clock: 33MHz capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=igc driverversion=6.2.0-39-generic firmware=2022:889d latency=0 link=no multicast=yes port=twisted pair resources: irq:78 memory:fc000000-fc0fffff memory:fc100000-fc103fff memory:fbf00000-fbffffff *-network:0 description: Ethernet interface product: Advanced Micro Devices, Inc. [AMD] vendor: Advanced Micro Devices, Inc. [AMD] physical id: 0.2 bus info: pci@0000:0a:00.2 logical name: enp10s0f2 version: 00 serial: d0:63:b4:05:47:43 size: 10Gbit/s width: 64 bits clock: 33MHz capabilities: pm pciexpress msi msix bus_master cap_list ethernet physical tp configuration: autonegotiation=off broadcast=yes driver=amd-xgbe driverversion=6.2.0-39-generic duplex=full firmware=19.118.48 ip=10.133.120.11 latency=0 link=yes multicast=yes speed=10Gbit/s resources: irq:42 memory:fcb60000-fcb7ffff memory:fcb40000-fcb5ffff memory:fcb82000-fcb83fff *-network:1 description: Ethernet interface product: Advanced Micro Devices, Inc. [AMD] vendor: Advanced Micro Devices, Inc. [AMD] physical id: 0.3 bus info: pci@0000:0a:00.3 logical name: enp10s0f3 version: 00 serial: d0:63:b4:05:47:44 size: 10Gbit/s width: 64 bits clock: 33MHz capabilities: pm pciexpress msi msix bus_master cap_list ethernet physical fibre configuration: autonegotiation=off broadcast=yes driver=amd-xgbe driverversion=6.2.0-39-generic duplex=full firmware=19.118.48 ip=10.133.120.12 latency=0 link=yes multicast=yes speed=10Gbit/s resources: irq:50 memory:fcb20000-fcb3ffff memory:fcb00000-fcb1ffff memory:fcb80000-fcb81fff
