Re: PCI passthrough not working with bhyve and NetBSD
I think I know the cause of the problem. I posted this same question on the FreeBSD virtualization list, and received this reply which sounds plausible: You have to use MSI interrupts for pass-through devices for bhyve. Those errors look as if the guests are trying to use legacy INTx interrupts. I found this paper explaining the problem with the lack of MSI-X support: https://www.netbsd.org/gallery/presentations/ozaki-r/2015_AsiaBSDCon/ABC2015-P4C-paper.pdf The paper is three years old. What is the state of MSI in NetBSD now? Is my problem due to lack of MSI support, or is the support there but not used? On Sun, Jul 15, 2018, 21:00 Farid Joubbi wrote: > I have current installed on it now. > It has both the Intel and Broadcom NICs with passthrough as well as the > virtual NIC from bhyve. > The virtual NIC works nice. I used it to fetch the installation sets. > > From dmesg: > [ 1.0448813] virtio1 at pci0 dev 5 function 0 > [ 1.0448813] virtio1: Virtio Network Device (rev. 0x00) > [ 1.0448813] vioif0 at virtio1: Features: > 0x11010020 > [ 1.0448813] vioif0: Ethernet address 58:9c:fc:0b:96:50 > [ 1.0448813] virtio1: config interrupting at msix0 vec 0 > [ 1.0448813] virtio1: queues interrupting at msix0 vec 1 > [ 1.0448813] wm0 at pci0 dev 6 function 0: 82576 quad-1000BaseT Ethernet > (rev. 0x01) > [ 1.0448813] wm0: interrupting at msi1 vec 0 > [ 1.0448813] wm0: Ethernet address 00:1b:21:84:e9:ed > [ 1.0448813] igphy0 at wm0 phy 1: i82566 10/100/1000 media interface, > rev. 1 > [ 1.0448813] igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > 1000baseT, 1000baseT-FDX, auto > [ 1.0448813] bge0 at pci0 dev 7 function 0: Broadcom BCM5720 Gigabit > Ethernet > [ 1.0448813] bge0: APE firmware NCSI 1.4.12.0 > [ 1.0448813] bge0: interrupting at msi2 vec 0 > [ 1.0448813] bge0: HW config 002b1094, 6014, 0002aa38, > 000c > [ 1.0448813] bge0: ASIC BCM5720 A0 (0x572), Ethernet address > 3c:a8:2a:e4:bc:eb > [ 1.0448813] bge0: no PHY found! > [ 1.0448813] pcib0 at pci0 dev 31 function 0: vendor 8086 product 7000 > (rev. 0x00) > [ 1.0448813] isa0 at pcib0 > [ 1.0448813] com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo > [ 1.0448813] com0: console > [ 1.0448813] com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo > [ 1.5402764] WARNING: 1 error while detecting hardware; check system log. > > > The intel NIC is connected at the moment. > It should receive an address with DHCP. > > localhost# ifconfig > vioif0: flags=0x8943 mtu > 1500 > ec_capabilities=1 > ec_enabled=0 > address: 58:9c:fc:0b:96:50 > inet6 fe80::5a9c:fcff:fe0b:9650%vioif0/64 flags 0x0 scopeid 0x1 > wm0: flags=0x8843 mtu 1500 > capabilities=7ff80 > capabilities=7ff80 > capabilities=7ff80 > enabled=0 > ec_capabilities=7 > ec_enabled=0 > address: 00:1b:21:84:e9:ed > media: Ethernet autoselect (1000baseT > full-duplex,flowcontrol,rxpause,txpause) > status: active > inet 169.254.58.45/16 broadcast 169.254.255.255 flags 0x0 > inet6 fe80::220e:9ccf:9196:97d1%wm0/64 flags 0x0 scopeid 0x2 > bge0: flags=0x8802 mtu 1500 > > capabilities=3f80 > capabilities=3f80 > enabled=0 > ec_capabilities=7 > ec_enabled=0 > address: 3c:a8:2a:e4:bc:eb > media: Ethernet manual (none) > lo0: flags=0x8049 mtu 33624 > inet 127.0.0.1/8 flags 0x0 > inet6 ::1/128 flags 0x20 > inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x4 > localhost# > > rc.conf: > dhcpcd=YES > dhcpcd_flags="-qM wm0" > > Any help would be much appreciated. > If I'm able to get one of the physical NICs to work somehow, I could give > anyone here willing to help debug SSH access to the server. > > > On Sun, Jul 15, 2018 at 1:49 PM wrote: > >> "Farid Joubbi" wrote: >> => I tried the install CD for the latest HEAD-201807121210Z. >> => It can see both the NICs, but they don't work. >> => Trying to configure them with DHCP, it looks as if the NIC is sending a >> => DISCOVERY but timing out never receiving anything back (I doubt that >> it is >> => actually sending anything on the wire). >> => Then it constantly throws this on the console: "wm0: device timeout >> (lost >> => interrupt)". >> >>That sounds like the kernel missing interrupts from the NIC, in which >> case it may actually be sending out the requests but failing to see the >> replies. Perhaps posting the dmesg, or at least the lines about the >> NICs, could help. >> >>
Re: PCI passthrough not working with bhyve and NetBSD
I tried the install CD for the latest HEAD-201807121210Z. It can see both the NICs, but they don't work. Trying to configure them with DHCP, it looks as if the NIC is sending a DISCOVERY but timing out never receiving anything back (I doubt that it is actually sending anything on the wire). Then it constantly throws this on the console: "wm0: device timeout (lost interrupt)". On Sat, Jul 14, 2018 at 8:34 PM Farid Joubbi wrote: > I have a working configuration for NetBSD on bhyve. It's not a problem > with vm-bhyve https://github.com/churchers/vm-bhyve > > NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve, as I > wrote earlier. > It seems to run perfectly fine other than that. > The "normal" vioif0 works fine. > > I tried NetBSD 8.0 RC2. > It finds both NICs! All of a sudden I feel hope ;-) > Unfortunately I can only get link-local addresses on both NICs. > And I also get this printed on the console every now and then: "bge0: > watchdog timeout -- resetting". > > I will try and get it to NetBSD-current and see. > > > On Tue, Jul 10, 2018 at 5:30 AM Travis Paul wrote: > >> >> >> > On 9 Jul 2018, at 5:58 AM, Farid Joubbi wrote: >> > >> > Thanks for the reply. >> > After reading it, I realize that the learning curve for me to >> understand what is going on is a bit too steep. >> > I know only some basic C programming from university courses several >> years ago. >> > This kind of learning was not what I had in mind when I figured that I >> want to run a new NetBSD installation this summer... ;-( >> >> FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a few >> NetBSD 7 and 8 VMs on FreeBSD 11. >> >> I can share some commands to get a NetBSD system up if you're interested >> but I'm not using PCI passthru so I can't assist there. >> >> Best, >> Travis >> >
Re: PCI passthrough not working with bhyve and NetBSD
I have a working configuration for NetBSD on bhyve. It's not a problem with vm-bhyve https://github.com/churchers/vm-bhyve NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve, as I wrote earlier. It seems to run perfectly fine other than that. The "normal" vioif0 works fine. I tried NetBSD 8.0 RC2. It finds both NICs! All of a sudden I feel hope ;-) Unfortunately I can only get link-local addresses on both NICs. And I also get this printed on the console every now and then: "bge0: watchdog timeout -- resetting". I will try and get it to NetBSD-current and see. On Tue, Jul 10, 2018 at 5:30 AM Travis Paul wrote: > > > > On 9 Jul 2018, at 5:58 AM, Farid Joubbi wrote: > > > > Thanks for the reply. > > After reading it, I realize that the learning curve for me to understand > what is going on is a bit too steep. > > I know only some basic C programming from university courses several > years ago. > > This kind of learning was not what I had in mind when I figured that I > want to run a new NetBSD installation this summer... ;-( > > FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a few > NetBSD 7 and 8 VMs on FreeBSD 11. > > I can share some commands to get a NetBSD system up if you're interested > but I'm not using PCI passthru so I can't assist there. > > Best, > Travis >
Re: PCI passthrough not working with bhyve and NetBSD
Thanks for the reply. After reading it, I realize that the learning curve for me to understand what is going on is a bit too steep. I know only some basic C programming from university courses several years ago. This kind of learning was not what I had in mind when I figured that I want to run a new NetBSD installation this summer... ;-( I had no idea that it was so uncommon to use the BSD hypervisor to run the two other big BSD's beside FreeBSD until I started googling for answers to my problem. On Sun, Jul 8, 2018 at 4:29 PM Greg Troxel wrote: > > Farid Joubbi writes: > > > I have a server running FreeBSD 11.1-RELEASE-p11. > > I would like to run NetBSD using bhyve on that. > > That's interesting; there has been little discussion of running NetBSD > under bhyve so far. > > > I have configured passthrough of two different PCI Express network > > interface cards. I can't get neither OpenBSD nor NetBSD to work with > these > > NICs. > > > > I get this in dmesg when booting the NetBSD install: > > > > wm0 at pci0 dev 5 function 0: 82576 quad-1000BaseT Ethernet (rev. 0x01) > > pci_intr_map: no mapping for pin B (line=ff) > > wm0: unable to map interrupt > > bge0 at pci0 dev 6 function 0: Broadcom BCM5720 Gigabit Ethernet > > pci_intr_map: no mapping for pin B (line=ff) > > bge0: couldn't map interrupt > > It seems that while setting up interrupts the wm driver and code it > calls runs into a situation that it can't handle, either because the > situation is buggy or because the code doesn't handle something which is > legitimate (per the PCI spec) but unusual. > > I would read the code in the wm driver and then find pci_intr_map, and > see what debugging variables are defined, and build a kernel with them > turned on. Then I would either figure out how to run kgdb or add > printfs. > > I would also look at the demsg from FreeBSD and CentOS, and turn on any > verbosity you can, and understand how the interrupt is mapped there. > > > With OpenBSD it's the same error. > > I have no problems running FreeBSD or CentOS with the exact same setup. > > > > I realize that the problem is probably in bhyve and not in NetBSD, but is > > Could be either way. But you'll only know when you find out what's > wrong and read the specs. > > > there a (easy) way to debug this from NetBSD in order to figure out what > > the actual problem is? > > Why aren't FreeBSD and CentOS having the same issue? > > My guess is that the bhyve code, as it was developed, was made to be > enough like real hardware to get FreeBSD and Linux to work, and nobody > has debugged the other situations. >
PCI passthrough not working with bhyve and NetBSD
I have a server running FreeBSD 11.1-RELEASE-p11. I would like to run NetBSD using bhyve on that. I have configured passthrough of two different PCI Express network interface cards. I can't get neither OpenBSD nor NetBSD to work with these NICs. I get this in dmesg when booting the NetBSD install: wm0 at pci0 dev 5 function 0: 82576 quad-1000BaseT Ethernet (rev. 0x01) pci_intr_map: no mapping for pin B (line=ff) wm0: unable to map interrupt bge0 at pci0 dev 6 function 0: Broadcom BCM5720 Gigabit Ethernet pci_intr_map: no mapping for pin B (line=ff) bge0: couldn't map interrupt With OpenBSD it's the same error. I have no problems running FreeBSD or CentOS with the exact same setup. I realize that the problem is probably in bhyve and not in NetBSD, but is there a (easy) way to debug this from NetBSD in order to figure out what the actual problem is? Why aren't FreeBSD and CentOS having the same issue? Thanks.
