Re: PCI passthrough not working with bhyve and NetBSD
MSI and MSI-X themselves are supported on amd64, though not every driver use it. Both wm(4) and bge(4) however do support it and use it, so should work. Maybe there is either some misunderstanding about the MSI support (e.g. our PCI subsystem doesn't detect it), or the PCI probe code tries to allocate (and fails) the INTx interrupt regardless of MSI. Jaromir 2018-08-10 0:34 GMT+02:00 Farid Joubbi : > I think I know the cause of the problem. > I posted this same question on the FreeBSD virtualization list, and received > this reply which sounds plausible: > > You have to use MSI interrupts for pass-through devices for bhyve. Those > errors look as if the guests are trying to use legacy INTx interrupts. > > I found this paper explaining the problem with the lack of MSI-X support: > > https://www.netbsd.org/gallery/presentations/ozaki-r/2015_AsiaBSDCon/ABC2015-P4C-paper.pdf > > The paper is three years old. What is the state of MSI in NetBSD now? > Is my problem due to lack of MSI support, or is the support there but not > used? > > > On Sun, Jul 15, 2018, 21:00 Farid Joubbi wrote: >> >> I have current installed on it now. >> It has both the Intel and Broadcom NICs with passthrough as well as the >> virtual NIC from bhyve. >> The virtual NIC works nice. I used it to fetch the installation sets. >> >> From dmesg: >> [ 1.0448813] virtio1 at pci0 dev 5 function 0 >> [ 1.0448813] virtio1: Virtio Network Device (rev. 0x00) >> [ 1.0448813] vioif0 at virtio1: Features: >> 0x11010020 >> [ 1.0448813] vioif0: Ethernet address 58:9c:fc:0b:96:50 >> [ 1.0448813] virtio1: config interrupting at msix0 vec 0 >> [ 1.0448813] virtio1: queues interrupting at msix0 vec 1 >> [ 1.0448813] wm0 at pci0 dev 6 function 0: 82576 quad-1000BaseT Ethernet >> (rev. 0x01) >> [ 1.0448813] wm0: interrupting at msi1 vec 0 >> [ 1.0448813] wm0: Ethernet address 00:1b:21:84:e9:ed >> [ 1.0448813] igphy0 at wm0 phy 1: i82566 10/100/1000 media interface, >> rev. 1 >> [ 1.0448813] igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, >> 1000baseT, 1000baseT-FDX, auto >> [ 1.0448813] bge0 at pci0 dev 7 function 0: Broadcom BCM5720 Gigabit >> Ethernet >> [ 1.0448813] bge0: APE firmware NCSI 1.4.12.0 >> [ 1.0448813] bge0: interrupting at msi2 vec 0 >> [ 1.0448813] bge0: HW config 002b1094, 6014, 0002aa38, >> 000c >> [ 1.0448813] bge0: ASIC BCM5720 A0 (0x572), Ethernet address >> 3c:a8:2a:e4:bc:eb >> [ 1.0448813] bge0: no PHY found! >> [ 1.0448813] pcib0 at pci0 dev 31 function 0: vendor 8086 product 7000 >> (rev. 0x00) >> [ 1.0448813] isa0 at pcib0 >> [ 1.0448813] com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo >> [ 1.0448813] com0: console >> [ 1.0448813] com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo >> [ 1.5402764] WARNING: 1 error while detecting hardware; check system >> log. >> >> >> The intel NIC is connected at the moment. >> It should receive an address with DHCP. >> >> localhost# ifconfig >> vioif0: flags=0x8943 mtu >> 1500 >> ec_capabilities=1 >> ec_enabled=0 >> address: 58:9c:fc:0b:96:50 >> inet6 fe80::5a9c:fcff:fe0b:9650%vioif0/64 flags 0x0 scopeid 0x1 >> wm0: flags=0x8843 mtu 1500 >> capabilities=7ff80 >> >> capabilities=7ff80 >> capabilities=7ff80 >> enabled=0 >> ec_capabilities=7 >> ec_enabled=0 >> address: 00:1b:21:84:e9:ed >> media: Ethernet autoselect (1000baseT >> full-duplex,flowcontrol,rxpause,txpause) >> status: active >> inet 169.254.58.45/16 broadcast 169.254.255.255 flags 0x0 >> inet6 fe80::220e:9ccf:9196:97d1%wm0/64 flags 0x0 scopeid 0x2 >> bge0: flags=0x8802 mtu 1500 >> >> capabilities=3f80 >> capabilities=3f80 >> enabled=0 >> ec_capabilities=7 >> ec_enabled=0 >> address: 3c:a8:2a:e4:bc:eb >> media: Ethernet manual (none) >> lo0: flags=0x8049 mtu 33624 >> inet 127.0.0.1/8 flags 0x0 >> inet6 ::1/128 flags 0x20 >> inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x4 >> localhost# >> >> rc.conf: >> dhcpcd=YES >> dhcpcd_flags="-qM wm0" >> >> Any help would be much appreciated. >> If I'm able to get one of the physical NICs to work somehow, I could give >> anyone here willing to help debug SSH access to the server. >> >> >> On Sun, Jul 15, 2018 at 1:49 PM wrote: >>> >>> "Farid Joubbi" wrote: >>> => I tried the install CD for the latest HEAD-201807121210Z. >>> => It can see both the NICs, but they don't work. >>> => Trying to configure them with DHCP, it looks as if the NIC is sending >>> a >>> => DISCOVERY but timing out never receiving anything back (I doubt that >>> it is >>> => actually sending anything on the wire). >>> => Then it constantly throws this on the console: "wm0: device timeout >>> (lost >>> => interrupt)". >>> >>>That sounds like the kernel missing interrupts from the NIC, in which >>> case it may actually be sending out the requests but
Re: PCI passthrough not working with bhyve and NetBSD
I think I know the cause of the problem. I posted this same question on the FreeBSD virtualization list, and received this reply which sounds plausible: You have to use MSI interrupts for pass-through devices for bhyve. Those errors look as if the guests are trying to use legacy INTx interrupts. I found this paper explaining the problem with the lack of MSI-X support: https://www.netbsd.org/gallery/presentations/ozaki-r/2015_AsiaBSDCon/ABC2015-P4C-paper.pdf The paper is three years old. What is the state of MSI in NetBSD now? Is my problem due to lack of MSI support, or is the support there but not used? On Sun, Jul 15, 2018, 21:00 Farid Joubbi wrote: > I have current installed on it now. > It has both the Intel and Broadcom NICs with passthrough as well as the > virtual NIC from bhyve. > The virtual NIC works nice. I used it to fetch the installation sets. > > From dmesg: > [ 1.0448813] virtio1 at pci0 dev 5 function 0 > [ 1.0448813] virtio1: Virtio Network Device (rev. 0x00) > [ 1.0448813] vioif0 at virtio1: Features: > 0x11010020 > [ 1.0448813] vioif0: Ethernet address 58:9c:fc:0b:96:50 > [ 1.0448813] virtio1: config interrupting at msix0 vec 0 > [ 1.0448813] virtio1: queues interrupting at msix0 vec 1 > [ 1.0448813] wm0 at pci0 dev 6 function 0: 82576 quad-1000BaseT Ethernet > (rev. 0x01) > [ 1.0448813] wm0: interrupting at msi1 vec 0 > [ 1.0448813] wm0: Ethernet address 00:1b:21:84:e9:ed > [ 1.0448813] igphy0 at wm0 phy 1: i82566 10/100/1000 media interface, > rev. 1 > [ 1.0448813] igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > 1000baseT, 1000baseT-FDX, auto > [ 1.0448813] bge0 at pci0 dev 7 function 0: Broadcom BCM5720 Gigabit > Ethernet > [ 1.0448813] bge0: APE firmware NCSI 1.4.12.0 > [ 1.0448813] bge0: interrupting at msi2 vec 0 > [ 1.0448813] bge0: HW config 002b1094, 6014, 0002aa38, > 000c > [ 1.0448813] bge0: ASIC BCM5720 A0 (0x572), Ethernet address > 3c:a8:2a:e4:bc:eb > [ 1.0448813] bge0: no PHY found! > [ 1.0448813] pcib0 at pci0 dev 31 function 0: vendor 8086 product 7000 > (rev. 0x00) > [ 1.0448813] isa0 at pcib0 > [ 1.0448813] com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo > [ 1.0448813] com0: console > [ 1.0448813] com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo > [ 1.5402764] WARNING: 1 error while detecting hardware; check system log. > > > The intel NIC is connected at the moment. > It should receive an address with DHCP. > > localhost# ifconfig > vioif0: flags=0x8943 mtu > 1500 > ec_capabilities=1 > ec_enabled=0 > address: 58:9c:fc:0b:96:50 > inet6 fe80::5a9c:fcff:fe0b:9650%vioif0/64 flags 0x0 scopeid 0x1 > wm0: flags=0x8843 mtu 1500 > capabilities=7ff80 > capabilities=7ff80 > capabilities=7ff80 > enabled=0 > ec_capabilities=7 > ec_enabled=0 > address: 00:1b:21:84:e9:ed > media: Ethernet autoselect (1000baseT > full-duplex,flowcontrol,rxpause,txpause) > status: active > inet 169.254.58.45/16 broadcast 169.254.255.255 flags 0x0 > inet6 fe80::220e:9ccf:9196:97d1%wm0/64 flags 0x0 scopeid 0x2 > bge0: flags=0x8802 mtu 1500 > > capabilities=3f80 > capabilities=3f80 > enabled=0 > ec_capabilities=7 > ec_enabled=0 > address: 3c:a8:2a:e4:bc:eb > media: Ethernet manual (none) > lo0: flags=0x8049 mtu 33624 > inet 127.0.0.1/8 flags 0x0 > inet6 ::1/128 flags 0x20 > inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x4 > localhost# > > rc.conf: > dhcpcd=YES > dhcpcd_flags="-qM wm0" > > Any help would be much appreciated. > If I'm able to get one of the physical NICs to work somehow, I could give > anyone here willing to help debug SSH access to the server. > > > On Sun, Jul 15, 2018 at 1:49 PM wrote: > >> "Farid Joubbi" wrote: >> => I tried the install CD for the latest HEAD-201807121210Z. >> => It can see both the NICs, but they don't work. >> => Trying to configure them with DHCP, it looks as if the NIC is sending a >> => DISCOVERY but timing out never receiving anything back (I doubt that >> it is >> => actually sending anything on the wire). >> => Then it constantly throws this on the console: "wm0: device timeout >> (lost >> => interrupt)". >> >>That sounds like the kernel missing interrupts from the NIC, in which >> case it may actually be sending out the requests but failing to see the >> replies. Perhaps posting the dmesg, or at least the lines about the >> NICs, could help. >> >> Gary Duzan >> >> >> => On Sat, Jul 14, 2018 at 8:34 PM Farid Joubbi >> wrote: >> => >> =>> I have a working configuration for NetBSD on bhyve. It's not a problem >> =>> with vm-bhyve https://github.com/churchers/vm-bhyve >> =>> >> =>> NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve, >> as I >> =>> wrote earlier. >> =>> It seems to run perfectly fine other than that.
Re: PCI passthrough not working with bhyve and NetBSD
"Farid Joubbi" wrote: => I tried the install CD for the latest HEAD-201807121210Z. => It can see both the NICs, but they don't work. => Trying to configure them with DHCP, it looks as if the NIC is sending a => DISCOVERY but timing out never receiving anything back (I doubt that it is => actually sending anything on the wire). => Then it constantly throws this on the console: "wm0: device timeout (lost => interrupt)". That sounds like the kernel missing interrupts from the NIC, in which case it may actually be sending out the requests but failing to see the replies. Perhaps posting the dmesg, or at least the lines about the NICs, could help. Gary Duzan => On Sat, Jul 14, 2018 at 8:34 PM Farid Joubbi wrote: => =>> I have a working configuration for NetBSD on bhyve. It's not a problem =>> with vm-bhyve https://github.com/churchers/vm-bhyve =>> =>> NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve, as I =>> wrote earlier. =>> It seems to run perfectly fine other than that. =>> The "normal" vioif0 works fine. =>> =>> I tried NetBSD 8.0 RC2. =>> It finds both NICs! All of a sudden I feel hope ;-) =>> Unfortunately I can only get link-local addresses on both NICs. =>> And I also get this printed on the console every now and then: "bge0: =>> watchdog timeout -- resetting". =>> =>> I will try and get it to NetBSD-current and see. =>> =>> =>> On Tue, Jul 10, 2018 at 5:30 AM Travis Paul wrote: =>> =>>> =>>> =>>> > On 9 Jul 2018, at 5:58 AM, Farid Joubbi wrote: =>>> > =>>> > Thanks for the reply. =>>> > After reading it, I realize that the learning curve for me to =>>> understand what is going on is a bit too steep. =>>> > I know only some basic C programming from university courses several =>>> years ago. =>>> > This kind of learning was not what I had in mind when I figured that =>>> I =>>> want to run a new NetBSD installation this summer... ;-( =>>> =>>> FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a few =>>> NetBSD 7 and 8 VMs on FreeBSD 11. =>>> =>>> I can share some commands to get a NetBSD system up if you're =>>> interested =>>> but I'm not using PCI passthru so I can't assist there. =>>> =>>> Best, =>>> Travis =>>> =>> =>
Re: PCI passthrough not working with bhyve and NetBSD
I tried the install CD for the latest HEAD-201807121210Z. It can see both the NICs, but they don't work. Trying to configure them with DHCP, it looks as if the NIC is sending a DISCOVERY but timing out never receiving anything back (I doubt that it is actually sending anything on the wire). Then it constantly throws this on the console: "wm0: device timeout (lost interrupt)". On Sat, Jul 14, 2018 at 8:34 PM Farid Joubbi wrote: > I have a working configuration for NetBSD on bhyve. It's not a problem > with vm-bhyve https://github.com/churchers/vm-bhyve > > NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve, as I > wrote earlier. > It seems to run perfectly fine other than that. > The "normal" vioif0 works fine. > > I tried NetBSD 8.0 RC2. > It finds both NICs! All of a sudden I feel hope ;-) > Unfortunately I can only get link-local addresses on both NICs. > And I also get this printed on the console every now and then: "bge0: > watchdog timeout -- resetting". > > I will try and get it to NetBSD-current and see. > > > On Tue, Jul 10, 2018 at 5:30 AM Travis Paul wrote: > >> >> >> > On 9 Jul 2018, at 5:58 AM, Farid Joubbi wrote: >> > >> > Thanks for the reply. >> > After reading it, I realize that the learning curve for me to >> understand what is going on is a bit too steep. >> > I know only some basic C programming from university courses several >> years ago. >> > This kind of learning was not what I had in mind when I figured that I >> want to run a new NetBSD installation this summer... ;-( >> >> FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a few >> NetBSD 7 and 8 VMs on FreeBSD 11. >> >> I can share some commands to get a NetBSD system up if you're interested >> but I'm not using PCI passthru so I can't assist there. >> >> Best, >> Travis >> >
Re: PCI passthrough not working with bhyve and NetBSD
I have a working configuration for NetBSD on bhyve. It's not a problem with vm-bhyve https://github.com/churchers/vm-bhyve NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve, as I wrote earlier. It seems to run perfectly fine other than that. The "normal" vioif0 works fine. I tried NetBSD 8.0 RC2. It finds both NICs! All of a sudden I feel hope ;-) Unfortunately I can only get link-local addresses on both NICs. And I also get this printed on the console every now and then: "bge0: watchdog timeout -- resetting". I will try and get it to NetBSD-current and see. On Tue, Jul 10, 2018 at 5:30 AM Travis Paul wrote: > > > > On 9 Jul 2018, at 5:58 AM, Farid Joubbi wrote: > > > > Thanks for the reply. > > After reading it, I realize that the learning curve for me to understand > what is going on is a bit too steep. > > I know only some basic C programming from university courses several > years ago. > > This kind of learning was not what I had in mind when I figured that I > want to run a new NetBSD installation this summer... ;-( > > FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a few > NetBSD 7 and 8 VMs on FreeBSD 11. > > I can share some commands to get a NetBSD system up if you're interested > but I'm not using PCI passthru so I can't assist there. > > Best, > Travis >
Re: PCI passthrough not working with bhyve and NetBSD
> On 9 Jul 2018, at 5:58 AM, Farid Joubbi wrote: > > Thanks for the reply. > After reading it, I realize that the learning curve for me to understand what > is going on is a bit too steep. > I know only some basic C programming from university courses several years > ago. > This kind of learning was not what I had in mind when I figured that I want > to run a new NetBSD installation this summer... ;-( FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a few NetBSD 7 and 8 VMs on FreeBSD 11. I can share some commands to get a NetBSD system up if you're interested but I'm not using PCI passthru so I can't assist there. Best, Travis signature.asc Description: Message signed with OpenPGP
Re: PCI passthrough not working with bhyve and NetBSD
On 8 July 2018 at 22:58, Farid Joubbi wrote: > Thanks for the reply. > After reading it, I realize that the learning curve for me to understand > what is going on is a bit too steep. > I know only some basic C programming from university courses several years > ago. > This kind of learning was not what I had in mind when I figured that I want > to run a new NetBSD installation this summer... ;-( > > I had no idea that it was so uncommon to use the BSD hypervisor to run the > two other big BSD's beside FreeBSD until I started googling for answers to > my problem. You might find more luck using VirtualBox under FreeBSD to run NetBSD https://www.freebsd.org/doc/handbook/virtualization-host-virtualbox.html If you are willing to try digging into bhyve a little more you are likely to find people willing to help both on the NetBSD and FreeBSD lists (particularly if you treat it as an opportunity to learn along the way and not get frustrated if it doesn't work out in the end :) I would also try booting a current NetBSD install image to see if it shows the same issue (can download from http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/ ) David
Re: PCI passthrough not working with bhyve and NetBSD
Thanks for the reply. After reading it, I realize that the learning curve for me to understand what is going on is a bit too steep. I know only some basic C programming from university courses several years ago. This kind of learning was not what I had in mind when I figured that I want to run a new NetBSD installation this summer... ;-( I had no idea that it was so uncommon to use the BSD hypervisor to run the two other big BSD's beside FreeBSD until I started googling for answers to my problem. On Sun, Jul 8, 2018 at 4:29 PM Greg Troxel wrote: > > Farid Joubbi writes: > > > I have a server running FreeBSD 11.1-RELEASE-p11. > > I would like to run NetBSD using bhyve on that. > > That's interesting; there has been little discussion of running NetBSD > under bhyve so far. > > > I have configured passthrough of two different PCI Express network > > interface cards. I can't get neither OpenBSD nor NetBSD to work with > these > > NICs. > > > > I get this in dmesg when booting the NetBSD install: > > > > wm0 at pci0 dev 5 function 0: 82576 quad-1000BaseT Ethernet (rev. 0x01) > > pci_intr_map: no mapping for pin B (line=ff) > > wm0: unable to map interrupt > > bge0 at pci0 dev 6 function 0: Broadcom BCM5720 Gigabit Ethernet > > pci_intr_map: no mapping for pin B (line=ff) > > bge0: couldn't map interrupt > > It seems that while setting up interrupts the wm driver and code it > calls runs into a situation that it can't handle, either because the > situation is buggy or because the code doesn't handle something which is > legitimate (per the PCI spec) but unusual. > > I would read the code in the wm driver and then find pci_intr_map, and > see what debugging variables are defined, and build a kernel with them > turned on. Then I would either figure out how to run kgdb or add > printfs. > > I would also look at the demsg from FreeBSD and CentOS, and turn on any > verbosity you can, and understand how the interrupt is mapped there. > > > With OpenBSD it's the same error. > > I have no problems running FreeBSD or CentOS with the exact same setup. > > > > I realize that the problem is probably in bhyve and not in NetBSD, but is > > Could be either way. But you'll only know when you find out what's > wrong and read the specs. > > > there a (easy) way to debug this from NetBSD in order to figure out what > > the actual problem is? > > Why aren't FreeBSD and CentOS having the same issue? > > My guess is that the bhyve code, as it was developed, was made to be > enough like real hardware to get FreeBSD and Linux to work, and nobody > has debugged the other situations. >
Re: PCI passthrough not working with bhyve and NetBSD
Farid Joubbi writes: > I have a server running FreeBSD 11.1-RELEASE-p11. > I would like to run NetBSD using bhyve on that. That's interesting; there has been little discussion of running NetBSD under bhyve so far. > I have configured passthrough of two different PCI Express network > interface cards. I can't get neither OpenBSD nor NetBSD to work with these > NICs. > > I get this in dmesg when booting the NetBSD install: > > wm0 at pci0 dev 5 function 0: 82576 quad-1000BaseT Ethernet (rev. 0x01) > pci_intr_map: no mapping for pin B (line=ff) > wm0: unable to map interrupt > bge0 at pci0 dev 6 function 0: Broadcom BCM5720 Gigabit Ethernet > pci_intr_map: no mapping for pin B (line=ff) > bge0: couldn't map interrupt It seems that while setting up interrupts the wm driver and code it calls runs into a situation that it can't handle, either because the situation is buggy or because the code doesn't handle something which is legitimate (per the PCI spec) but unusual. I would read the code in the wm driver and then find pci_intr_map, and see what debugging variables are defined, and build a kernel with them turned on. Then I would either figure out how to run kgdb or add printfs. I would also look at the demsg from FreeBSD and CentOS, and turn on any verbosity you can, and understand how the interrupt is mapped there. > With OpenBSD it's the same error. > I have no problems running FreeBSD or CentOS with the exact same setup. > > I realize that the problem is probably in bhyve and not in NetBSD, but is Could be either way. But you'll only know when you find out what's wrong and read the specs. > there a (easy) way to debug this from NetBSD in order to figure out what > the actual problem is? > Why aren't FreeBSD and CentOS having the same issue? My guess is that the bhyve code, as it was developed, was made to be enough like real hardware to get FreeBSD and Linux to work, and nobody has debugged the other situations. signature.asc Description: PGP signature
