Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Nikos Vassiliadis]

On 11/30/14 20:42, Neel Natu wrote:

this CPU does not support VT-d, which is needed for PCI passthru.



Indeed. Good catch, Neel should this be the case?


Definitely!

vmm.ko should probably not even attach the ppt driver to "pptdevs" if
an IOMMU is absent.


OK, the mystery is solved then.
Thank you all for your answers

PS:
I just started using bhyve and it's amazing to say the least! Amazing 
work people!

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Neel Natu]

Hi Nikos,

On Sun, Nov 30, 2014 at 7:37 AM, Nikos Vassiliadis  wrote:
> Hi,
>
> On 11/30/14 00:48, Craig Rodrigues wrote:
>>
>> I'm not sure, but according to this datasheet:
>>
>>
>> http://ark.intel.com/products/65700/Intel-Core-i3-3110M-Processor-3M-Cache-2_40-GHz
>>
>> this CPU does not support VT-d, which is needed for PCI passthru.
>
>
> Indeed. Good catch, Neel should this be the case?

Definitely!

vmm.ko should probably not even attach the ppt driver to "pptdevs" if
an IOMMU is absent.

best
Neel
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Neel Natu]

Hi Nikos,

On Sun, Nov 30, 2014 at 7:50 AM, Nikos Vassiliadis  wrote:
> Hi,
>
> On 11/30/14 02:37, Neel Natu wrote:
>>
>> The "Unsupported MSI-X configuration" referred to here is that bhyve
>> doesn't emulate the 'Pending Bit Array'.
>>
>> In most cases this is not relevant because the PBA and the MSI-X
>> tables are in different page frames. In this case the MSI-X tables are
>> emulated and the pending bit array page is passed through to the
>> guest.
>>
>> If the PBA and MSI-X tables are on the same page frame then you see
>> this error. You can confirm this by doing 'pciconf -lvbc pci0:2:0:0'.
>
>
> Here is the ouput:
>>
>> bge0@pci0:2:0:0:class=0x02 card=0x06471025 chip=0x16b514e4
>> rev=0x10 hdr=0x00
>> vendor = 'Broadcom Corporation'
>> device = 'NetLink BCM57785 Gigabit Ethernet PCIe'
>> class  = network
>> subclass   = ethernet
>> bar   [10] = type Prefetchable Memory, range 64, base 0xb343, size
>> 65536, enabled
>> bar   [18] = type Prefetchable Memory, range 64, base 0xb344, size
>> 65536, enabled
>> cap 01[48] = powerspec 3  supports D0 D3  current D0
>> cap 05[58] = MSI supports 8 messages, 64 bit enabled with 1 message
>> cap 11[a0] = MSI-X supports 5 messages
>>  Table in map 0x18[0x0], PBA in map 0x18[0x120]

Indeed, the MSI-X tables and PBA are both in the first page in the BAR.

>> cap 10[ac] = PCI-Express 2 endpoint max data 128(128) link x1(x1)
>>  speed 2.5(2.5) ASPM L1(L0s/L1)
>> ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
>> ecap 0003[13c] = Serial 1 208984c6b5ed
>> ecap 0004[150] = Power Budgeting 1
>> ecap 0002[160] = VC 1 max VC0
>
>
>
>
>> Having said that the drivers I have seen don't rely at all on the PBA.
>> So it may be possible that this works just fine with this patch:
>>
>> https://people.freebsd.org/~neel/patches/bhyve_ignore_unsupported_pba.patch
>>
>> If it does then I can add a tunable to bhyve to ignore this check.
>
>
> It seems a tunable can help.
>
> I tried the patch and now bhyve continues and indeed I see the bge device in
> my VM. What is interesting/strange is that the device is partly functioning.
> For example, it does detect a link change when I remove the cable but that's
> all. The kernel also complains with a "bge watchdog timeout" message.
>
> In case there is no hardware support for VT-d, as Craig noticed, how could
> this behaviour be explained? Why does link detection work?
>

I suspect that link status detection is done by polling device
registers in its MMIO space which is mapped in the guest's address
space. The MMIO accesses initiated by the guest will work independent
of VT-d.

It is only the reads/writes to system memory initiated by the device
that won't work properly without VT-d.

best
Neel

> Again, thank you both!
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Neel Natu]

Hi Nikos,

On Sun, Nov 30, 2014 at 7:58 AM, Nikos Vassiliadis  wrote:
> Hi,
>
> On 11/30/14 02:43, Neel Natu wrote:
>>
>> Can you provide the output of 'pciconf -lvbc pci0:3:0:0' on the FreeBSD
>> host?
>>
>> This is assuming pci0:3:0:0 is the wlan device being passed through
>> based on an earlier email.
>
>
> Yes, it is. Here is the output:
>>
>> none0@pci0:3:0:0:   class=0x028000 card=0xe042105b chip=0x472714e4
>> rev=0x01 hdr=0x00
>> vendor = 'Broadcom Corporation'
>> device = 'BCM4313 802.11b/g/n Wireless LAN Controller'
>> class  = network
>> bar   [10] = type Memory, range 64, base 0xb350, size 16384,
>> enabled
>> cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
>> cap 09[58] = vendor (length 120)
>> cap 05[48] = MSI supports 1 message, 64 bit

Ok, this device does have MSI which is what I wanted to confirm.

Its very likely that the device doesn't work when passed to the guest
because your system doesn't have VT-d.

best
Neel

>> cap 10[d0] = PCI-Express 1 endpoint max data 128(128) link x1(x1)
>>  speed 2.5(2.5) ASPM L1(L1)
>> ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
>> ecap 0002[13c] = VC 1 max VC0
>> ecap 0003[160] = Serial 1 dc210c84
>> ecap 0004[16c] = Power Budgeting 1
>
>
> Thanks, Nikos
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Nikos Vassiliadis]

Hi,

On 11/30/14 02:43, Neel Natu wrote:

Can you provide the output of 'pciconf -lvbc pci0:3:0:0' on the FreeBSD host?

This is assuming pci0:3:0:0 is the wlan device being passed through
based on an earlier email.


Yes, it is. Here is the output:

none0@pci0:3:0:0:   class=0x028000 card=0xe042105b chip=0x472714e4 rev=0x01 
hdr=0x00
vendor = 'Broadcom Corporation'
device = 'BCM4313 802.11b/g/n Wireless LAN Controller'
class  = network
bar   [10] = type Memory, range 64, base 0xb350, size 16384, enabled
cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
cap 09[58] = vendor (length 120)
cap 05[48] = MSI supports 1 message, 64 bit
cap 10[d0] = PCI-Express 1 endpoint max data 128(128) link x1(x1)
 speed 2.5(2.5) ASPM L1(L1)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0002[13c] = VC 1 max VC0
ecap 0003[160] = Serial 1 dc210c84
ecap 0004[16c] = Power Budgeting 1


Thanks, Nikos
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Nikos Vassiliadis]

Hi,

On 11/30/14 02:37, Neel Natu wrote:

The "Unsupported MSI-X configuration" referred to here is that bhyve
doesn't emulate the 'Pending Bit Array'.

In most cases this is not relevant because the PBA and the MSI-X
tables are in different page frames. In this case the MSI-X tables are
emulated and the pending bit array page is passed through to the
guest.

If the PBA and MSI-X tables are on the same page frame then you see
this error. You can confirm this by doing 'pciconf -lvbc pci0:2:0:0'.


Here is the ouput:

bge0@pci0:2:0:0:class=0x02 card=0x06471025 chip=0x16b514e4 rev=0x10 
hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetLink BCM57785 Gigabit Ethernet PCIe'
class  = network
subclass   = ethernet
bar   [10] = type Prefetchable Memory, range 64, base 0xb343, size 
65536, enabled
bar   [18] = type Prefetchable Memory, range 64, base 0xb344, size 
65536, enabled
cap 01[48] = powerspec 3  supports D0 D3  current D0
cap 05[58] = MSI supports 8 messages, 64 bit enabled with 1 message
cap 11[a0] = MSI-X supports 5 messages
 Table in map 0x18[0x0], PBA in map 0x18[0x120]
cap 10[ac] = PCI-Express 2 endpoint max data 128(128) link x1(x1)
 speed 2.5(2.5) ASPM L1(L0s/L1)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[13c] = Serial 1 208984c6b5ed
ecap 0004[150] = Power Budgeting 1
ecap 0002[160] = VC 1 max VC0





Having said that the drivers I have seen don't rely at all on the PBA.
So it may be possible that this works just fine with this patch:
https://people.freebsd.org/~neel/patches/bhyve_ignore_unsupported_pba.patch

If it does then I can add a tunable to bhyve to ignore this check.


It seems a tunable can help.

I tried the patch and now bhyve continues and indeed I see the bge 
device in my VM. What is interesting/strange is that the device is 
partly functioning. For example, it does detect a link change when I 
remove the cable but that's all. The kernel also complains with a "bge 
watchdog timeout" message.


In case there is no hardware support for VT-d, as Craig noticed, how 
could this behaviour be explained? Why does link detection work?


Again, thank you both!
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Nikos Vassiliadis]

Hi,

On 11/30/14 00:48, Craig Rodrigues wrote:

I'm not sure, but according to this datasheet:

http://ark.intel.com/products/65700/Intel-Core-i3-3110M-Processor-3M-Cache-2_40-GHz

this CPU does not support VT-d, which is needed for PCI passthru.


Indeed. Good catch, Neel should this be the case?
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: RFC: Enabling VIMAGE in GENERIC

[Julian Elischer]

On 11/29/14, 5:28 PM, Craig Rodrigues wrote:
On Mon, Nov 24, 2014 at 9:03 AM, Julian Elischer > wrote:

>
>
> also look at the following: (a little dated)
>
> 
http://p4web.freebsd.org/@md=d&cd=//depot/projects/vimage/&cdf=//depot/projects/vimage/porting_to_vimage.txt&c=tO0@//depot/projects/vimage/porting_to_vimage.txt?ac=22



 This is a useful document.  I put it on the wiki: 
https://wiki.freebsd.org/VIMAGE/porting-to-vimage


Thanks.. wow, did I actually know ALL that only 5 years ago?
Scary.  probbaly worth having someone who is currently active and up 
to date look at it to see if it's all still correct..

especially the module load/unload stuff.



--
Craig


___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"