[Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
Hi, We have got some issues with 64bit PCI devices support under Windows guests. Windows fails to start any PCI devices with 64bit BARs when size exceeds 512MB with error code 12 (can't allocate resources). In other words it fails when the PCI memory region is located above 4GB. I tried to investigate this problem and found document from Microsoft: http://msdn.microsoft.com/en-us/windows/hardware/gg462986 It states that the _DSM method must be implemented to support devices above 4GB. Unfortunately I'm not familiar with ACPI code of seabios, so can't implement the _DSM method. May be you have other ideas what needs to be done to fix it? Any help is appreciated. Thanks, Alexey
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
On Wed, Nov 30, 2011 at 02:04:08PM +1300, Alexey Korolev wrote: Hi, We have got some issues with 64bit PCI devices support under Windows guests. Windows fails to start any PCI devices with 64bit BARs when size exceeds 512MB with error code 12 (can't allocate resources). In other words it fails when the PCI memory region is located above 4GB. I tried to investigate this problem and found document from Microsoft: http://msdn.microsoft.com/en-us/windows/hardware/gg462986 It states that the _DSM method must be implemented to support devices above 4GB. That's not how I read it: Windows Vista always respects the boot configuration of devices above 4 GB, considering the processor’s addressing capability and the version of the Windows operating system that is running. Well, it's written in a very interesting way. They do recommend to have _DSM method in order to support 64bit. It is quite possible that the source of the problem is somewhere else. Actually it is also quite interesting to know what is the resource window for 440FX? Perhaps Windows OS does not believe that 440FX supports 64bit PCI resources. But I see no way to check this. Unfortunately I'm not familiar with ACPI code of seabios, so can't implement the _DSM method. May be you have other ideas what needs to be done to fix it? Any help is appreciated. Thanks, Alexey Could you share some more detail about your setup? Ok Here are logs the faulty device is: 00:05.0. In the first case the BAR1 size is 512MB and it fails in Windows (Linux is fine, the kernel enumerates the bus and assigns the resource above 4GB), In the second case the BAR1 size is 256M and works fine. The first one causing issues. As PCI memory region for 00.0.5 is not assigned after windows boot. (Note. I also tried to supply a particular physical addresses over 4GB in pciinit.c code but this leads to completely the same behaviour. Windows just don't pick-up the resource.) - Seabios log PCI BAR 512MB: Changing serial settings was 0/0 now 3/0 Start bios (version pre-0.6.3-20111201_162012-core-3000-3.et.endace.com) enabling shadow ram qemu_cfg_present=1 Find memory size Ram Size=0xe000 (0x00032000 high) malloc setup Relocating init from 0x000e01c0 to 0xdffe0980 (size 62800) malloc fixup reloc init ivt init bda init pic init timer CPU Mhz=2801 math cp init init mtrr pci setup === PCI bus & bridge init === PCI: pci_bios_init_bus_rec bus = 0x0 === PCI device probing === PCI probe PCI device 00:00.0 (vd=8086:1237 c=0600) PCI device 00:01.0 (vd=8086:7000 c=0601) PCI device 00:01.1 (vd=8086:7010 c=0101) PCI device 00:01.2 (vd=8086:7020 c=0c03) PCI device 00:01.3 (vd=8086:7113 c=0680) PCI device 00:02.0 (vd=1013:00b8 c=0300) PCI device 00:03.0 (vd=10ec:8139 c=0200) PCI device 00:05.0 (vd=eace:eace c=0280) PCI device 00:06.0 (vd=1af4:1002 c=0500) Found 9 PCI devices (max PCI bus is 00) === PCI new allocation pass #1 === PCI: check devices bus 0 === PCI new allocation pass #2 === PCI: init bases bus 0 (primary) type io max 100 sum 150 base c000 size 100: 1 bar(s), c000 -> c0ff size 20: 2 bar(s), c100 -> c13f size 10: 1 bar(s), c140 -> c14f type mem max 1 sum 32000 base febc size1: 3 bar(s), febc -> febe size 1000: 2 bar(s), febf -> febf1fff type prefmem max 200 sum 200 base fc00 size 200: 1 bar(s), fc00 -> fdff PCI: map device bus 0, bfd 0x0 PCI: map device bus 0, bfd 0x8 PCI: map device bus 0, bfd 0x9 bar 4, addr c140, size 10 [io] PCI: map device bus 0, bfd 0xa bar 4, addr c100, size 20 [io] PCI: map device bus 0, bfd 0xb PCI: map device bus 0, bfd 0x10 bar 0, addr fc00, size 200 [mem] bar 1, addr febf, size 1000 [mem] bar 6, addr febc, size 1 [mem] PCI: map device bus 0, bfd 0x18 bar 0, addr c000, size 100 [io] bar 1, addr febf1000, size 100 [mem] bar 6, addr febd, size 1 [mem] PCI: map device bus 0, bfd 0x28 bar 0, addr febe, size 1 [mem] bar 1, addr 0, size 2000 [mem] PCI: map device bus 0, bfd 0x30 bar 0, addr c120, size 20 [io] PCI: bus=0 devfn=0x00: vendor_id=0x8086 device_id=0x1237 PCI: bus=0 devfn=0x08: vendor_id=0x8086 device_id=0x7000 PCI: bus=0 devfn=0x09: vendor_id=0x8086 device_id=0x7010 PCI: bus=0 devfn=0x0a: vendor_id=0x8086 device_id=0x7020 PCI: bus=0 devfn=0x0b: vendor_id=0x8086 device_id=0x7113 PCI: bus=0 devfn=0x10: vendor_id=0x1013 device_id=0x00b8 PCI: bus=0 devfn=0x18: vendor_id=0x10ec device_id=0x8139 PCI: bus=0 devfn=0x28: vendor_id=0xeace device_id=0xeace PCI: bus=0 devfn=0x30: vendor_id=0x1af4 device_id=0x1002 init smm boot order: 1: /pci@i0cf8/ide@1,1/drive@0/disk@0 2: /pci@i0cf8/ide@1,1/drive@1/disk@0 Found 2 cpu(s) max supported 2 cpu(s) init bios32 init PMM init PNPBIOS table init keyboard init mouse init
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
Isaku san, I've just added you to discussion. There are some issues with PCI 64bit support in Windows. Windows fails to assign the resource if it doesn't fit in first 4GB window. I really don't know why it happens. One of the possibilities is related to lack of _DSM method in ACPI. Another guesse could be related to the fact that 440FX only supports 32bit PCI bus interface and windows may limit PCI address range to first 4GB for PCI devices under this bridge. I remember you were working on Q35 chipset simulation, I wonder if it is working and would it be possible to try? Thanks, Alexey On Wed, Nov 30, 2011 at 02:04:08PM +1300, Alexey Korolev wrote: Hi, We have got some issues with 64bit PCI devices support under Windows guests. Windows fails to start any PCI devices with 64bit BARs when size exceeds 512MB with error code 12 (can't allocate resources). In other words it fails when the PCI memory region is located above 4GB. I tried to investigate this problem and found document from Microsoft: http://msdn.microsoft.com/en-us/windows/hardware/gg462986 It states that the _DSM method must be implemented to support devices above 4GB. That's not how I read it: Windows Vista always respects the boot configuration of devices above 4 GB, considering the processor’s addressing capability and the version of the Windows operating system that is running. Unfortunately I'm not familiar with ACPI code of seabios, so can't implement the _DSM method. May be you have other ideas what needs to be done to fix it? Any help is appreciated. Thanks, Alexey Could you share some more detail about your setup?
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
Hi Michael, Thank you for good advice, you are right. When I added new range above 4GB in _CRS the problem has gone. QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x, // Address Space Granularity 0x1,// Address Range Minimum 0x3,// Address Range Maximum 0x, // Address Translation Offset 0x4,// Address Length ,, , AddressRangeMemory, TypeStatic) The only big problem with this range - as soon as I have more than 3GB of RAM, windows will boot in BSOD. The problem relates to memory range intersection. Unfortunately it is not possible to predict how many GB of RAM the virtual machine could have - so it's difficult to specify a particular region. Do you have any ideas what can be done to solve this problem? Regards, Alexey On Thu, Dec 01, 2011 at 06:49:54PM +1300, Alexey Korolev wrote: Isaku san, I've just added you to discussion. There are some issues with PCI 64bit support in Windows. Windows fails to assign the resource if it doesn't fit in first 4GB window. I really don't know why it happens. One of the possibilities is related to lack of _DSM method in ACPI. Another guesse could be related to the fact that 440FX only supports 32bit PCI bus interface and windows may limit PCI address range to first 4GB for PCI devices under this bridge. I remember you were working on Q35 chipset simulation, I wonder if it is working and would it be possible to try? Thanks, Alexey Maybe the range above 4G needs to be declared in the _CRS resource?
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
Hi Gerd, We have very early prototype of data acquisition device, with quite large MMIO buffer. It is an emulated device. We are running the 0.15 release. 0.15 doesn't work correctly with 64bit BARs so I've already added some hacks to Seabios to let OS to choose the memory region. Thus you see bar 1, addr 0 in seabios log. Sorry that I haven't specified all this initially. I just want to make 64bit PCI bar working properly. Linux guests works correctly (except early versions - not investigated this yet). At the moment I have some issues with windows which relies on ACPI _CRS. Thanks, Alexey Hi, PCI: map device bus 0, bfd 0x28 bar 0, addr febe, size 1 [mem] bar 1, addr 0, size 2000 [mem] Somehow seabios didn't recognise the bar correctly it seems (both 512 and 256 MB cases look the same). For the 256 MB case seabios should have mapped the bar @ 0xe000. ... and it should also have figured it is prefetchable memory. Was pci config space messed up somehow? What does 'lspci -v' say once you've booted the machine with linux? What qemu version you are running? What kind of device is this? Emulated? Code somewhere? Or a real device passed through to the guest? cheers, Gerd
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
On 05/12/11 19:31, Michael S. Tsirkin wrote: On Mon, Dec 05, 2011 at 05:20:32PM +1300, Alexey Korolev wrote: Hi Michael, Thank you for good advice, you are right. When I added new range above 4GB in _CRS the problem has gone. QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x, // Address Space Granularity 0x1,// Address Range Minimum 0x3,// Address Range Maximum 0x, // Address Translation Offset 0x4,// Address Length ,, , AddressRangeMemory, TypeStatic) The only big problem with this range - as soon as I have more than 3GB of RAM, windows will boot in BSOD. The problem relates to memory range intersection. Unfortunately it is not possible to predict how many GB of RAM the virtual machine could have - so it's difficult to specify a particular region. Do you have any ideas what can be done to solve this problem? Regards, Alexey Two possible ideas: 1. Pass the value in from qemu Do you mean to create the 64bit region dynamically? I've tried to obtain RAM size somehow inside DSL code, but the ACPI spec doesn't tell how to do that. 2. Get a range toward the upper end of the memory, around 1<<40 Yes. This solution works great. And this is very simple as well. Just wondering if it will be good as a complete solution?
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
Hi Gerd, I'd strongly suggest to move forward to qemu 1.0. Memory region handling has seen a major rewrite in 1.0 (memory api patches by avi). Chances are good that the 64bit bar bugs in qemu have been fixed meanwhile. Thanks, will try it. Hope it will be better. I have experimental patches which add a 64bit bar to the qxl device and seabios handles it just fine (although memory-backed not mmio), except that there is no support yet to map 64bit bars above 4G. It shouldn't be that hard to add the latter though. seabios needs two more pci_region_type (PCI_REGION_TYPE_MEM_64 and PCI_REGION_TYPE_PREFMEM_64) to track and map 64bit bars separately. And a address space window where it can map 64bit bars to. Right. This is a thing I'm thinking about now. I seems that the specifying 0x0 address in 64bit BARS is a bad idea. At least older versions of Linux just hang as soon as requested range does not fit in first 4GB. So the only option would be specifying the particular address range. It seems this works for everything - yet. Sorry that I haven't specified all this initially. I just want to make 64bit PCI bar working properly. Linux guests works correctly (except early versions - not investigated this yet). At the moment I have some issues with windows which relies on ACPI _CRS. ... and a _CRS entry for the 64bit bar address space window of course. Yes. Cheers, Alexey
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
Hi, It shouldn't be that hard to add the latter though. seabios needs two more pci_region_type (PCI_REGION_TYPE_MEM_64 and PCI_REGION_TYPE_PREFMEM_64) to track and map 64bit bars separately. And a address space window where it can map 64bit bars to. Right. This is a thing I'm thinking about now. I seems that the specifying 0x0 address in 64bit BARS is a bad idea. At least older versions of Linux just hang as soon as requested range does not fit in first 4GB. So the only option would be specifying the particular address range. It seems this works for everything - yet. I wonder if there any particular reason to separate prefetchable a non-prefetchable memory regions in pciinit? Extra two more regions would make code more complex. Regards, Alexey
Re: [Qemu-devel] [SeaBIOS] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
2. Get a range toward the upper end of the memory, around 1<<40 Yes. This solution works great. And this is very simple as well. Just wondering if it will be good as a complete solution? I would have tried this first too ;) Care to share the patches so we can review& merge them? No worries. Just need to complete and test the code and I'll post RFC patches shortly.
Re: [Qemu-devel] [BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
I wonder if there any particular reason to separate prefetchable a non-prefetchable memory regions in pciinit? Extra two more regions would make code more complex. Oh yes, there is. Which reminds me that the whole thing isn't that easy unfortunaly ... The reason are pci bridges. They have two memory regions, one for prefetchable and one for non-prefetchable memory. All devices behind a pci bridge must have the bars within the bridges memory regions, thats why they are grouped together. This also implies that a 32bit and a 64bit bar (of the same type) behind a pci bridge must be mapped next to each other, so moving up 64bit bars unconditionally isn't going to fly. Devices behind bridges need some extra care, only when all bars are 64bit capable they can actually be mapped above 4G. The qemu actually does not simulate PCI bridges at all. So good question shall we bother about this? I did some preliminary tests: 64bit BARs are working quite well for linux 2.6.18 - 3.0 and windows 2008. Also I've found important detail that according to PCI architecture specification, the bridges can describe 64bit ranges for prefetchable type of memory only. So it's very unlikely that devices exporting 64bit non-prefetchable BARs. I guess we just need to add one extra type. Even if there are two separate prefetchable memory regions (32 bit and 64bit), it won't be a problem as there is no bridge on the 440FX inside the virtual machine. Cheers, Alexey
[Qemu-devel] [BUG] BSOD on Win2003 Server when 64bit PCI resource is present
HI, Current version of Seabios is causing blue screen on Windows2003 when 64bit PCI resource is present and occupies high memory. BSOD Error code is: 0x00A5 (0x02, 0xfADF6A446880, 0x1, 0xFADFAA34690) The issue is localized, it is related to presence of 64bit resource in _CRS method. If we disable a 64bit region from _CRS the Win2003 load normally but this doesn't allow Windows to use 64bit resources. At the moment I have no idea how to fix this. Please help! P/S Yet another issue is related to debug messages. If I add any DBUG call in acpi-dst it will cause BSOD on win2003 (win2008 works fine)
Re: [Qemu-devel] [BUG] BSOD on Win2003 Server when 64bit PCI resource is present
Hi Kevin, >Unfortunately, it's very difficult to debug acpi issues on Windows. >Gerd's been on vacation this week - so, lets give him a chance to look >at it when he gets back. If it can't be resolved, we'll need to >revert the patch that broke Win2003. Thank you for your reply. Right - acpi issues are painful. Actually I'm on a long holidays too. I'll do my best if you or Gerd need any help with the issue. Kind regards, Alexey
[Qemu-devel] [PATCH] [Seabios] Over 4GB address ranges for 64bit PCI BARs
Hi,
We have seen some issues with 64bit PCI address and large BAR region support in
seabios.
On attempting to register a 64bit BAR of 1GB size we had some strange failures
in qemu. After some debugging we find out that source of the issue was in
seabios PCI enumeration code.
The issue takes place because of u32 overflow in pciinit.c.
….
if (ALIGN(*paddr, size) + size >= BUILD_PCIPREFMEM_END) {
dprintf(1,
"prefmem region of (bdf 0x%x bar %d) can't be mapped. "
"decrease BUILD_PCIMEM_SIZE and recompile. size %x\n",
bdf, region_num, BUILD_PCIPREFMEM_SIZE);
size = 0;
}
…...
if (size > 0) {
*paddr = ALIGN(*paddr, size);
pci_set_io_region_addr(bdf, region_num, *paddr);
*paddr += size;
}
If size is greater than 0x – BUILD_PCIPREFMEM_START (256MB),
this call ALIGN(*paddr, size) will always return 0. The protection test fails,
size remains > 0, and guest memory mapping will be corrupted.
We have found that a very similar problem was discussed previously:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg38090.html
The discussion also touches on the question of 64bit addressing for PCI BARs.
In the current implementation regardless of whether the PCI BAR type is 64bit
or not
the addressable range will be limited to the first 4GB. We also want to have
"true" 64bit
addressable range for PCI BARs.
So to solve these issues some changes were made.
Tracking for overflows has been added as well as support for 64bit range.
To support the full 64bit address range a pci_bios_64bit_addr variable has been
added.
The 64bit range can be used only if BAR has PCI_BASE_ADDRESS_MEM_TYPE_64
attribute and the given region doesn't fit in first 4GB. The patch has been
tested on Linux and BSD guest OS's and it appears to work fine.
Signed-off-by Alexey Korolev
Signed-off-by Stephen Donelly
---
pciinit.c | 83 --
1 file changed, 43 insertions(+), 40 deletions(-)
diff --git a/src/pciinit.c b/src/pciinit.c
index 0346423..86c8137 100644
--- a/src/pciinit.c
+++ b/src/pciinit.c
@@ -20,6 +20,7 @@ static void pci_bios_init_device_in_bus(int bus);
static u32 pci_bios_io_addr;
static u32 pci_bios_mem_addr;
static u32 pci_bios_prefmem_addr;
+static u64 pci_bios_64bit_addr;
/* host irqs corresponding to PCI irqs A-D */
const u8 pci_irqs[4] = {
10, 10, 11, 11
@@ -56,9 +57,11 @@ static int pci_bios_allocate_region(u16 bdf, int region_num)
{
u32 *paddr;
u32 ofs = pci_bar(bdf, region_num);
-
u32 old = pci_config_readl(bdf, ofs);
u32 mask;
+u32 limit_addr;
+int is_64bit;
+
if (region_num == PCI_ROM_SLOT) {
mask = PCI_ROM_ADDRESS_MASK;
pci_config_writel(bdf, ofs, mask);
@@ -73,50 +76,49 @@ static int pci_bios_allocate_region(u16 bdf, int region_num)
pci_config_writel(bdf, ofs, old);
u32 size = (~(val & mask)) + 1;
-if (val != 0) {
-if (val & PCI_BASE_ADDRESS_SPACE_IO) {
-paddr = &pci_bios_io_addr;
-if (ALIGN(*paddr, size) + size >= 64 * 1024) {
-dprintf(1,
-"io region of (bdf 0x%x bar %d) can't be mapped.\n",
-bdf, region_num);
-size = 0;
-}
-} else if ((val & PCI_BASE_ADDRESS_MEM_PREFETCH) &&
- /* keep behaviour on bus = 0 */
- pci_bdf_to_bus(bdf) != 0 &&
- /* If pci_bios_prefmem_addr == 0, keep old behaviour */
- pci_bios_prefmem_addr != 0) {
-paddr = &pci_bios_prefmem_addr;
-if (ALIGN(*paddr, size) + size >= BUILD_PCIPREFMEM_END) {
-dprintf(1,
-"prefmem region of (bdf 0x%x bar %d) can't be mapped. "
-"decrease BUILD_PCIMEM_SIZE and recompile. size %x\n",
-bdf, region_num, BUILD_PCIPREFMEM_SIZE);
-size = 0;
-}
-} else {
-paddr = &pci_bios_mem_addr;
-if (ALIGN(*paddr, size) + size >= BUILD_PCIMEM_END) {
+if (!val || !size)
+return 0;
+
+is_64bit = !(val & PCI_BASE_ADDRESS_SPACE_IO) &&
+(val & PCI_BASE_ADDRESS_MEM_TYPE_MASK) == PCI_BASE_ADDRESS_MEM_TYPE_64;
+
+if (val & PCI_BASE_ADDRESS_SPACE_IO) {
+paddr = &pci_bios_io_addr;
+limit_addr = 64 * 1024;
+} else if ((val & PCI_BASE_ADDRESS_MEM_PREFETCH) &&
+ /* keep behaviour on bus = 0 */
+ pci_bdf_to_bus(bdf) != 0 &&
+ /* If pci_bios_prefmem_addr == 0, keep old behaviour */
+ pci_bios_prefmem_
[Qemu-devel] Re: [PATCH] [Seabios] Over 4GB address ranges for 64bit PCI BARs
Hi, > Hi. > The current BAR allocation doesn't check overflow and some patches > are floating around which aren't merged yet. > There are several issues. > > - overflow check > This should be fixed. > Some patches are proposed. None hasn't been merged yet. > Your patch also addresses this issue. > http://www.seabios.org/pipermail/seabios/2010-July/000794.html > http://www.seabios.org/pipermail/seabios/2010-October/001089.html Right. It would be great if the fix get included in seabios/qemu. Update: I've seen today's message from Kevin. He is going to merge your patch. So it will be good news for us. > - >4GB 64bit bar allocation > Your patche tries to address this issue. But it breaks PCI-to-PCI > bridge filtering support. Hmm, it is quite possible, as we don't know a lot about seabios PCI-to-PCI bridge filtering support. Just out of curiosity: what is the issue? > If the BAR size is huge (or there are too many BARs), the bar can't > be allocated under 4G. So several persons want seabios to allocate > such BARs at >4GB area complaining that OS can't use BARs that seabios > didn't assigned. > > Others think such BAR can be left unallocated. > Seabios role is to setup minimal basic environment for bootloader > to boot OS, 64bit bar allocation is beyond it's role. > bootloader/rombios usually doesn't handle BARs that is allocated > beyond 4GB, and Modern OSes can re-arrange PCI bar allocation itself. > So 64bit bar allocation support wouldn't be needed. > > I'm not sure if there is enough demand to support 64bit BAR allocation > and if Kevin will accept it or not. Consensus is needed. > What OS are you using? > For us >4GB allocation is welcome but not critical, because we mainly use Linux versions 2.6.18 and newer. We've tested the seabios without assignment of the regions which do not fit in first 32bit and it appears to work fine. So for us 64bit bar allocation support wouldn't be needed. It is possible that people will use an ancient version of Linux, but the probability of this event is very low. Thanks, Alexey
