On 06/30/16 20:14, Brian J. Johnson wrote: > On 06/30/2016 11:47 AM, Laszlo Ersek wrote: >> On 06/30/16 18:39, Marcel Apfelbaum wrote: >>> On 06/30/2016 07:21 PM, Marcel Apfelbaum wrote: >>>> On 06/30/2016 04:07 PM, Laszlo Ersek wrote: >>>>> Hi, >>>>> >>>>> does anyone on this list have experience with $SUBJECT, using physical >>>>> UEFI firmware derived from edk2? >>>>> >>>>> The problem we're seeing with OVMF is the following: PCIe hotplug >>>>> works >>>>> with PCIe root ports, but it doesn't work with PCIe downstream ports. >>>>> The error messages reported by both Linux and Windows guests indicate >>>>> "lack of resources" for the device being plugged. >>>>> >>>>> In my understanding: >>>>> >>>>> * Each PCIe downstream port qualifies as a separate bus / bridge. >>>>> >>>>> * Bridges need IO and MMIO resources for themselves. When a device is >>>>> behind a bridge at boot, the device's IO and MMIO BARs are >>>>> carved out >>>>> of the bridge's IO and MMIO apertures. For IO, the standard bridge >>>>> aperture size is 4096 (byte-sized) ports. >>>>> >>>>> * If no device behind a bridge needs a specific type of resource (for >>>>> example, no devices behind a bridge have IO BARs), then that >>>>> kind of >>>>> aperture is not necessary for the bridge either. >>>>> >>>>> A special case of this is when there are no devices behind a >>>>> bridge >>>>> at all. Example: PCIe downstream port empty at boot. In this case, >>>>> the bridge needs no resources at all, at boot. >>>>> >>>>> * Different firmwares seem to follow different policies in the last >>>>> case (-> bridge empty at boot). SeaBIOS for example allocates the >>>>> 0x1000 IO ports nonetheless. The PCI bus driver in edk2 does >>>>> not: it >>>>> allocates no apertures for bridges that have no devices behind >>>>> them >>>>> at boot. >>>>> >>>>> This appears to be a trade-off: if the bridge remains empty for >>>>> the >>>>> entire lifetime of the OS, then any IO ports that have been >>>>> allocated >>>>> in advance will be in vain (wasted). However, if the aperture >>>>> is not >>>>> allocated at boot, then hotplug under the bridge may not work -- >>>>> resources for the device being hot-plugged cannot be allocated >>>>> from >>>>> nonexistent bridge apertures. >>>>> >>>> >>>> Hi Laszlo, >>>> I fully agree with your assessment. >>>> >>>> From the firmware point of view it seems like a waste to allocate an >>>> IO/MEM range for >>>> an unused PCI/PCIe-bridge indeed. Especially IO. >>>> >>>> But what about MEM? >>>> Let's have a look to this special case: PCIe downstream ports on >>>> 64-bit systems. >>>> >>>> 1. PCIe spec requires devices to work correctly even without enabling >>>> their IO BARs. >>>> 2. 64-bit Memory space is pretty large, decent CPUs support 46-bit >>>> addresses, so even if the system uses Terabytes of RAM >>>> and has dozens of switches we still have enough unused MEM space. >>>> >>>> My point is we can skip assigning IO for PCIe ports but we can assign >>>> a 4K MEM range in 64-bit memory for them >>>> without being too greedy. >>> >>> I meant 2M MEM range, sorry. 4k is the min range for IO, of course. >> > > One thing to watch out for is prefetchable vs. non-prefetchable MEM > ranges. On Intel chipsets, the high (64-bit) MMIO range is always > prefetchable, which mainly makes it useful for frame buffers, > accelerator cards with RAM mapped to MMIO in the host, and the like. > Only the low (32-bit) MMIO space is non-prefetchable, and suitable for > registers and things which need precise ordering of loads and stores. > > So you can demote a prefetchable BAR into non-prefetchable MMIO space if > necessary, but not the other way around. Which is a shame, because the > 32-bit MMIO space is much more limited in size than the 64-bit space.
Good point! But, in "OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.c", we have AllocationAttributes = EFI_PCI_HOST_BRIDGE_COMBINE_MEM_PMEM; We only offer non-prefetchable 32-bit and 64-bit MMIO spaces. >> I've just been looking at the values reserved by SeaBIOS >> [src/fw/pciinit.c]: >> >> #define PCI_BRIDGE_MEM_MIN (1<<21) // 2M == hugepage size >> #define PCI_BRIDGE_IO_MIN 0x1000 // mandated by pci bridge spec >> >> 2MB sounds perfectly fine for MEM, especially given that OVMF supports a >> 64-bit MMIO aperture (32GB in size by default). >> >> We discussed the IO size earlier -- I think 4K is overkill (despite >> being required by the standard). We considered half K, 1K, 2K earlier... >> I think I'll take a stab at half K first. > > It depends on what your hardware (physical or virtual) supports. A > standards-compliant bridge may not be able to map I/O ports to targets > on less than a 4k boundary. Yes, 4k is ridiculous overkill, but it may > be your only choice, depending on the details of the h/w and/or VMM. Right, I'll carefully check the resource maps that the PCI bus driver prints, with my patch (work in progress) applied. > We've had to add special cases in BIOS for certain cards we support > which request IO space, but don't actually use it. O_o > We can safely skip > allocating IO ports to those cards, saving significant space in the > overall IO port map. We've worked with the card vendors on cleaning up > their PCIe advertisements and eliminating the use of IO ports, but it > takes time. I surprisingly often hear about non-conformance or unjustified behavior... I guess with huge, elaborate specs, this is unavoidable. No single programmer might be able to internalize it all. Thanks! Laszlo _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
