Thanks Laszlo, I really appreciate the detailed explanation and feedback! On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote: > (a) your original (non-functional) use case: > > qemu-system-x86_64 \ > -no-user-config \ > -nodefaults \ > -machine q35,accel=kvm \ > -m 1G \ > -cdrom "$iso" \ > -device pcie-root-port,id=pci.1,bus=pcie.0 \ > -device VGA,bus=pci.1 > > violates the following part of "pcie.txt": > >> Plugging a PCI device into a PCI Express slot might not always work and >> is weird anyway since it cannot be done for "bare metal". > > AIUI, what Gerd explains in that SeaBIOS thread is why and how exactly > such an attempt breaks. The statement that it would break is already > spelled out in "pcie.txt". > > So I think that, given strictly your original report on the SeaBIOS > list, no updates to "pcie.txt" are necessary. > > Note that your original (functional) use case: > > qemu-system-x86_64 \ > -no-user-config \ > -nodefaults \ > -machine q35,accel=kvm \ > -m 1G \ > -cdrom "$iso" \ > -device VGA > > does not conflict with > >> Place only the following kinds of devices directly on the Root Complex: >> (1) PCI Devices (e.g. network card, graphics card, IDE controller), >> not controllers. Place only legacy PCI devices on >> the Root Complex. These will be considered Integrated Endpoints. >> Note: Integrated Endpoints are not hot-pluggable. >> >> Although the PCI Express spec does not forbid PCI Express devices as >> Integrated Endpoints, existing hardware mostly integrates legacy PCI >> devices with the Root Complex. Guest OSes are suspected to behave >> strangely when PCI Express devices are integrated >> with the Root Complex. > > because "-device VGA" is a legacy PCI device, not a PCI Express device. > So the second quoted paragraph does not apply to it at all, and the > first paragraph is in sync with your functional use case.
Ah, right you are. My mistake. Sorry for the noise. I had thought virtio-vga might be an exception, since there was discussion about changing virtio-vga to PCIe when connected to the Root Complex[4], but it looks like the change was not made due to compatibility concerns[5]. As you note below, this is currently the case for all devices with VGA compatibility, so this docs change is mostly a non-issue as long as this continues to be the case. On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote: > While VGA is quirky ("there was only ever intended to be one device" -- > see > <http://vfio.blogspot.com/2014/08/whats-deal-with-vga-arbitration.html>), > I maintain that it's sufficiently covered already (although not > specifically) by "pcie.txt"; see above. "-device VGA" is a legacy PCI > device, you can't plug it in an Express slot (root complex or port > alike). > > In the blog post at > <https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/>, Gerd > explains that (effectively) only "bochs-display" and "virtio-gpu-pci" > (from QEMU's emulated devices) may be placed in PCI Express slots (root > complex or separate port, alike). Those posts are both enlightening, thanks! On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote: > On 06/09/22 04:00, Kevin Locke wrote: >> and whether "Guest OSes are suspected to behave strangely when PCI >> Express devices are integrated with the Root Complex" is still the case. > > I vaguely recall that Alex didn't fully agree with this statement when > the document was being reviewed. > >> David Gibson had previously noted that graphics cards appear on the Root >> Complex on real hardware.[3] > > You forgot to provide the link for [3]. Whoops. Links to my qemu-discuss post[2] and David Gibson's post[3] mentioning "the GPU shows up as an integrated PCI Express endpoint" on his laptop are included in this email below. On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote: > On 06/09/22 04:00, Kevin Locke wrote: > Anyway, the full paragraph in the document goes like > >> Although the PCI Express spec does not forbid PCI Express devices as >> Integrated Endpoints, existing hardware mostly integrates legacy PCI >> devices with the Root Complex. Guest OSes are suspected to behave >> strangely when PCI Express devices are integrated >> with the Root Complex. > > So the recommendation is based on motherboards seen in the wild, and the > potential consequent assumptions in OSes. Whether that still holds up > with today's motherboards (and OSes), I have no clue about. > > [...] > >> --- a/docs/pcie.txt >> +++ b/docs/pcie.txt >> @@ -48,6 +48,11 @@ Place only the following kinds of devices directly on the >> Root Complex: >> strangely when PCI Express devices are integrated >> with the Root Complex. >> >> + An exception to this rule is PCI Express devices which will be >> + accessed using IO ports. > > I find this too general; a PCI Express device is supposed to work > without IO resources. Graphics cards with legacy VGA compatibility are > the exception AIUI (see again Alex's blog about VGA arbitration), so we > should spell that out. > > Furthermore, from QEMU's emulated graphics cards, all device models that > provide VGA compatibility are legacy PCI (not PCI Express) devices (see > again Gerd's post: VGA, virtio-vga, qxl-vga, cirrus-vga, ati-vga), so > I'd even restrict this docs update to assigned GPUs. > > [...] > > How about inserting a new paragraph here, instead: > > (2) Assigned PCI Express GPUs that offer legacy VGA compatibility, > and that such compatibility is expected of (due to booting with > SeaBIOS, or due to UEFI driver bugs or native OS driver bugs). That sounds great to me. Would you like me to send that as a v2 patch, or would you prefer to make the change? Thanks again, Kevin [2]: https://lists.nongnu.org/archive/html/qemu-discuss/2022-05/msg00053.html [3]: https://lore.kernel.org/qemu-devel/20170209041634.GC14524@umbus/ [4]: https://lore.kernel.org/qemu-devel/20170208061602.17666-1-da...@gibson.dropbear.id.au/T/ [5]: https://lore.kernel.org/qemu-devel/5ea3785c-b979-8b8c-3ab0-243d69384...@redhat.com/