On 06/03/2015 01:20 PM, Laszlo Ersek wrote:
> On 06/03/15 10:15, Marcel Apfelbaum wrote:
>> On 06/02/2015 07:25 PM, Laszlo Ersek wrote:
>>> On 06/02/15 17:04, Marcel Apfelbaum wrote:
>>>> Hi,
>>>>
>>>> The following series:
>>>> - [Qemu-devel] [PATCH V8 00/17] hw/pc: implement multiple primary
>>>> busses for pc machines
>>>> - https://www.mail-archive.com/qemu-devel@nongnu.org/msg300089.html
>>>> adds a PCI Expander Device to QEMU that exposes a new PCI root
>>>> bus.
>>>
>>> (Let's tie this thread to the v7 question too:
>>>
>>> http://thread.gmane.org/gmane.comp.emulators.qemu/338583/focus=338599
>>> )
>>>
>>>> The PXB is a "light-weight" host bridge whose purpose is to enable
>>>> the main host bridge to support multiple PCI root buses.
>>>>
>>>> It does not have its own registers for configuration cycles, but is
>>>> snoops on main host bridge registers and it lives on the same PCI
>>>> segment.
>>>>
>>>> The device receives from the command line the bus number and expects
>>>> the firmware (bios/UEFI) to probe the bus for devices behind it and
>>>> configure them.
>>>>
>>>> My question is how can it be supported in edk2? Are there any
>>>> architecture limitations that will prevent it to work?
>>>>
>>>> My edk2/UEFI knowledge is rather limited, but I did see in the spec
>>>> that there is support for this kind of device:
>>>>
>>>> 13.1.1 PCI Root Bridge I/O Overview
>>>> ...
>>>> Depending on the chipset, a single
>>>> EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL may abstract a portion of a PCI
>>>> Segment, or an entire PCI Segment. A PCI Host Bridge may produce
>>>> one or more PCI Root Bridges. When a PCI Host Bridge produces
>>>> multiple PCI Root Bridges, it is possible to have more than one
>>>> PCI Segment.
>>>> ...
>>>>
>>>> It seems that multiple EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL instances for
>>>> the same PCI Host Bridge mapped into the same PCI Segment is the
>>>> answer. First instance belongs to the "main" host bridge and the other
>>>> to the PXBs.
>>>>
>>>> The open questions are of course how to assign resources (bus
>>>> numbers/IO/MEM) to the EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL instances.
>>>>
>>>> For the bus numbers I think that the PCI Host Bridge can scan the 0x0
>>>> - 0xff range and build incrementally the bus ranges.
>>>>
>>>> Regarding IO/MEM ranges I am still not sure. The way it is done in
>>>> SeaBIOS is that all devices behind PXB root bus are "considered" as
>>>> being behind bus 0 for resources allocation. Once the resources
>>>> allocation is done, each EFI_PCI_ROOT_BRIDGE gets the list of MEM/IO
>>>> ranges corresponding with the devices behind them.
>>>>
>>>> Any comments and suggestions would be greatly appreciated.
>>>> Thank you in advance,
>>>> Marcel
>>>
>>> I'm attaching a horrible patch (applies on top of edk2 SVN r17543, aka
>>> git commit d4848bb9df) that allows OVMF to recognize the e1000 NIC with
>>> the following QEMU command line:
>>>
>>> ISO=/mnt/data/isos/Fedora-Live-Xfce-x86_64-20-1.iso
>>> CODE=/home/virt-images/OVMF_CODE.fd
>>> TMPL=/home/virt-images/OVMF_VARS.fd
>>>
>>> cp $TMPL vars.fd
>>>
>>> qemu-system-x86_64 \
>>> -m 2048 \
>>> -M pc \
>>> -enable-kvm \
>>> -device qxl-vga \
>>> -drive if=pflash,readonly,format=raw,file=$CODE \
>>> -drive if=pflash,format=raw,file=vars.fd \
>>> -drive id=cdrom,if=none,readonly,format=raw,file=$ISO \
>>> -device virtio-scsi-pci,id=scsi0 \
>>> -device scsi-cd,bus=scsi0.0,drive=cdrom,bootindex=0 \
>>> -debugcon file:debug.log \
>>> -global isa-debugcon.iobase=0x402 \
>>> -device pxb,id=bridge1,bus_nr=128 \
>>> -netdev user,id=netdev0,hostfwd=tcp:127.0.0.1:2222-:22 \
>>> -device e1000,netdev=netdev0,bus=bridge1,addr=1 \
>>> -monitor stdio
>>>
>>> With this hack in place, and using the above QEMU command line,
>>> "debug.log" bears witness to the PCI enumeration succeeding, and the
>>> "PCI" command in the UEFI shell lists the e1000 NIC.
>> Hi Laszlo,
>>
>> These are very good news, that means that the device *can work* with edk2.
>>
>>>
>>> I agree with your analysis that the way to support this QEMU feature in
>>> OVMF is to produce several EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL instances.
>>> Beyond that agreement, I must say that invalidating the assumption that
>>> "there is only one root bridge" breaks about everything in OVMF.
>>>
>>> Just skimming my hack-patch identifies the problems (or in some cases,
>>> questions):
>>>
>>> * The PCI host bridge driver under PcAtChipsetPkg needs to be cloned
>>> under OvmfPkg, and (as you say) the bus ranges need to be determined
>>> dynamically. On IRC you said that probing for device 0 on a bus is
>>> sufficient to see if the bus lives, but for now I'm unsure if this would
>>> be a layering violation or not for the UEFI protocols in question. Maybe
>>> not.
>>>
>>> * The bus ranges assigned to each "pxb" device (ie. root bridge) would
>>> have to be able to accommodate any subordinate buses enumerated off that
>>> root bridge. At least this is what PciRootBridgeEnumerator() in
>>> "MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumerator.c" seems to require. I've
>>> got no clue how to size the bus ranges properly for the root bridges to
>>> satisfy this.
>>>
>>> * In fact, the bus range presented over the
>>> EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL.Configuration() function cannot be just
>>> a range. As far as I tested the PCI bus driver (see path above), it
>>> doesn't find anything if the range retrieved from
>>> EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL.Configuration() doesn't start *exactly*
>>> with a live bus. In other words, it doesn't look for sibling buses, or
>>> independent buses in the bus range exposed by the root bridge protocol.
>>> It looks for *one* bus *at* the start of the range (and subordinate
>>> buses hanging off of that). I have absolutely no clue why this is so,
>>> but it means that for each pxb found, one root bridge io protocol would
>>> have to be produced, and that proto should expose one bus range, with
>>> the low end matching exactly the bus number, and the high end enabling
>>> child buses to be enumerated.
>>>
>>> * In the attached hack, I'm splitting the pre-patch, static, IO & MMIO
>>> apertures in the middle. Maybe they could be the same shared ranges, as
>>> you say. I don't know.
>>>
>>> * In the OVMF BDS (boot device selection) code, we manually connect the
>>> only one root bridge. This would have to be made dynamic, to connect all
>>> of them. This connection basically amounts to "starting the enumeration".
>>>
>>> * The OVMF boot order processing code hardcodes PciRoot(0x0) in a bunch
>>> of device path matching logic. That would not be appropriate any longer.
>>> In fact the above command line should boot the fedora live CD, but it
>>> doesn't, and in the UEFI setup utility I cannot even browse the CD
>>> filesystem.
>>>
>>> * Gabriel wrote earlier some code for setting the INTx interrupt pin
>>> registers of all PCI devices in OVMF's BDS. That code breaks now, an
>>> assert is triggered ("PCI host bridge (00:00.0) should have no
>>> interrupts"). Not sure why this happens.
>>>
>>> * The UEFI device paths for the PCI root bridges (textually,
>>> PciRoot(0x0), PciRoot(0x1) etc) actually start with ACPI device path
>>> nodes. They consist of a PNP0A03 _HID and a numeric _UID. If my reading
>>> of the UEFI spec is correct, the _UIDs that OVMF would assign to these
>>> device path notes would have to match the *actual* ACPI payload that
>>> QEMU exports. The _UID assignment is now static (just a 0), and my
>>> hack-patch assigns a static 1 to the "other" root bridge's device path.
>>> This is not good. OVMF would either have to parse ACPI payload
>>> (horrible) or get the _UID<->pxb assignment via fw_cfg.
>>>
>>> That's all the carnage I can think of right now, but I'm sure this is
>>> just the tip of the iceberg. This would be a very large project, and
>>> QEMU might have to expose a lot more info over fw_cfg than it does now.
>>>
>>> In any case, the device model itself could be digestible for OVMF, based
>>> on the results of this hack.
>>
>>
>> Thanks a lot for your analysis.
>> Since I am new to edk2, I cannot take this project by myself, but if PCI
>> guys can come up with a plan or design, I'll be glad to implement it,
>> or at least to contribute to it.
>
> After sleeping on it :), I'd certainly like to find the time to
> collaborate on this myself.
Great!
Maybe we can experiment some more; for
> example we could start by you explaining to me how exactly to probe for
> a root bus's presence (you mentioned device 0, but I'll need more than
> that).
Well, I lied. :)
I had a look now on seabios and it does the following:
- Receives using a fw_config file the number of extra root buses.
- It starts scanning from bus 0 to bus 0xff until it discovers all
the extra root buses. The 'discovery' is "go over all bus's slots and
probe for a non empty PCI header". If you find at least one device you
just discovered a new PCI root bus.
I think that we can improve the fw_config file to pass the actually
bus numbers and not only the total. In this way should be relatively easy
for edk2 to handle the extra root buses.
>
> For the bus range allocation, here's an idea:
> - create a bitmap with 256 bits (32 bytes) with all bits zero
> - probe all root buses; whatever is found, flip its bit to 1
> - assuming N root buses were found, divide the number of remaining zero
> bits with N. The quotient Q means how many subordinate buses each root
> bus would be able to accommodate
> - for each root bus:
> - create an ACPI bus range descriptor that includes only the root
> bus's number
> - pull out Q zero bits from the bitmap, from the left, flipping them
> to one as you proceed
> - for each zero bit pulled, try to append that bus number to the ACPI
> bus range descriptor (simply bumping the end). If there's a
> discontinuity, start a new ACPI bus range descriptor.
>
> This greedy algorithm would grant each root bus the same number of
> possible subordinate buses, could be implemented in linear time, and
> would keep the individual bus ranges "reasonably continuous" (ie. there
> should be a reasonably low number of ACPI bus range descriptors, per
> root bus).
>
> What do you think? This wouldn't be a very hard patch to write, and then
> we could experiment with various -device pxb,bus_nr=xxx parameters.
Well, it looks nice but I think that we can do something much simpler :)
Let's continue the above idea that QEMU passes to edk2 the *extra* root bus
numbers
in ascending order for simplicity.
For example 8,16,32. From here you can derive that the bus ranges are:
0-7 host bridge 0
8-15 pxb root bridge 1
16-31 pxb root bridge 2
32-0xff pxb root bridge 3
BTW, this is the way, as far as I know, that the real hw divides the ranges.
Limitation:
- How do you know you have enough bus numbers for a host bridge to cover
all PCI-2-PCI bridges behind it? Let's say bus 0 has 10 bridges, 0-7 range
is not enough.
Reasoning:
-This is *hw vendor* issue, not firmware, in our case QEMU should check
the ranges are enough before starting edk2.
In conclusion, this assumption does not break anything or gives as a big
limitation.
And Seabios already assumes that... and QEMU is not going to break it.
>
> The MMIO and IO spaces I would just share between all of them; the
> allocations from those are delegated back to the host bridge / root
> bridge driver, and the current implementation seems sufficient -- it
> just assings blocks from the same big MMIO ( / IO) space downwards
Yes, this is how it should be done, I am happy that it already works that way.
Thanks,
Marcel
>
> Thanks
> Laszlo
>
------------------------------------------------------------------------------
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/edk2-devel
