Hi all Here is a minimal working version of supporting multiple pci domains. The next a few paragraphs will illustrate the purpose and use example. Current issue and limitations will be in last 2 paragraphs, followed by the changelog of each verison.
Currently only q35 host bridge is allocated an item in MCFG table, all pxb-pcie host bridges stay within pci domain 0. This series of patches allow each pxb-pcie to be put in separate pci domain, allocating a new MCFG table item for it. Users can configure whether to put pxb host bridge into separate domain by specifying property 'domain_nr' of pxb-pcie device. 'bus_nr' property indicates the Base Bus Number(BBN) of the pxb-pcie host bridge. Another property max_bus specifies the maximum desired bus numbers to reduce MCFG space cost. Example command is -device pxb-pcie,id=bridge3,bus="pcie.0",domain_nr=1,max_bus=15 Then this pxb-pcie host bridge is placed at pci domain 1, and only reserve (15+1)=16 buses, which is much smaller than the default 256 buses. Compared with previous version, this version is much simpler because mcfg of extra domain now has a relatively fixed address, as suggested by Marcel and Gerd. Putting extra mmconfig above 4G and let seabios leave them for guest os will be expected in next version. The range is [0x80000000, 0xb0000000), which allows us to hold 4x busses compared with before. A complete command line for test is follows, you need to replace GUEST_IMAGE, DATA_IMAGE and SEABIOS_BIN with proper environment variable ./x86_64-softmmu/qemu-system-x86_64 \ -machine q35,accel=kvm -smp 2 -m 2048 \ -drive file=${GUEST_IMAGE} -netdev user,id=realnet0 \ -device e1000e,netdev=realnet0,mac=52:54:00:12:34:56 \ -device pxb-pcie,id=bridge3,bus="pcie.0",domain_nr=1 \ -device pcie-root-port,id=rp1,bus=bridge3,addr=1c.0,port=8,chassis=8 \ -drive if=none,id=drive0,file=${DATA_IMAGE} \ -device virtio-scsi-pci,id=scsi,bus=rp1,addr=00.0 \ -bios ${SEABIOS_BIN} There are a few limitations, though 1. Legacy interrupt routing is not dealt with yet. There is only support for devices using MSI/MSIX 2. Only 4x devices is supported, you need to be careful not to overuse 3. I have not fully tested the functionality of devices under separate domain yet, but Linux can recognize then when typing `lspci` Current issue: * SCSI storage device will be recognized twice, one in domain 0 as 0000:01.0, the other in domain 1 as 0001:01.0. I will try to fix it in next version v5 <- v4: - Refactor the design and place pxb-pcie's mcfg in [0x80000000, 0xb0000000) - QEMU only decides the desired mcfg_size and leaves mcfg_base for seabios - Does not connect PXBDev and PXBPCIEHost with link property anymore, but with the pcibus under them, which makes code simpler. v4 <- v3: - Fix bug in setting mcfg table - bus_nr is not used when pxb-pcie is in a new pci domain v3 <- v2: - Replace duplicate properties in pxb pcie host with link property to PXBDev - Allow seabios to access config space and data space of expander bridge through a different ioport, because 0xcf8 is attached only to sysbus. - Add a new property start_bus to indicate the BBN of pxb host bridge. The bus_nr property is used as the bus number of pxb-pcie device on pcie.0 bus v2 <- v1: - Allow user to configure whether to put pxb-pcie into seperate domain - Add AML description part of each host bridge - Modify the location of MCFG space to between RAM hotplug and pci hole64 Many thanks to Please let me know if you have any suggestions. Zihan Yang (6): pci_expander_bridge: add type TYPE_PXB_PCIE_HOST pci_expander_bridge: add domain_nr and max_bus property for pxb-pcie acpi-build: allocate mcfg for pxb-pcie host bridges i386/acpi-build: describe new pci domain in AML pci_expander_bridge: add config_write callback for pxb-pcie pci_expander_bridge: inform seabios of desired mcfg size via hidden bar hw/i386/acpi-build.c | 162 ++++++++++++++++++-------- hw/pci-bridge/pci_expander_bridge.c | 172 +++++++++++++++++++++++++++- hw/pci/pci.c | 30 ++++- include/hw/pci-bridge/pci_expander_bridge.h | 25 ++++ include/hw/pci/pci.h | 2 + include/hw/pci/pci_bus.h | 2 + include/hw/pci/pci_host.h | 2 +- 7 files changed, 336 insertions(+), 59 deletions(-) create mode 100644 include/hw/pci-bridge/pci_expander_bridge.h -- 2.7.4