[Bug 1921444] Re: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest
[Expired for QEMU because there has been no activity for 60 days.] ** Changed in: qemu Status: Incomplete => Expired -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921444 Title: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest Status in QEMU: Expired Bug description: KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git branch: next, commit: 4a98623d Qemu: https://git.qemu.org/git/qemu.git branch: master, commit: 9e2e9fe3 Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device to guest with qemu internal command device_add, the 1st device can be added successfully, but the 2nd device failed to hot add. If guest chipset is legacy i440fx, the 2 device can be added successfully. 1. Enable VT-d in BIOS 2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel 3. Bind 2 device to vfio-pci echo :b1:00.0 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id echo :b1:00.1 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 4. create guest with Q35 chipset: qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device pcie-root-port,id=root1 -daemonize 5. hot add the 1st device to guest successfully in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1" 6. hot add the 2nd device to guest in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1" The 2nd device doesn't be added in guest, and the 1st device is removed from guest. Guest partial log: [ 110.452272] pcieport :00:04.0: pciehp: Slot(0): Attention button pressed [ 110.453314] pcieport :00:04.0: pciehp: Slot(0) Powering on due to button press [ 110.454156] pcieport :00:04.0: pciehp: Slot(0): Card present [ 110.454792] pcieport :00:04.0: pciehp: Slot(0): Link Up [ 110.580927] pci :01:00.0: [8086:1572] type 00 class 0x02 [ 110.582560] pci :01:00.0: reg 0x10: [mem 0x-0x007f 64bit pref] [ 110.583453] pci :01:00.0: reg 0x1c: [mem 0x-0x7fff 64bit pref] [ 110.584278] pci :01:00.0: reg 0x30: [mem 0x-0x0007 pref] [ 110.585051] pci :01:00.0: Max Payload Size set to 128 (was 512, max 2048) [ 110.586621] pci :01:00.0: PME# supported from D0 D3hot D3cold [ 110.588140] pci :01:00.0: BAR 0: no space for [mem size 0x0080 64bit pref] [ 110.588954] pci :01:00.0: BAR 0: failed to assign [mem size 0x0080 64bit pref] [ 110.589797] pci :01:00.0: BAR 6: assigned [mem 0xfe80-0xfe87 pref] [ 110.590703] pci :01:00.0: BAR 3: assigned [mem 0xfe00-0xfe007fff 64bit pref] [ 110.592085] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.592755] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.594403] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.595847] pcieport :00:04.0: bridge window [mem 0xfe00-0xfe1f 64bit pref] [ 110.597867] PCI: No. 2 try to assign unassigned res [ 110.597870] release child resource [mem 0xfe00-0xfe007fff 64bit pref] [ 110.597871] pcieport :00:04.0: resource 15 [mem 0xfe00-0xfe1f 64bit pref] released [ 110.598881] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.600789] pcieport :00:04.0: BAR 15: assigned [mem 0x18000-0x180bf 64bit pref] [ 110.601731] pci :01:00.0: BAR 0: assigned [mem 0x18000-0x1807f 64bit pref] [ 110.602849] pci :01:00.0: BAR 3: assigned [mem 0x18080-0x180807fff 64bit pref] [ 110.604069] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.604941] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.606237] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.607401] pcieport :00:04.0: bridge window [mem 0x18000-0x180bf 64bit pref] [ 110.653661] i40e: Intel(R) Ethernet Connection XL710 Network Driver [ 110.654443] i40e: Copyright (c) 2013 - 2019 Intel Corporation. [ 110.655314] i40e :01:00.0: enabling device (0140 -> 0142) [ 110.672396] i40e :01:00.0: fw 6.0.48442 api 1.7 nvm 6.01 0x800035b1 1.1747.0 [8086:1572] [8086:0008] [ 110.750054] i40e :01:00.0: MAC address: 3c:fd:fe:c0:59:98 [ 110.751792] i40e :01:00.0: FW LLDP is enabled [ 110.764644] i40e :01:00.0 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None [ 110.779390] i40e :01:00.0: PCI-Express: Speed 8.0GT/s Width x8 [ 110.789841] i40e :01:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 4 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA [ 111.817553] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[Bug 1921444] Re: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest
The QEMU project is currently moving its bug tracking to another system. For this we need to know which bugs are still valid and which could be closed already. Thus we are setting the bug state to "Incomplete" now. If the bug has already been fixed in the latest upstream version of QEMU, then please close this ticket as "Fix released". If it is not fixed yet and you think that this bug report here is still valid, then you have two options: 1) If you already have an account on gitlab.com, please open a new ticket for this problem in our new tracker here: https://gitlab.com/qemu-project/qemu/-/issues and then close this ticket here on Launchpad (or let it expire auto- matically after 60 days). Please mention the URL of this bug ticket on Launchpad in the new ticket on GitLab. 2) If you don't have an account on gitlab.com and don't intend to get one, but still would like to keep this ticket opened, then please switch the state back to "New" or "Confirmed" within the next 60 days (other- wise it will get closed as "Expired"). We will then eventually migrate the ticket automatically to the new system (but you won't be the reporter of the bug in the new system and thus you won't get notified on changes anymore). Thank you and sorry for the inconvenience. ** Changed in: qemu Status: New => Incomplete -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921444 Title: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest Status in QEMU: Incomplete Bug description: KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git branch: next, commit: 4a98623d Qemu: https://git.qemu.org/git/qemu.git branch: master, commit: 9e2e9fe3 Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device to guest with qemu internal command device_add, the 1st device can be added successfully, but the 2nd device failed to hot add. If guest chipset is legacy i440fx, the 2 device can be added successfully. 1. Enable VT-d in BIOS 2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel 3. Bind 2 device to vfio-pci echo :b1:00.0 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id echo :b1:00.1 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 4. create guest with Q35 chipset: qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device pcie-root-port,id=root1 -daemonize 5. hot add the 1st device to guest successfully in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1" 6. hot add the 2nd device to guest in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1" The 2nd device doesn't be added in guest, and the 1st device is removed from guest. Guest partial log: [ 110.452272] pcieport :00:04.0: pciehp: Slot(0): Attention button pressed [ 110.453314] pcieport :00:04.0: pciehp: Slot(0) Powering on due to button press [ 110.454156] pcieport :00:04.0: pciehp: Slot(0): Card present [ 110.454792] pcieport :00:04.0: pciehp: Slot(0): Link Up [ 110.580927] pci :01:00.0: [8086:1572] type 00 class 0x02 [ 110.582560] pci :01:00.0: reg 0x10: [mem 0x-0x007f 64bit pref] [ 110.583453] pci :01:00.0: reg 0x1c: [mem 0x-0x7fff 64bit pref] [ 110.584278] pci :01:00.0: reg 0x30: [mem 0x-0x0007 pref] [ 110.585051] pci :01:00.0: Max Payload Size set to 128 (was 512, max 2048) [ 110.586621] pci :01:00.0: PME# supported from D0 D3hot D3cold [ 110.588140] pci :01:00.0: BAR 0: no space for [mem size 0x0080 64bit pref] [ 110.588954] pci :01:00.0: BAR 0: failed to assign [mem size 0x0080 64bit pref] [ 110.589797] pci :01:00.0: BAR 6: assigned [mem 0xfe80-0xfe87 pref] [ 110.590703] pci :01:00.0: BAR 3: assigned [mem 0xfe00-0xfe007fff 64bit pref] [ 110.592085] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.592755] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.594403] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.595847] pcieport :00:04.0: bridge window [mem 0xfe00-0xfe1f 64bit pref] [ 110.597867] PCI: No. 2 try to assign unassigned res [ 110.597870] release child resource [mem 0xfe00-0xfe007fff 64bit pref] [ 110.597871] pcieport :00:04.0: resource 15 [mem 0xfe00-0xfe1f 64bit pref] released [ 110.598881] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.600789] pcieport :00:04.0: BAR 15: assigned [mem 0x18000-0x180bf 64bit pref] [ 110.601731] pci :01:00.0: BAR 0: assigned [mem 0x18000-0x1807f 64bit pref] [ 110.602849]
[Bug 1921444] Re: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest
Alex, thanks for you response. I tried to create guest with multiple root ports, but qemu report error: # qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive file=/home/hao/rhel8.2.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -device virtio-net-pci,netdev=nic0,mac=00:16:3e:0d:e4:ab -netdev tap,id=nic0,br=virbr0,helper=/usr/local/libexec/qemu-bridge-helper -cpu host -machine q35 -device pcie-root-port,id=root1 -device pcie-root-port,id=root2 -daemonize qemu-system-x86_64: -device pcie-root-port,id=root2: Can't add chassis slot, error -16 Is the command wrong? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921444 Title: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest Status in QEMU: New Bug description: KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git branch: next, commit: 4a98623d Qemu: https://git.qemu.org/git/qemu.git branch: master, commit: 9e2e9fe3 Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device to guest with qemu internal command device_add, the 1st device can be added successfully, but the 2nd device failed to hot add. If guest chipset is legacy i440fx, the 2 device can be added successfully. 1. Enable VT-d in BIOS 2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel 3. Bind 2 device to vfio-pci echo :b1:00.0 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id echo :b1:00.1 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 4. create guest with Q35 chipset: qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device pcie-root-port,id=root1 -daemonize 5. hot add the 1st device to guest successfully in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1" 6. hot add the 2nd device to guest in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1" The 2nd device doesn't be added in guest, and the 1st device is removed from guest. Guest partial log: [ 110.452272] pcieport :00:04.0: pciehp: Slot(0): Attention button pressed [ 110.453314] pcieport :00:04.0: pciehp: Slot(0) Powering on due to button press [ 110.454156] pcieport :00:04.0: pciehp: Slot(0): Card present [ 110.454792] pcieport :00:04.0: pciehp: Slot(0): Link Up [ 110.580927] pci :01:00.0: [8086:1572] type 00 class 0x02 [ 110.582560] pci :01:00.0: reg 0x10: [mem 0x-0x007f 64bit pref] [ 110.583453] pci :01:00.0: reg 0x1c: [mem 0x-0x7fff 64bit pref] [ 110.584278] pci :01:00.0: reg 0x30: [mem 0x-0x0007 pref] [ 110.585051] pci :01:00.0: Max Payload Size set to 128 (was 512, max 2048) [ 110.586621] pci :01:00.0: PME# supported from D0 D3hot D3cold [ 110.588140] pci :01:00.0: BAR 0: no space for [mem size 0x0080 64bit pref] [ 110.588954] pci :01:00.0: BAR 0: failed to assign [mem size 0x0080 64bit pref] [ 110.589797] pci :01:00.0: BAR 6: assigned [mem 0xfe80-0xfe87 pref] [ 110.590703] pci :01:00.0: BAR 3: assigned [mem 0xfe00-0xfe007fff 64bit pref] [ 110.592085] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.592755] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.594403] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.595847] pcieport :00:04.0: bridge window [mem 0xfe00-0xfe1f 64bit pref] [ 110.597867] PCI: No. 2 try to assign unassigned res [ 110.597870] release child resource [mem 0xfe00-0xfe007fff 64bit pref] [ 110.597871] pcieport :00:04.0: resource 15 [mem 0xfe00-0xfe1f 64bit pref] released [ 110.598881] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.600789] pcieport :00:04.0: BAR 15: assigned [mem 0x18000-0x180bf 64bit pref] [ 110.601731] pci :01:00.0: BAR 0: assigned [mem 0x18000-0x1807f 64bit pref] [ 110.602849] pci :01:00.0: BAR 3: assigned [mem 0x18080-0x180807fff 64bit pref] [ 110.604069] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.604941] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.606237] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.607401] pcieport :00:04.0: bridge window [mem 0x18000-0x180bf 64bit pref] [ 110.653661] i40e: Intel(R) Ethernet Connection XL710 Network Driver [ 110.654443] i40e: Copyright (c) 2013 - 2019 Intel Corporation. [ 110.655314] i40e :01:00.0: enabling device (0140 -> 0142) [ 110.672396] i40e :01:00.0: fw 6.0.48442 api 1.7 nvm 6.01 0x800035b1 1.1747.0 [8086:1572] [8086:0008] [ 110.750054] i40e
[Bug 1921444] Re: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest
You need a root port per device you want to hotplug, you cannot independently hotplug multiple devices to a PCIe link. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921444 Title: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest Status in QEMU: New Bug description: KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git branch: next, commit: 4a98623d Qemu: https://git.qemu.org/git/qemu.git branch: master, commit: 9e2e9fe3 Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device to guest with qemu internal command device_add, the 1st device can be added successfully, but the 2nd device failed to hot add. If guest chipset is legacy i440fx, the 2 device can be added successfully. 1. Enable VT-d in BIOS 2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel 3. Bind 2 device to vfio-pci echo :b1:00.0 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id echo :b1:00.1 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 4. create guest with Q35 chipset: qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device pcie-root-port,id=root1 -daemonize 5. hot add the 1st device to guest successfully in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1" 6. hot add the 2nd device to guest in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1" The 2nd device doesn't be added in guest, and the 1st device is removed from guest. Guest partial log: [ 110.452272] pcieport :00:04.0: pciehp: Slot(0): Attention button pressed [ 110.453314] pcieport :00:04.0: pciehp: Slot(0) Powering on due to button press [ 110.454156] pcieport :00:04.0: pciehp: Slot(0): Card present [ 110.454792] pcieport :00:04.0: pciehp: Slot(0): Link Up [ 110.580927] pci :01:00.0: [8086:1572] type 00 class 0x02 [ 110.582560] pci :01:00.0: reg 0x10: [mem 0x-0x007f 64bit pref] [ 110.583453] pci :01:00.0: reg 0x1c: [mem 0x-0x7fff 64bit pref] [ 110.584278] pci :01:00.0: reg 0x30: [mem 0x-0x0007 pref] [ 110.585051] pci :01:00.0: Max Payload Size set to 128 (was 512, max 2048) [ 110.586621] pci :01:00.0: PME# supported from D0 D3hot D3cold [ 110.588140] pci :01:00.0: BAR 0: no space for [mem size 0x0080 64bit pref] [ 110.588954] pci :01:00.0: BAR 0: failed to assign [mem size 0x0080 64bit pref] [ 110.589797] pci :01:00.0: BAR 6: assigned [mem 0xfe80-0xfe87 pref] [ 110.590703] pci :01:00.0: BAR 3: assigned [mem 0xfe00-0xfe007fff 64bit pref] [ 110.592085] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.592755] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.594403] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.595847] pcieport :00:04.0: bridge window [mem 0xfe00-0xfe1f 64bit pref] [ 110.597867] PCI: No. 2 try to assign unassigned res [ 110.597870] release child resource [mem 0xfe00-0xfe007fff 64bit pref] [ 110.597871] pcieport :00:04.0: resource 15 [mem 0xfe00-0xfe1f 64bit pref] released [ 110.598881] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.600789] pcieport :00:04.0: BAR 15: assigned [mem 0x18000-0x180bf 64bit pref] [ 110.601731] pci :01:00.0: BAR 0: assigned [mem 0x18000-0x1807f 64bit pref] [ 110.602849] pci :01:00.0: BAR 3: assigned [mem 0x18080-0x180807fff 64bit pref] [ 110.604069] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.604941] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.606237] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.607401] pcieport :00:04.0: bridge window [mem 0x18000-0x180bf 64bit pref] [ 110.653661] i40e: Intel(R) Ethernet Connection XL710 Network Driver [ 110.654443] i40e: Copyright (c) 2013 - 2019 Intel Corporation. [ 110.655314] i40e :01:00.0: enabling device (0140 -> 0142) [ 110.672396] i40e :01:00.0: fw 6.0.48442 api 1.7 nvm 6.01 0x800035b1 1.1747.0 [8086:1572] [8086:0008] [ 110.750054] i40e :01:00.0: MAC address: 3c:fd:fe:c0:59:98 [ 110.751792] i40e :01:00.0: FW LLDP is enabled [ 110.764644] i40e :01:00.0 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None [ 110.779390] i40e :01:00.0: PCI-Express: Speed 8.0GT/s Width x8 [ 110.789841] i40e :01:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 4 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA [ 111.817553] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 205.13
[Bug 1921444] Re: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest
** Attachment added: "guest_dmesg.log" https://bugs.launchpad.net/qemu/+bug/1921444/+attachment/5481022/+files/guest_dmesg.log -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921444 Title: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest Status in QEMU: New Bug description: KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git branch: next, commit: 4a98623d Qemu: https://git.qemu.org/git/qemu.git branch: master, commit: 9e2e9fe3 Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device to guest with qemu internal command device_add, the 1st device can be added successfully, but the 2nd device failed to hot add. If guest chipset is legacy i440fx, the 2 device can be added successfully. 1. Enable VT-d in BIOS 2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel 3. Bind 2 device to vfio-pci echo :b1:00.0 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id echo :b1:00.1 > /sys/bus/pci/drivers/i40e/unbind echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 4. create guest with Q35 chipset: qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device pcie-root-port,id=root1 -daemonize 5. hot add the 1st device to guest successfully in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1" 6. hot add the 2nd device to guest in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1" The 2nd device doesn't be added in guest, and the 1st device is removed from guest. Guest partial log: [ 110.452272] pcieport :00:04.0: pciehp: Slot(0): Attention button pressed [ 110.453314] pcieport :00:04.0: pciehp: Slot(0) Powering on due to button press [ 110.454156] pcieport :00:04.0: pciehp: Slot(0): Card present [ 110.454792] pcieport :00:04.0: pciehp: Slot(0): Link Up [ 110.580927] pci :01:00.0: [8086:1572] type 00 class 0x02 [ 110.582560] pci :01:00.0: reg 0x10: [mem 0x-0x007f 64bit pref] [ 110.583453] pci :01:00.0: reg 0x1c: [mem 0x-0x7fff 64bit pref] [ 110.584278] pci :01:00.0: reg 0x30: [mem 0x-0x0007 pref] [ 110.585051] pci :01:00.0: Max Payload Size set to 128 (was 512, max 2048) [ 110.586621] pci :01:00.0: PME# supported from D0 D3hot D3cold [ 110.588140] pci :01:00.0: BAR 0: no space for [mem size 0x0080 64bit pref] [ 110.588954] pci :01:00.0: BAR 0: failed to assign [mem size 0x0080 64bit pref] [ 110.589797] pci :01:00.0: BAR 6: assigned [mem 0xfe80-0xfe87 pref] [ 110.590703] pci :01:00.0: BAR 3: assigned [mem 0xfe00-0xfe007fff 64bit pref] [ 110.592085] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.592755] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.594403] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.595847] pcieport :00:04.0: bridge window [mem 0xfe00-0xfe1f 64bit pref] [ 110.597867] PCI: No. 2 try to assign unassigned res [ 110.597870] release child resource [mem 0xfe00-0xfe007fff 64bit pref] [ 110.597871] pcieport :00:04.0: resource 15 [mem 0xfe00-0xfe1f 64bit pref] released [ 110.598881] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.600789] pcieport :00:04.0: BAR 15: assigned [mem 0x18000-0x180bf 64bit pref] [ 110.601731] pci :01:00.0: BAR 0: assigned [mem 0x18000-0x1807f 64bit pref] [ 110.602849] pci :01:00.0: BAR 3: assigned [mem 0x18080-0x180807fff 64bit pref] [ 110.604069] pcieport :00:04.0: PCI bridge to [bus 01] [ 110.604941] pcieport :00:04.0: bridge window [io 0x1000-0x1fff] [ 110.606237] pcieport :00:04.0: bridge window [mem 0xfe80-0xfe9f] [ 110.607401] pcieport :00:04.0: bridge window [mem 0x18000-0x180bf 64bit pref] [ 110.653661] i40e: Intel(R) Ethernet Connection XL710 Network Driver [ 110.654443] i40e: Copyright (c) 2013 - 2019 Intel Corporation. [ 110.655314] i40e :01:00.0: enabling device (0140 -> 0142) [ 110.672396] i40e :01:00.0: fw 6.0.48442 api 1.7 nvm 6.01 0x800035b1 1.1747.0 [8086:1572] [8086:0008] [ 110.750054] i40e :01:00.0: MAC address: 3c:fd:fe:c0:59:98 [ 110.751792] i40e :01:00.0: FW LLDP is enabled [ 110.764644] i40e :01:00.0 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None [ 110.779390] i40e :01:00.0: PCI-Express: Speed 8.0GT/s Width x8 [ 110.789841] i40e :01:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 4 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA [ 111.817553] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
