Hi, It seems to me that there is a scale issue O(n) in memory_region_transaction_commit().
Basically the time it takes to rebuild the memory view during device assignment pci_bridge_update_mappings() increase linearly with respect to the number of device already assigned to the guest. I'm running on a recent qemu.git and I merged from git://github.com/bonzini/qemu.git memory the following patches that seems to be related to the scale issue I'm facing: Fam Zheng (1): memory: Don't call memory_region_update_coalesced_range if nothing changed Gonglei (1): memory: Don't update all memory region when ioeventfd changed Those patches help but don't fix the issue. The problem become more noticeable when lots of device are being assigned to the guest. I'm running my test on a QEMU q35 machine with the following topology: ioh3420 ( root port ) x3130-upstream xio3130-downstream xio3130-downstream xio3130-downstream ... I have added instrumentation in kvm_cpu_exec() to track to amount of time spend in the emulation ( patch at the end but not relevant for this discussion ) Here what I see when I assign device one after to other. NOTE the time-stamp is in msec. The linear increase in the time comes from memory_region_transaction_commit(). (qemu) device_add pci-assign,host=28:10.1,bus=pciehp.3.7 QEMU long exit vCPU 0 25 2 QEMU long exit vCPU 0 22 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 22 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 21 2 QEMU long exit vCPU 0 45 2 <<< QEMU long exit vCPU 0 23 2 (qemu) device_add pci-assign,host=28:10.2,bus=pciehp.3.8 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 24 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 23 2 QEMU long exit vCPU 0 49 2 <<< QEMU long exit vCPU 0 25 2 (qemu) device_add pci-assign,host=28:10.3,bus=pciehp.3.9 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 25 2 QEMU long exit vCPU 0 25 2 QEMU long exit vCPU 0 25 2 QEMU long exit vCPU 0 25 2 QEMU long exit vCPU 0 24 2 QEMU long exit vCPU 0 24 2 QEMU long exit vCPU 0 24 2 QEMU long exit vCPU 0 24 2 QEMU long exit vCPU 0 24 2 QEMU long exit vCPU 0 52 2 <<< QEMU long exit vCPU 0 26 2 (qemu) device_add pci-assign,host=28:10.4,bus=pciehp.3.10 QEMU long exit vCPU 0 35 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 27 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 26 2 QEMU long exit vCPU 0 56 2 <<< QEMU long exit vCPU 0 28 2 (qemu) device_add pci-assign,host=28:10.5,bus=pciehp.3.11 QEMU long exit vCPU 0 33 2 QEMU long exit vCPU 0 30 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 29 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 28 2 QEMU long exit vCPU 0 59 2 <<< QEMU long exit vCPU 0 30 2 diff --git a/kvm-all.c b/kvm-all.c index 3ae30ee..e3a1964 100644 --- a/kvm-all.c +++ b/kvm-all.c @@ -1685,6 +1685,8 @@ int kvm_cpu_exec(CPUState *cpu) { struct kvm_run *run = cpu->kvm_run; int ret, run_ret; + int64_t clock_ns, delta_ms; + __u32 last_exit_reason, last_vcpu; DPRINTF("kvm_cpu_exec()\n"); @@ -1711,6 +1713,12 @@ int kvm_cpu_exec(CPUState *cpu) } qemu_mutex_unlock_iothread(); + delta_ms = (get_clock() - clock_ns)/(1000*1000); + if( delta_ms >= 10){ + fprintf(stderr, "QEMU long exit vCPU %d %ld %d\n",last_vcpu, + delta_ms, last_exit_reason); + } + run_ret = kvm_vcpu_ioctl(cpu, KVM_RUN, 0); qemu_mutex_lock_iothread(); @@ -1727,7 +1735,15 @@ int kvm_cpu_exec(CPUState *cpu) abort(); } + /* + * Capture exit reasons + */ + clock_ns = get_clock(); + last_exit_reason = run->exit_reason; + last_vcpu = cpu->cpu_index; + thanks, Etienne