Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
On Fri, 19 May 2017 at 16:10, Jay Zhou wrote: > > Hi Paolo and Wanpeng, > > On 2017/5/17 16:38, Wanpeng Li wrote: > > 2017-05-17 15:43 GMT+08:00 Paolo Bonzini : > >>> Recently, I have tested the performance before migration and after > >>> migration failure > >>> using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard > >>> performance > >>> evaluation tool. > >>> > >>> These are the steps: > >>> == > >>> (1) the version of kmod is 4.4.11(with slightly modified) and the > >>> version of > >>> qemu is 2.6.0 > >>> (with slightly modified), the kmod is applied with the following > >>> patch > >>> > >>> diff --git a/source/x86/x86.c b/source/x86/x86.c > >>> index 054a7d3..75a4bb3 100644 > >>> --- a/source/x86/x86.c > >>> +++ b/source/x86/x86.c > >>> @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, > >>> */ > >>> if ((change != KVM_MR_DELETE) && > >>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && > >>> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) > >>> - kvm_mmu_zap_collapsible_sptes(kvm, new); > >>> + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { > >>> + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); > >>> + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); > >>> + } > >>> > >>> /* > >>> * Set up write protection and/or dirty logging for the new slot. > >> > >> Try these modifications to the setup: > >> > >> 1) set up 1G hugetlbfs hugepages and use those for the guest's memory > >> > >> 2) test both without and with the above patch. > >> > > In order to avoid random memory allocation issues, I reran the test cases: > (1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a > pcpu respectively, these resources(memory and pcpu) allocated to VM are all > from NUMA node 0 > (2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration, and > get a result. And then, migration failure is constructed. At last, I run the > test case again, and get an another result. > (3) results: > Host hugepages THP on(2M) THP on(2M) THP on(2M) THP on(2M) > Patchpatch1 patch2 patch3 - > Before migration No No No Yes > After migration failed Yes Yes Yes No > Largepages 67->186262->1890 95->1865 1926 > score of 429.mcf 189 188 188 189 > > Host hugepages 1G hugepages 1G hugepages 1G hugepages 1G > hugepages > Patchpatch1patch2patch3- > Before migration NoNoNoYes > After migration failed Yes Yes Yes No > Largepages 21212639 > score of 429.mcf 188 188 186 188 > > Notes: > patch1 means with "lazy collapse small sptes into large sptes" codes > patch2 means comment out "lazy collapse small sptes into large sptes" codes > patch3 means using kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD) > instead of kvm_mmu_zap_collapsible_sptes(kvm, new) > > "Largepages" means the value of /sys/kernel/debug/kvm/largepages > > > In addition, we can compare /sys/kernel/debug/kvm/largepages w/ and > > w/o the patch. IIRC, /sys/kernel/debug/kvm/largepages will drop during > > live migration, it will keep a small value if live migration fails and > > w/o "lazy collapse small sptes into large sptes" codes, however, it > > will increase gradually if w/ the "lazy collapse small sptes into > > large sptes" codes. > > > > No, without the "lazy collapse small sptes into large sptes" codes, > /sys/kernel/debug/kvm/largepages does drop during live migration, > but it still will increase gradually if live migration fails, see the result > above. I printed out the back trace when it increases after migration failure, > > [139574.369098] [] dump_stack+0x19/0x1b > [139574.369111] [] mmu_set_spte+0x2f6/0x310 [kvm] > [139574.369122] [] __direct_map.isra.109+0x1de/0x250 [kvm] > [139574.369133] [] tdp_page_fault+0x246/0x280 [kvm] > [139574.369144] [] kvm_mmu_page_fault+0x24/0x130 [kvm] > [139574.369148] [] handle_ept_violation+0x96/0x170 > [kvm_intel] > [139574.369153] [] vmx_handle_exit+0x299/0xbf0 [kvm_intel] > [139574.369157] [] ? uv_bau_message_intr1+0x80/0x80 > [139574.369161] [] ? vmx_inject_irq+0xf0/0xf0 [kvm_intel] > [139574.369172] [] vcpu_enter_guest+0x76d/0x1160 [kvm] > [139574.369184] [] ? kvm_apic_local_deliver+0x65/0x70 [kvm] > [139574.369196] [] kvm_arch_vcpu_ioctl_run+0xd5/0x440 [kvm] > [139574.369205] [] kvm_vcpu_ioctl+0x2b1/0x640 [kvm] > [139574.369209] [] ? do_futex+0x122/0x5b0 > [139574.369212] [] do_vfs_ioctl+0x2e5/0x4c0 > [139574.369223] [] ? kvm_on_user_return+0x75/0xb0 [kvm] > [139574.369225] [] SyS_ioctl+0xa
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi Xiao, On 2017/5/19 16:32, Xiao Guangrong wrote: I do not know why i was removed from the list. I was CCed to you... Your comments are very valuable to us, and thank for your quick response. On 05/19/2017 04:09 PM, Jay Zhou wrote: Hi Paolo and Wanpeng, On 2017/5/17 16:38, Wanpeng Li wrote: 2017-05-17 15:43 GMT+08:00 Paolo Bonzini : Recently, I have tested the performance before migration and after migration failure using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard performance evaluation tool. These are the steps: == (1) the version of kmod is 4.4.11(with slightly modified) and the version of qemu is 2.6.0 (with slightly modified), the kmod is applied with the following patch diff --git a/source/x86/x86.c b/source/x86/x86.c index 054a7d3..75a4bb3 100644 --- a/source/x86/x86.c +++ b/source/x86/x86.c @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); + } /* * Set up write protection and/or dirty logging for the new slot. Try these modifications to the setup: 1) set up 1G hugetlbfs hugepages and use those for the guest's memory 2) test both without and with the above patch. In order to avoid random memory allocation issues, I reran the test cases: (1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a pcpu respectively, these resources(memory and pcpu) allocated to VM are all from NUMA node 0 (2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration, and get a result. And then, migration failure is constructed. At last, I run the test case again, and get an another result. I guess this case purely writes the memory, that means the readonly mappings will Yes, I printed out the speed of dirty page rate, it is about 1GB per second. always be dropped by #PF, then huge mappings are established. If benchmark memory read, you show observe its difference. OK, thank for your suggestion! Regards, Jay Zhou
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
I do not know why i was removed from the list. On 05/19/2017 04:09 PM, Jay Zhou wrote: Hi Paolo and Wanpeng, On 2017/5/17 16:38, Wanpeng Li wrote: 2017-05-17 15:43 GMT+08:00 Paolo Bonzini : Recently, I have tested the performance before migration and after migration failure using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard performance evaluation tool. These are the steps: == (1) the version of kmod is 4.4.11(with slightly modified) and the version of qemu is 2.6.0 (with slightly modified), the kmod is applied with the following patch diff --git a/source/x86/x86.c b/source/x86/x86.c index 054a7d3..75a4bb3 100644 --- a/source/x86/x86.c +++ b/source/x86/x86.c @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); + } /* * Set up write protection and/or dirty logging for the new slot. Try these modifications to the setup: 1) set up 1G hugetlbfs hugepages and use those for the guest's memory 2) test both without and with the above patch. In order to avoid random memory allocation issues, I reran the test cases: (1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a pcpu respectively, these resources(memory and pcpu) allocated to VM are all from NUMA node 0 (2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration, and get a result. And then, migration failure is constructed. At last, I run the test case again, and get an another result. I guess this case purely writes the memory, that means the readonly mappings will always be dropped by #PF, then huge mappings are established. If benchmark memory read, you show observe its difference. Thanks!
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi Paolo and Wanpeng, On 2017/5/17 16:38, Wanpeng Li wrote: 2017-05-17 15:43 GMT+08:00 Paolo Bonzini : Recently, I have tested the performance before migration and after migration failure using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard performance evaluation tool. These are the steps: == (1) the version of kmod is 4.4.11(with slightly modified) and the version of qemu is 2.6.0 (with slightly modified), the kmod is applied with the following patch diff --git a/source/x86/x86.c b/source/x86/x86.c index 054a7d3..75a4bb3 100644 --- a/source/x86/x86.c +++ b/source/x86/x86.c @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); + } /* * Set up write protection and/or dirty logging for the new slot. Try these modifications to the setup: 1) set up 1G hugetlbfs hugepages and use those for the guest's memory 2) test both without and with the above patch. In order to avoid random memory allocation issues, I reran the test cases: (1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a pcpu respectively, these resources(memory and pcpu) allocated to VM are all from NUMA node 0 (2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration, and get a result. And then, migration failure is constructed. At last, I run the test case again, and get an another result. (3) results: Host hugepages THP on(2M) THP on(2M) THP on(2M) THP on(2M) Patchpatch1 patch2 patch3 - Before migration No No No Yes After migration failed Yes Yes Yes No Largepages 67->186262->1890 95->1865 1926 score of 429.mcf 189 188 188 189 Host hugepages 1G hugepages 1G hugepages 1G hugepages 1G hugepages Patchpatch1patch2patch3- Before migration NoNoNoYes After migration failed Yes Yes Yes No Largepages 21212639 score of 429.mcf 188 188 186 188 Notes: patch1 means with "lazy collapse small sptes into large sptes" codes patch2 means comment out "lazy collapse small sptes into large sptes" codes patch3 means using kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD) instead of kvm_mmu_zap_collapsible_sptes(kvm, new) "Largepages" means the value of /sys/kernel/debug/kvm/largepages In addition, we can compare /sys/kernel/debug/kvm/largepages w/ and w/o the patch. IIRC, /sys/kernel/debug/kvm/largepages will drop during live migration, it will keep a small value if live migration fails and w/o "lazy collapse small sptes into large sptes" codes, however, it will increase gradually if w/ the "lazy collapse small sptes into large sptes" codes. No, without the "lazy collapse small sptes into large sptes" codes, /sys/kernel/debug/kvm/largepages does drop during live migration, but it still will increase gradually if live migration fails, see the result above. I printed out the back trace when it increases after migration failure, [139574.369098] [] dump_stack+0x19/0x1b [139574.369111] [] mmu_set_spte+0x2f6/0x310 [kvm] [139574.369122] [] __direct_map.isra.109+0x1de/0x250 [kvm] [139574.369133] [] tdp_page_fault+0x246/0x280 [kvm] [139574.369144] [] kvm_mmu_page_fault+0x24/0x130 [kvm] [139574.369148] [] handle_ept_violation+0x96/0x170 [kvm_intel] [139574.369153] [] vmx_handle_exit+0x299/0xbf0 [kvm_intel] [139574.369157] [] ? uv_bau_message_intr1+0x80/0x80 [139574.369161] [] ? vmx_inject_irq+0xf0/0xf0 [kvm_intel] [139574.369172] [] vcpu_enter_guest+0x76d/0x1160 [kvm] [139574.369184] [] ? kvm_apic_local_deliver+0x65/0x70 [kvm] [139574.369196] [] kvm_arch_vcpu_ioctl_run+0xd5/0x440 [kvm] [139574.369205] [] kvm_vcpu_ioctl+0x2b1/0x640 [kvm] [139574.369209] [] ? do_futex+0x122/0x5b0 [139574.369212] [] do_vfs_ioctl+0x2e5/0x4c0 [139574.369223] [] ? kvm_on_user_return+0x75/0xb0 [kvm] [139574.369225] [] SyS_ioctl+0xa1/0xc0 [139574.369229] [] system_call_fastpath+0x16/0x1b Any suggestion will be appreciated, Thanks! Regards, Jay Zhou
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
2017-05-17 15:43 GMT+08:00 Paolo Bonzini : >> Recently, I have tested the performance before migration and after migration >> failure >> using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard >> performance >> evaluation tool. >> >> These are the steps: >> == >> (1) the version of kmod is 4.4.11(with slightly modified) and the version of >> qemu is 2.6.0 >> (with slightly modified), the kmod is applied with the following patch >> >> diff --git a/source/x86/x86.c b/source/x86/x86.c >> index 054a7d3..75a4bb3 100644 >> --- a/source/x86/x86.c >> +++ b/source/x86/x86.c >> @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, >> */ >> if ((change != KVM_MR_DELETE) && >> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && >> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) >> - kvm_mmu_zap_collapsible_sptes(kvm, new); >> + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { >> + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); >> + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); >> + } >> >> /* >> * Set up write protection and/or dirty logging for the new slot. > > Try these modifications to the setup: > > 1) set up 1G hugetlbfs hugepages and use those for the guest's memory > > 2) test both without and with the above patch. > In addition, we can compare /sys/kernel/debug/kvm/largepages w/ and w/o the patch. IIRC, /sys/kernel/debug/kvm/largepages will drop during live migration, it will keep a small value if live migration fails and w/o "lazy collapse small sptes into large sptes" codes, however, it will increase gradually if w/ the "lazy collapse small sptes into large sptes" codes. Regards, Wanpeng Li
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
> Recently, I have tested the performance before migration and after migration > failure > using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard > performance > evaluation tool. > > These are the steps: > == > (1) the version of kmod is 4.4.11(with slightly modified) and the version of > qemu is 2.6.0 > (with slightly modified), the kmod is applied with the following patch > > diff --git a/source/x86/x86.c b/source/x86/x86.c > index 054a7d3..75a4bb3 100644 > --- a/source/x86/x86.c > +++ b/source/x86/x86.c > @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, > */ > if ((change != KVM_MR_DELETE) && > (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && > - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) > - kvm_mmu_zap_collapsible_sptes(kvm, new); > + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { > + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); > + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); > + } > > /* > * Set up write protection and/or dirty logging for the new slot. Try these modifications to the setup: 1) set up 1G hugetlbfs hugepages and use those for the guest's memory 2) test both without and with the above patch. Thanks, Paolo
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
On 2017/5/17 13:47, Wanpeng Li wrote: Hi Zhoujian, 2017-05-17 10:20 GMT+08:00 Zhoujian (jay) : Hi Wanpeng, On 11/05/2017 14:07, Zhoujian (jay) wrote: -* Scan sptes if dirty logging has been stopped, dropping those -* which can be collapsed into a single large-page spte. Later -* page faults will create the large-page sptes. +* Reset each vcpu's mmu, then page faults will create the large-page +* sptes later. */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); This is an unlikely branch(unless guest live migration fails and continue to run on the source machine) instead of hot path, do you have any performance number for your real workloads? Sorry to bother you again. Recently, I have tested the performance before migration and after migration failure using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard performance evaluation tool. These are the results: ** Before migration the score is 153, and the TLB miss statistics of the qemu process is: linux-sjrfac:/mnt/zhoujian # perf stat -e dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 Performance counter stats for process id '26463': 698,938 dTLB-load-misses #0.13% of all dTLB cache hits (50.46%) 543,303,875 dTLB-loads (50.43%) 199,597 dTLB-store-misses (16.51%) 60,128,561 dTLB-stores (16.67%) 69,986 iTLB-load-misses #6.17% of all iTLB cache hits (16.67%) 1,134,097 iTLB-loads (33.33%) 10.000684064 seconds time elapsed After migration failure the score is 149, and the TLB miss statistics of the qemu process is: linux-sjrfac:/mnt/zhoujian # perf stat -e dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 Performance counter stats for process id '26463': 765,400 dTLB-load-misses #0.14% of all dTLB cache hits (50.50%) 540,972,144 dTLB-loads (50.47%) 207,670 dTLB-store-misses (16.50%) 58,363,787 dTLB-stores (16.67%) 109,772 iTLB-load-misses #9.52% of all iTLB cache hits (16.67%) 1,152,784 iTLB-loads (33.32%) 10.000703078 seconds time elapsed ** Could you comment out the original "lazy collapse small sptes into large sptes" codes in the function kvm_arch_commit_memory_region() and post the results here? With the patch below, diff --git a/source/x86/x86.c b/source/x86/x86.c index 054a7d3..e0288d5 100644 --- a/source/x86/x86.c +++ b/source/x86/x86.c @@ -8548,10 +8548,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, * which can be collapsed into a single large-page spte. Later * page faults will create the large-page sptes. */ - if ((change != KVM_MR_DELETE) && - (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); /* * Set up write protection and/or dirty logging for the new slot. After migration failure the score is 148, and the TLB miss statistics of the qemu process is: linux-sjrfac:/mnt/zhoujian # perf stat -e dTLB-load-misses,dTLB-loads,dTLB-store-misses,dTLB-stores,iTLB-load-misses,iTLB-loads -p 12432 sleep 10 Performance counter stats for process id '12432': 1,052,697 dTLB-load-misses #0.19% of all dTLB cache hits (50.45%) 551,828,702 dTLB-loads (50.46%) 147,228 dTLB-store-misses (16.55%) 60,427,834 dTLB-stores (16.50%) 93,793 iTLB-load-misses #7.43% of all iTLB cache hits (16.67%) 1,262,137 iTLB-loads (33.33%) 10.000709900 seconds time elapsed Regards, Jay Zhou Regards, Wanpeng Li These are the steps: == (1) the version of kmod is 4.4.11(with slightly modified) and the version of qemu is 2.6.0 (with slightly modified), the kmod is applied with the following patch according to Paolo's advice: diff --git a/source/x86/x86.c b/source/x86/x86.c index 054a7d3..75a4bb3 100644 --- a/source/x
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi Zhoujian, 2017-05-17 10:20 GMT+08:00 Zhoujian (jay) : > Hi Wanpeng, > >> > On 11/05/2017 14:07, Zhoujian (jay) wrote: >> >> -* Scan sptes if dirty logging has been stopped, dropping those >> >> -* which can be collapsed into a single large-page spte. Later >> >> -* page faults will create the large-page sptes. >> >> +* Reset each vcpu's mmu, then page faults will create the >> large-page >> >> +* sptes later. >> >> */ >> >> if ((change != KVM_MR_DELETE) && >> >> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && >> >> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) >> >> - kvm_mmu_zap_collapsible_sptes(kvm, new); >> >> This is an unlikely branch(unless guest live migration fails and continue >> to run on the source machine) instead of hot path, do you have any >> performance number for your real workloads? >> > > Sorry to bother you again. > > Recently, I have tested the performance before migration and after migration > failure > using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard > performance > evaluation tool. > > These are the results: > ** > Before migration the score is 153, and the TLB miss statistics of the > qemu process is: > linux-sjrfac:/mnt/zhoujian # perf stat -e > dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ > dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 > > Performance counter stats for process id '26463': > >698,938 dTLB-load-misses #0.13% of all dTLB > cache hits (50.46%) >543,303,875 dTLB-loads > (50.43%) >199,597 dTLB-store-misses > (16.51%) > 60,128,561 dTLB-stores > (16.67%) > 69,986 iTLB-load-misses #6.17% of all iTLB > cache hits (16.67%) > 1,134,097 iTLB-loads > (33.33%) > > 10.000684064 seconds time elapsed > > After migration failure the score is 149, and the TLB miss statistics of > the qemu process is: > linux-sjrfac:/mnt/zhoujian # perf stat -e > dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ > dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 > > Performance counter stats for process id '26463': > >765,400 dTLB-load-misses #0.14% of all dTLB > cache hits (50.50%) >540,972,144 dTLB-loads > (50.47%) >207,670 dTLB-store-misses > (16.50%) > 58,363,787 dTLB-stores > (16.67%) >109,772 iTLB-load-misses #9.52% of all iTLB > cache hits (16.67%) > 1,152,784 iTLB-loads > (33.32%) > > 10.000703078 seconds time elapsed > ** Could you comment out the original "lazy collapse small sptes into large sptes" codes in the function kvm_arch_commit_memory_region() and post the results here? Regards, Wanpeng Li > > These are the steps: > == > (1) the version of kmod is 4.4.11(with slightly modified) and the version of > qemu is 2.6.0 > (with slightly modified), the kmod is applied with the following patch > according to > Paolo's advice: > > diff --git a/source/x86/x86.c b/source/x86/x86.c > index 054a7d3..75a4bb3 100644 > --- a/source/x86/x86.c > +++ b/source/x86/x86.c > @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, > */ > if ((change != KVM_MR_DELETE) && > (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && > - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) > - kvm_mmu_zap_collapsible_sptes(kvm, new); > + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { > + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); > + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); > + } > > /* > * Set up write protection and/or dirty logging for the new slot. > > (2) I started up a memory preoccupied 10G VM(suse11sp3), which means its "RES > column" in top is 10G, > in order to set up the EPT table in advance. > (3) And then, I run the test case 429.mcf of spec cpu2006 before migration > and after migration failure. > The 429.mcf is a memory intensive workload, and the migration failure is > constructed deliberately > with the following patch of qemu: > > diff --git a/migration/migration.c b/migration/migration.c > index 5d725d0..88dfc59 100644 > --- a/migration/migration.c > +++ b/migration/migration.c > @@ -625,6 +625,9 @@ static void process_incoming_migration_co(void *opaque) >MIGRATION_STATUS_
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi Wanpeng, > > On 11/05/2017 14:07, Zhoujian (jay) wrote: > >> -* Scan sptes if dirty logging has been stopped, dropping those > >> -* which can be collapsed into a single large-page spte. Later > >> -* page faults will create the large-page sptes. > >> +* Reset each vcpu's mmu, then page faults will create the > large-page > >> +* sptes later. > >> */ > >> if ((change != KVM_MR_DELETE) && > >> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && > >> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) > >> - kvm_mmu_zap_collapsible_sptes(kvm, new); > > This is an unlikely branch(unless guest live migration fails and continue > to run on the source machine) instead of hot path, do you have any > performance number for your real workloads? > Sorry to bother you again. Recently, I have tested the performance before migration and after migration failure using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard performance evaluation tool. These are the results: ** Before migration the score is 153, and the TLB miss statistics of the qemu process is: linux-sjrfac:/mnt/zhoujian # perf stat -e dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 Performance counter stats for process id '26463': 698,938 dTLB-load-misses #0.13% of all dTLB cache hits (50.46%) 543,303,875 dTLB-loads (50.43%) 199,597 dTLB-store-misses (16.51%) 60,128,561 dTLB-stores (16.67%) 69,986 iTLB-load-misses #6.17% of all iTLB cache hits (16.67%) 1,134,097 iTLB-loads (33.33%) 10.000684064 seconds time elapsed After migration failure the score is 149, and the TLB miss statistics of the qemu process is: linux-sjrfac:/mnt/zhoujian # perf stat -e dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 Performance counter stats for process id '26463': 765,400 dTLB-load-misses #0.14% of all dTLB cache hits (50.50%) 540,972,144 dTLB-loads (50.47%) 207,670 dTLB-store-misses (16.50%) 58,363,787 dTLB-stores (16.67%) 109,772 iTLB-load-misses #9.52% of all iTLB cache hits (16.67%) 1,152,784 iTLB-loads (33.32%) 10.000703078 seconds time elapsed ** These are the steps: == (1) the version of kmod is 4.4.11(with slightly modified) and the version of qemu is 2.6.0 (with slightly modified), the kmod is applied with the following patch according to Paolo's advice: diff --git a/source/x86/x86.c b/source/x86/x86.c index 054a7d3..75a4bb3 100644 --- a/source/x86/x86.c +++ b/source/x86/x86.c @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); + } /* * Set up write protection and/or dirty logging for the new slot. (2) I started up a memory preoccupied 10G VM(suse11sp3), which means its "RES column" in top is 10G, in order to set up the EPT table in advance. (3) And then, I run the test case 429.mcf of spec cpu2006 before migration and after migration failure. The 429.mcf is a memory intensive workload, and the migration failure is constructed deliberately with the following patch of qemu: diff --git a/migration/migration.c b/migration/migration.c index 5d725d0..88dfc59 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -625,6 +625,9 @@ static void process_incoming_migration_co(void *opaque) MIGRATION_STATUS_ACTIVE); ret = qemu_loadvm_state(f); +// deliberately construct the migration failure +exit(EXIT_FAILURE); + ps = postcopy_state_get(); trace_process_incoming_migration_co_end(ret, ps); if (ps != POSTCOPY_INCOMING_NONE) { == Results of the score and TLB miss rate are almost the same, and I am confused. May I ask which tool do you use to evaluate the performance? And if my test steps are wrong, please
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
On 2017/5/12 16:09, Xiao Guangrong wrote: On 05/11/2017 08:24 PM, Paolo Bonzini wrote: On 11/05/2017 14:07, Zhoujian (jay) wrote: -* Scan sptes if dirty logging has been stopped, dropping those -* which can be collapsed into a single large-page spte. Later -* page faults will create the large-page sptes. +* Reset each vcpu's mmu, then page faults will create the large-page +* sptes later. */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_mmu_reset_context(vcpu); This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but I am not sure it is enough. I think that if you do not zap the SPTEs, the page faults will use 4K SPTEs, not large ones (though I'd have to check better; CCing Xiao and Wanpeng). Yes, Paolo is right. kvm_mmu_reset_context() just reloads vCPU's root page table, 4k mappings are still kept. There are two issues reported: - one is kvm_mmu_slot_apply_flags(), when enable dirty log tracking. Its root cause is kvm_mmu_slot_remove_write_access() takes too much time. We can make the code adaptive to use the new fast-write-protect faculty introduced by my patchset, i.e, if the number of pages contained in this memslot is more than > TOTAL * FAST_WRITE_PROTECT_PAGE_PERCENTAGE, then we use fast-write-protect instead. - another one is kvm_mmu_zap_collapsible_sptes() when disable dirty log tracking. collapsible_sptes zaps 4k mappings to make memory-read happy, it is not required by the semanteme of KVM_SET_USER_MEMORY_REGION and it is not urgent for vCPU's running, it could be done in a separate thread and use lock-break technology. How about move the action of stopping dirty log into migrate_fd_cleanup() directly, which is processed in main thread as BH after migration is completed? It will not has any side-effect even migration is failed, Or users cancel migration, No ? Thanks! .
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
On 05/11/2017 08:24 PM, Paolo Bonzini wrote: On 11/05/2017 14:07, Zhoujian (jay) wrote: -* Scan sptes if dirty logging has been stopped, dropping those -* which can be collapsed into a single large-page spte. Later -* page faults will create the large-page sptes. +* Reset each vcpu's mmu, then page faults will create the large-page +* sptes later. */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_mmu_reset_context(vcpu); This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but I am not sure it is enough. I think that if you do not zap the SPTEs, the page faults will use 4K SPTEs, not large ones (though I'd have to check better; CCing Xiao and Wanpeng). Yes, Paolo is right. kvm_mmu_reset_context() just reloads vCPU's root page table, 4k mappings are still kept. There are two issues reported: - one is kvm_mmu_slot_apply_flags(), when enable dirty log tracking. Its root cause is kvm_mmu_slot_remove_write_access() takes too much time. We can make the code adaptive to use the new fast-write-protect faculty introduced by my patchset, i.e, if the number of pages contained in this memslot is more than > TOTAL * FAST_WRITE_PROTECT_PAGE_PERCENTAGE, then we use fast-write-protect instead. - another one is kvm_mmu_zap_collapsible_sptes() when disable dirty log tracking. collapsible_sptes zaps 4k mappings to make memory-read happy, it is not required by the semanteme of KVM_SET_USER_MEMORY_REGION and it is not urgent for vCPU's running, it could be done in a separate thread and use lock-break technology. Thanks!
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
2017-05-11 22:18 GMT+08:00 Zhoujian (jay) : > Hi Wanpeng, > >> 2017-05-11 21:43 GMT+08:00 Wanpeng Li : >> > 2017-05-11 20:24 GMT+08:00 Paolo Bonzini : >> >> >> >> >> >> On 11/05/2017 14:07, Zhoujian (jay) wrote: >> >>> -* Scan sptes if dirty logging has been stopped, dropping >> those >> >>> -* which can be collapsed into a single large-page spte. >> Later >> >>> -* page faults will create the large-page sptes. >> >>> +* Reset each vcpu's mmu, then page faults will create the >> large-page >> >>> +* sptes later. >> >>> */ >> >>> if ((change != KVM_MR_DELETE) && >> >>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && >> >>> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) >> >>> - kvm_mmu_zap_collapsible_sptes(kvm, new); >> > >> > This is an unlikely branch(unless guest live migration fails and >> > continue to run on the source machine) instead of hot path, do you >> > have any performance number for your real workloads? >> >> I find the original discussion by google. >> https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg04143.html >> You will not go to this branch if the guest live migration successfully. > > In our tests, this branch is taken when living migration is successful. > AFAIK, the kmod does not know whether living migration successful or not > when dealing with KVM_SET_USER_MEMORY_REGION ioctl. Do I miss something? Original there is a bug which will not clear memslot dirty log flag after live migration fails, a patch is submitted to fix it, https://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg00794.html, however, I can't remember whether the dirty log flag will be cleared if live migration complete successfully at that time, but maybe not. Paolo replied to the patch he has a better method. Then I'm too busy and didn't follow the qemu patch for this fix any more, I just find this commit is merged currently: http://git.qemu.org/?p=qemu.git;a=commit;h=6f6a5ef3e429f92f987678ea8c396aab4dc6aa19. This commit will clear memslot dirty log flag after live migration no matter whether it is successful or not. Regards, Wanpeng Li
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi Wanpeng, > 2017-05-11 21:43 GMT+08:00 Wanpeng Li : > > 2017-05-11 20:24 GMT+08:00 Paolo Bonzini : > >> > >> > >> On 11/05/2017 14:07, Zhoujian (jay) wrote: > >>> -* Scan sptes if dirty logging has been stopped, dropping > those > >>> -* which can be collapsed into a single large-page spte. > Later > >>> -* page faults will create the large-page sptes. > >>> +* Reset each vcpu's mmu, then page faults will create the > large-page > >>> +* sptes later. > >>> */ > >>> if ((change != KVM_MR_DELETE) && > >>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && > >>> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) > >>> - kvm_mmu_zap_collapsible_sptes(kvm, new); > > > > This is an unlikely branch(unless guest live migration fails and > > continue to run on the source machine) instead of hot path, do you > > have any performance number for your real workloads? > > I find the original discussion by google. > https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg04143.html > You will not go to this branch if the guest live migration successfully. In our tests, this branch is taken when living migration is successful. AFAIK, the kmod does not know whether living migration successful or not when dealing with KVM_SET_USER_MEMORY_REGION ioctl. Do I miss something? Regards, Jay Zhou
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi all, After applying the patch below, the time which memory_global_dirty_log_stop() function takes is down to milliseconds of a 4T memory guest, but I'm not sure whether this patch will trigger other problems. Does this patch make sense? diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 464da93..fe26ee5 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8313,6 +8313,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, enum kvm_mr_change change) { int nr_mmu_pages = 0; + int i; + struct kvm_vcpu *vcpu; if (!kvm->arch.n_requested_mmu_pages) nr_mmu_pages = kvm_mmu_calculate_mmu_pages(kvm); @@ -8328,14 +8330,15 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, * in the source machine (for example if live migration fails), small * sptes will remain around and cause bad performance. * -* Scan sptes if dirty logging has been stopped, dropping those -* which can be collapsed into a single large-page spte. Later -* page faults will create the large-page sptes. +* Reset each vcpu's mmu, then page faults will create the large-page +* sptes later. */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_mmu_reset_context(vcpu); + } /* * Set up write protection and/or dirty logging for the new slot. > * Yang Hongyang (yanghongy...@huawei.com) wrote: > > > > > > On 2017/4/24 20:06, Juan Quintela wrote: > > > Yang Hongyang wrote: > > >> Hi all, > > >> > > >> We found dirty log switch costs more then 13 seconds while > > >> migrating a 4T memory guest, and dirty log switch is currently > > >> protected by QEMU BQL. This causes guest freeze for a long time > > >> when switching dirty log on, and the migration downtime is > unacceptable. > > >> Are there any chance to optimize the time cost for dirty log switch > operation? > > >> Or move the time consuming operation out of the QEMU BQL? > > > > > > Hi > > > > > > Could you specify what do you mean by dirty log switch? > > > The one inside kvm? > > > The merge between kvm one and migration bitmap? > > > > The call of the following functions: > > memory_global_dirty_log_start/stop(); > > I suppose there's a few questions; > a) Do we actually need the BQL - and if so why > b) What actually takes 13s? It's probably worth figuring out where it > goes, the whole bitmap is only 1GB isn't it even on a 4TB machine, and > even the simplest way to fill that takes way less than 13s. > > Dave > > > > > > > > > Thanks, Juan. > > > > > > > -- > > Thanks, > > Yang > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK Regards, Jay Zhou
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
2017-05-11 21:43 GMT+08:00 Wanpeng Li : > 2017-05-11 20:24 GMT+08:00 Paolo Bonzini : >> >> >> On 11/05/2017 14:07, Zhoujian (jay) wrote: >>> -* Scan sptes if dirty logging has been stopped, dropping those >>> -* which can be collapsed into a single large-page spte. Later >>> -* page faults will create the large-page sptes. >>> +* Reset each vcpu's mmu, then page faults will create the >>> large-page >>> +* sptes later. >>> */ >>> if ((change != KVM_MR_DELETE) && >>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && >>> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) >>> - kvm_mmu_zap_collapsible_sptes(kvm, new); > > This is an unlikely branch(unless guest live migration fails and > continue to run on the source machine) instead of hot path, do you > have any performance number for your real workloads? I find the original discussion by google. https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg04143.html You will not go to this branch if the guest live migration successfully. Regards, Wanpeng Li
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
2017-05-11 20:24 GMT+08:00 Paolo Bonzini : > > > On 11/05/2017 14:07, Zhoujian (jay) wrote: >> -* Scan sptes if dirty logging has been stopped, dropping those >> -* which can be collapsed into a single large-page spte. Later >> -* page faults will create the large-page sptes. >> +* Reset each vcpu's mmu, then page faults will create the large-page >> +* sptes later. >> */ >> if ((change != KVM_MR_DELETE) && >> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && >> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) >> - kvm_mmu_zap_collapsible_sptes(kvm, new); This is an unlikely branch(unless guest live migration fails and continue to run on the source machine) instead of hot path, do you have any performance number for your real workloads? Regards, Wanpeng Li >> + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { >> + kvm_for_each_vcpu(i, vcpu, kvm) >> + kvm_mmu_reset_context(vcpu); > > This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but > I am not sure it is enough. I think that if you do not zap the SPTEs, > the page faults will use 4K SPTEs, not large ones (though I'd have to > check better; CCing Xiao and Wanpeng). > > Paolo
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
On 11/05/2017 14:07, Zhoujian (jay) wrote: > -* Scan sptes if dirty logging has been stopped, dropping those > -* which can be collapsed into a single large-page spte. Later > -* page faults will create the large-page sptes. > +* Reset each vcpu's mmu, then page faults will create the large-page > +* sptes later. > */ > if ((change != KVM_MR_DELETE) && > (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && > - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) > - kvm_mmu_zap_collapsible_sptes(kvm, new); > + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { > + kvm_for_each_vcpu(i, vcpu, kvm) > + kvm_mmu_reset_context(vcpu); This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but I am not sure it is enough. I think that if you do not zap the SPTEs, the page faults will use 4K SPTEs, not large ones (though I'd have to check better; CCing Xiao and Wanpeng). Paolo
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi Paolo, Dave, On 2017/4/26 23:46, Paolo Bonzini wrote: > > > On 24/04/2017 18:42, Dr. David Alan Gilbert wrote: >> I suppose there's a few questions; >> a) Do we actually need the BQL - and if so why Enable/disable dirty log tracking are operations on memory regions. That's why they need to be in BQL I think. >> b) What actually takes 13s? It's probably worth figuring >> out where it goes, the whole bitmap is only 1GB isn't it >> even on a 4TB machine, and even the simplest way to fill >> that takes way less than 13s. I found two time consuming operations in KVM module, - one is kvm_mmu_slot_apply_flags(), when enable dirty log tracking kvm_vm_ioctl_set_memory_region |->kvm_set_memory_region |->__kvm_set_memory_region |->kvm_arch_commit_memory_region |->kvm_mmu_slot_apply_flags ... - the other is kvm_mmu_zap_collapsible_sptes(), when disable dirty log tracking kvm_vm_ioctl_set_memory_region |->kvm_set_memory_region |->__kvm_set_memory_region |->kvm_arch_commit_memory_region |->kvm_mmu_zap_collapsible_sptes ... Any ideas that could optimize the time spending for these operations? > > It's more likely that it is the migration_bitmap_sync immediately after > that. It's not, it's enable/disable dirty log tracking that costs time. > > I think it is possible to move migration_bitmap_sync outside BQL. It > should be simpler to evaluate that after Juan's cleanups go in. > > Paolo > -- Thanks, Yang
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
On 24/04/2017 18:42, Dr. David Alan Gilbert wrote: > I suppose there's a few questions; > a) Do we actually need the BQL - and if so why > b) What actually takes 13s? It's probably worth figuring > out where it goes, the whole bitmap is only 1GB isn't it > even on a 4TB machine, and even the simplest way to fill > that takes way less than 13s. It's more likely that it is the migration_bitmap_sync immediately after that. I think it is possible to move migration_bitmap_sync outside BQL. It should be simpler to evaluate that after Juan's cleanups go in. Paolo
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
* Yang Hongyang (yanghongy...@huawei.com) wrote: > > > On 2017/4/24 20:06, Juan Quintela wrote: > > Yang Hongyang wrote: > >> Hi all, > >> > >> We found dirty log switch costs more then 13 seconds while migrating > >> a 4T memory guest, and dirty log switch is currently protected by QEMU > >> BQL. This causes guest freeze for a long time when switching dirty log on, > >> and the migration downtime is unacceptable. > >> Are there any chance to optimize the time cost for dirty log switch > >> operation? > >> Or move the time consuming operation out of the QEMU BQL? > > > > Hi > > > > Could you specify what do you mean by dirty log switch? > > The one inside kvm? > > The merge between kvm one and migration bitmap? > > The call of the following functions: > memory_global_dirty_log_start/stop(); I suppose there's a few questions; a) Do we actually need the BQL - and if so why b) What actually takes 13s? It's probably worth figuring out where it goes, the whole bitmap is only 1GB isn't it even on a 4TB machine, and even the simplest way to fill that takes way less than 13s. Dave > > > > > Thanks, Juan. > > > > -- > Thanks, > Yang -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
On 2017/4/24 20:06, Juan Quintela wrote: > Yang Hongyang wrote: >> Hi all, >> >> We found dirty log switch costs more then 13 seconds while migrating >> a 4T memory guest, and dirty log switch is currently protected by QEMU >> BQL. This causes guest freeze for a long time when switching dirty log on, >> and the migration downtime is unacceptable. >> Are there any chance to optimize the time cost for dirty log switch >> operation? >> Or move the time consuming operation out of the QEMU BQL? > > Hi > > Could you specify what do you mean by dirty log switch? > The one inside kvm? > The merge between kvm one and migration bitmap? The call of the following functions: memory_global_dirty_log_start/stop(); > > Thanks, Juan. > -- Thanks, Yang
Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration
Yang Hongyang wrote: > Hi all, > > We found dirty log switch costs more then 13 seconds while migrating > a 4T memory guest, and dirty log switch is currently protected by QEMU > BQL. This causes guest freeze for a long time when switching dirty log on, > and the migration downtime is unacceptable. > Are there any chance to optimize the time cost for dirty log switch operation? > Or move the time consuming operation out of the QEMU BQL? Hi Could you specify what do you mean by dirty log switch? The one inside kvm? The merge between kvm one and migration bitmap? Thanks, Juan.
[Qemu-devel] About QEMU BQL and dirty log switch in Migration
Hi all, We found dirty log switch costs more then 13 seconds while migrating a 4T memory guest, and dirty log switch is currently protected by QEMU BQL. This causes guest freeze for a long time when switching dirty log on, and the migration downtime is unacceptable. Are there any chance to optimize the time cost for dirty log switch operation? Or move the time consuming operation out of the QEMU BQL? -- Thanks, Yang