Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Wanpeng Li]

On Fri, 19 May 2017 at 16:10, Jay Zhou  wrote:
>
> Hi Paolo and Wanpeng,
>
> On 2017/5/17 16:38, Wanpeng Li wrote:
> > 2017-05-17 15:43 GMT+08:00 Paolo Bonzini :
> >>> Recently, I have tested the performance before migration and after 
> >>> migration failure
> >>> using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
> >>> performance
> >>> evaluation tool.
> >>>
> >>> These are the steps:
> >>> ==
> >>>   (1) the version of kmod is 4.4.11(with slightly modified) and the 
> >>> version of
> >>>   qemu is 2.6.0
> >>>  (with slightly modified), the kmod is applied with the following 
> >>> patch
> >>>
> >>> diff --git a/source/x86/x86.c b/source/x86/x86.c
> >>> index 054a7d3..75a4bb3 100644
> >>> --- a/source/x86/x86.c
> >>> +++ b/source/x86/x86.c
> >>> @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
> >>>   */
> >>>  if ((change != KVM_MR_DELETE) &&
> >>>  (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
> >>> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
> >>> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
> >>> +   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
> >>> +   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
> >>> +   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
> >>> +   }
> >>>
> >>>  /*
> >>>   * Set up write protection and/or dirty logging for the new slot.
> >>
> >> Try these modifications to the setup:
> >>
> >> 1) set up 1G hugetlbfs hugepages and use those for the guest's memory
> >>
> >> 2) test both without and with the above patch.
> >>
>
> In order to avoid random memory allocation issues, I reran the test cases:
> (1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a
> pcpu respectively, these resources(memory and pcpu) allocated to VM are all
> from NUMA node 0
> (2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration, and
> get a result. And then, migration failure is constructed. At last, I run the
> test case again, and get an another result.
> (3) results:
> Host hugepages   THP on(2M)  THP on(2M)   THP on(2M)   THP on(2M)
> Patchpatch1  patch2   patch3   -
> Before migration No  No   No   Yes
> After migration failed   Yes Yes  Yes  No
> Largepages   67->186262->1890 95->1865 1926
> score of 429.mcf 189 188  188  189
>
> Host hugepages   1G hugepages  1G hugepages  1G hugepages  1G 
> hugepages
> Patchpatch1patch2patch3-
> Before migration NoNoNoYes
> After migration failed   Yes   Yes   Yes   No
> Largepages   21212639
> score of 429.mcf 188   188   186   188
>
> Notes:
> patch1  means with "lazy collapse small sptes into large sptes" codes
> patch2  means comment out "lazy collapse small sptes into large sptes" codes
> patch3  means using kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD)
>  instead of kvm_mmu_zap_collapsible_sptes(kvm, new)
>
> "Largepages" means the value of /sys/kernel/debug/kvm/largepages
>
> > In addition, we can compare /sys/kernel/debug/kvm/largepages w/ and
> > w/o the patch. IIRC, /sys/kernel/debug/kvm/largepages will drop during
> > live migration, it will keep a small value if live migration fails and
> > w/o "lazy collapse small sptes into large sptes" codes, however, it
> > will increase gradually if w/ the "lazy collapse small sptes into
> > large sptes" codes.
> >
>
> No, without the "lazy collapse small sptes into large sptes" codes,
> /sys/kernel/debug/kvm/largepages does drop during live migration,
> but it still will increase gradually if live migration fails, see the result
> above. I printed out the back trace when it increases after migration failure,
>
> [139574.369098]  [] dump_stack+0x19/0x1b
> [139574.369111]  [] mmu_set_spte+0x2f6/0x310 [kvm]
> [139574.369122]  [] __direct_map.isra.109+0x1de/0x250 [kvm]
> [139574.369133]  [] tdp_page_fault+0x246/0x280 [kvm]
> [139574.369144]  [] kvm_mmu_page_fault+0x24/0x130 [kvm]
> [139574.369148]  [] handle_ept_violation+0x96/0x170 
> [kvm_intel]
> [139574.369153]  [] vmx_handle_exit+0x299/0xbf0 [kvm_intel]
> [139574.369157]  [] ? uv_bau_message_intr1+0x80/0x80
> [139574.369161]  [] ? vmx_inject_irq+0xf0/0xf0 [kvm_intel]
> [139574.369172]  [] vcpu_enter_guest+0x76d/0x1160 [kvm]
> [139574.369184]  [] ? kvm_apic_local_deliver+0x65/0x70 [kvm]
> [139574.369196]  [] kvm_arch_vcpu_ioctl_run+0xd5/0x440 [kvm]
> [139574.369205]  [] kvm_vcpu_ioctl+0x2b1/0x640 [kvm]
> [139574.369209]  [] ? do_futex+0x122/0x5b0
> [139574.369212]  [] do_vfs_ioctl+0x2e5/0x4c0
> [139574.369223]  [] ? kvm_on_user_return+0x75/0xb0 [kvm]
> [139574.369225]  [] 

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Jay Zhou]

Hi Xiao,

On 2017/5/19 16:32, Xiao Guangrong wrote:


I do not know why i was removed from the list.


I was CCed to you...
Your comments are very valuable to us, and thank for your quick response.



On 05/19/2017 04:09 PM, Jay Zhou wrote:

Hi Paolo and Wanpeng,

On 2017/5/17 16:38, Wanpeng Li wrote:

2017-05-17 15:43 GMT+08:00 Paolo Bonzini :

Recently, I have tested the performance before migration and after
migration failure
using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard
performance
evaluation tool.

These are the steps:
==
  (1) the version of kmod is 4.4.11(with slightly modified) and the
version of
  qemu is 2.6.0
 (with slightly modified), the kmod is applied with the following patch

diff --git a/source/x86/x86.c b/source/x86/x86.c
index 054a7d3..75a4bb3 100644
--- a/source/x86/x86.c
+++ b/source/x86/x86.c
@@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
  */
 if ((change != KVM_MR_DELETE) &&
 (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);
+   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
+   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
+   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
+   }

 /*
  * Set up write protection and/or dirty logging for the new slot.


Try these modifications to the setup:

1) set up 1G hugetlbfs hugepages and use those for the guest's memory

2) test both without and with the above patch.



In order to avoid random memory allocation issues, I reran the test cases:
(1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a
pcpu respectively, these resources(memory and pcpu) allocated to VM are all
from NUMA node 0
(2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration,
and get a result. And then, migration failure is constructed. At last, I run
the test case again, and get an another result.


I guess this case purely writes the memory, that means the readonly mappings 
will


Yes, I printed out the speed of dirty page rate, it is about 1GB per second.


always be dropped by #PF, then huge mappings are established.

If benchmark memory read, you show observe its difference.



OK, thank for your suggestion!

Regards,
Jay Zhou

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Xiao Guangrong]

I do not know why i was removed from the list.


On 05/19/2017 04:09 PM, Jay Zhou wrote:

Hi Paolo and Wanpeng,

On 2017/5/17 16:38, Wanpeng Li wrote:

2017-05-17 15:43 GMT+08:00 Paolo Bonzini :

Recently, I have tested the performance before migration and after migration 
failure
using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
performance
evaluation tool.

These are the steps:
==
  (1) the version of kmod is 4.4.11(with slightly modified) and the version of
  qemu is 2.6.0
 (with slightly modified), the kmod is applied with the following patch

diff --git a/source/x86/x86.c b/source/x86/x86.c
index 054a7d3..75a4bb3 100644
--- a/source/x86/x86.c
+++ b/source/x86/x86.c
@@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
  */
 if ((change != KVM_MR_DELETE) &&
 (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);
+   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
+   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
+   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
+   }

 /*
  * Set up write protection and/or dirty logging for the new slot.


Try these modifications to the setup:

1) set up 1G hugetlbfs hugepages and use those for the guest's memory

2) test both without and with the above patch.



In order to avoid random memory allocation issues, I reran the test cases:
(1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a 
pcpu respectively, these resources(memory and pcpu) allocated to VM are all 
from NUMA node 0
(2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration, and 
get a result. And then, migration failure is constructed. At last, I run the 
test case again, and get an another result.


I guess this case purely writes the memory, that means the readonly mappings 
will
always be dropped by #PF, then huge mappings are established.

If benchmark memory read, you show observe its difference.

Thanks!

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Jay Zhou]

Hi Paolo and Wanpeng,

On 2017/5/17 16:38, Wanpeng Li wrote:

2017-05-17 15:43 GMT+08:00 Paolo Bonzini :

Recently, I have tested the performance before migration and after migration 
failure
using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
performance
evaluation tool.

These are the steps:
==
  (1) the version of kmod is 4.4.11(with slightly modified) and the version of
  qemu is 2.6.0
 (with slightly modified), the kmod is applied with the following patch

diff --git a/source/x86/x86.c b/source/x86/x86.c
index 054a7d3..75a4bb3 100644
--- a/source/x86/x86.c
+++ b/source/x86/x86.c
@@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
  */
 if ((change != KVM_MR_DELETE) &&
 (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);
+   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
+   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
+   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
+   }

 /*
  * Set up write protection and/or dirty logging for the new slot.


Try these modifications to the setup:

1) set up 1G hugetlbfs hugepages and use those for the guest's memory

2) test both without and with the above patch.



In order to avoid random memory allocation issues, I reran the test cases:
(1) setup: start a 4U10G VM with memory preoccupied, each vcpu is pinned to a 
pcpu respectively, these resources(memory and pcpu) allocated to VM are all 
from NUMA node 0
(2) sequence: firstly, I run the 429.mcf of spec cpu2006 before migration, and 
get a result. And then, migration failure is constructed. At last, I run the 
test case again, and get an another result.

(3) results:
Host hugepages   THP on(2M)  THP on(2M)   THP on(2M)   THP on(2M)
Patchpatch1  patch2   patch3   -
Before migration No  No   No   Yes
After migration failed   Yes Yes  Yes  No
Largepages   67->186262->1890 95->1865 1926
score of 429.mcf 189 188  188  189

Host hugepages   1G hugepages  1G hugepages  1G hugepages  1G hugepages
Patchpatch1patch2patch3-
Before migration NoNoNoYes
After migration failed   Yes   Yes   Yes   No
Largepages   21212639
score of 429.mcf 188   188   186   188

Notes:
patch1  means with "lazy collapse small sptes into large sptes" codes
patch2  means comment out "lazy collapse small sptes into large sptes" codes
patch3  means using kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD)
instead of kvm_mmu_zap_collapsible_sptes(kvm, new)

"Largepages" means the value of /sys/kernel/debug/kvm/largepages


In addition, we can compare /sys/kernel/debug/kvm/largepages w/ and
w/o the patch. IIRC, /sys/kernel/debug/kvm/largepages will drop during
live migration, it will keep a small value if live migration fails and
w/o "lazy collapse small sptes into large sptes" codes, however, it
will increase gradually if w/ the "lazy collapse small sptes into
large sptes" codes.



No, without the "lazy collapse small sptes into large sptes" codes,
/sys/kernel/debug/kvm/largepages does drop during live migration,
but it still will increase gradually if live migration fails, see the result
above. I printed out the back trace when it increases after migration failure,

[139574.369098]  [] dump_stack+0x19/0x1b
[139574.369111]  [] mmu_set_spte+0x2f6/0x310 [kvm]
[139574.369122]  [] __direct_map.isra.109+0x1de/0x250 [kvm]
[139574.369133]  [] tdp_page_fault+0x246/0x280 [kvm]
[139574.369144]  [] kvm_mmu_page_fault+0x24/0x130 [kvm]
[139574.369148]  [] handle_ept_violation+0x96/0x170 
[kvm_intel]
[139574.369153]  [] vmx_handle_exit+0x299/0xbf0 [kvm_intel]
[139574.369157]  [] ? uv_bau_message_intr1+0x80/0x80
[139574.369161]  [] ? vmx_inject_irq+0xf0/0xf0 [kvm_intel]
[139574.369172]  [] vcpu_enter_guest+0x76d/0x1160 [kvm]
[139574.369184]  [] ? kvm_apic_local_deliver+0x65/0x70 [kvm]
[139574.369196]  [] kvm_arch_vcpu_ioctl_run+0xd5/0x440 [kvm]
[139574.369205]  [] kvm_vcpu_ioctl+0x2b1/0x640 [kvm]
[139574.369209]  [] ? do_futex+0x122/0x5b0
[139574.369212]  [] do_vfs_ioctl+0x2e5/0x4c0
[139574.369223]  [] ? kvm_on_user_return+0x75/0xb0 [kvm]
[139574.369225]  [] SyS_ioctl+0xa1/0xc0
[139574.369229]  [] system_call_fastpath+0x16/0x1b

Any suggestion will be appreciated, Thanks!


Regards,
Jay Zhou

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Wanpeng Li]

2017-05-17 15:43 GMT+08:00 Paolo Bonzini :
>> Recently, I have tested the performance before migration and after migration 
>> failure
>> using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
>> performance
>> evaluation tool.
>>
>> These are the steps:
>> ==
>>  (1) the version of kmod is 4.4.11(with slightly modified) and the version of
>>  qemu is 2.6.0
>> (with slightly modified), the kmod is applied with the following patch
>>
>> diff --git a/source/x86/x86.c b/source/x86/x86.c
>> index 054a7d3..75a4bb3 100644
>> --- a/source/x86/x86.c
>> +++ b/source/x86/x86.c
>> @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
>>  */
>> if ((change != KVM_MR_DELETE) &&
>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
>> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
>> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
>> +   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
>> +   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
>> +   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
>> +   }
>>
>> /*
>>  * Set up write protection and/or dirty logging for the new slot.
>
> Try these modifications to the setup:
>
> 1) set up 1G hugetlbfs hugepages and use those for the guest's memory
>
> 2) test both without and with the above patch.
>

In addition, we can compare /sys/kernel/debug/kvm/largepages w/ and
w/o the patch. IIRC, /sys/kernel/debug/kvm/largepages will drop during
live migration, it will keep a small value if live migration fails and
w/o "lazy collapse small sptes into large sptes" codes, however, it
will increase gradually if w/ the "lazy collapse small sptes into
large sptes" codes.

Regards,
Wanpeng Li

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Paolo Bonzini]

> Recently, I have tested the performance before migration and after migration 
> failure
> using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
> performance
> evaluation tool.
>
> These are the steps:
> ==
>  (1) the version of kmod is 4.4.11(with slightly modified) and the version of
>  qemu is 2.6.0
> (with slightly modified), the kmod is applied with the following patch
> 
> diff --git a/source/x86/x86.c b/source/x86/x86.c
> index 054a7d3..75a4bb3 100644
> --- a/source/x86/x86.c
> +++ b/source/x86/x86.c
> @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
>  */
> if ((change != KVM_MR_DELETE) &&
> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
> +   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
> +   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
> +   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
> +   }
>  
> /*
>  * Set up write protection and/or dirty logging for the new slot.

Try these modifications to the setup:

1) set up 1G hugetlbfs hugepages and use those for the guest's memory

2) test both without and with the above patch.

Thanks,

Paolo

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Jay Zhou]

On 2017/5/17 13:47, Wanpeng Li wrote:

Hi Zhoujian,
2017-05-17 10:20 GMT+08:00 Zhoujian (jay) :

Hi Wanpeng,


On 11/05/2017 14:07, Zhoujian (jay) wrote:

-* Scan sptes if dirty logging has been stopped, dropping those
-* which can be collapsed into a single large-page spte.  Later
-* page faults will create the large-page sptes.
+* Reset each vcpu's mmu, then page faults will create the

large-page

+* sptes later.
  */
 if ((change != KVM_MR_DELETE) &&
 (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);


This is an unlikely branch(unless guest live migration fails and continue
to run on the source machine) instead of hot path, do you have any
performance number for your real workloads?



Sorry to bother you again.

Recently, I have tested the performance before migration and after migration 
failure
using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
performance
evaluation tool.

These are the results:
**
 Before migration the score is 153, and the TLB miss statistics of the qemu 
process is:
 linux-sjrfac:/mnt/zhoujian # perf stat -e 
dTLB-load-misses,dTLB-loads,dTLB-store-misses, \
 dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10

 Performance counter stats for process id '26463':

698,938  dTLB-load-misses  #0.13% of all dTLB cache 
hits   (50.46%)
543,303,875  dTLB-loads 
   (50.43%)
199,597  dTLB-store-misses  
   (16.51%)
 60,128,561  dTLB-stores
   (16.67%)
 69,986  iTLB-load-misses  #6.17% of all iTLB cache 
hits   (16.67%)
  1,134,097  iTLB-loads 
   (33.33%)

   10.000684064 seconds time elapsed

 After migration failure the score is 149, and the TLB miss statistics of 
the qemu process is:
 linux-sjrfac:/mnt/zhoujian # perf stat -e 
dTLB-load-misses,dTLB-loads,dTLB-store-misses, \
 dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10

 Performance counter stats for process id '26463':

765,400  dTLB-load-misses  #0.14% of all dTLB cache 
hits   (50.50%)
540,972,144  dTLB-loads 
   (50.47%)
207,670  dTLB-store-misses  
   (16.50%)
 58,363,787  dTLB-stores
   (16.67%)
109,772  iTLB-load-misses  #9.52% of all iTLB cache 
hits   (16.67%)
  1,152,784  iTLB-loads 
   (33.32%)

   10.000703078 seconds time elapsed
**


Could you comment out the original "lazy collapse small sptes into
large sptes" codes in the function kvm_arch_commit_memory_region() and
post the results here?



  With the patch below,

diff --git a/source/x86/x86.c b/source/x86/x86.c
index 054a7d3..e0288d5 100644
--- a/source/x86/x86.c
+++ b/source/x86/x86.c
@@ -8548,10 +8548,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 * which can be collapsed into a single large-page spte.  Later
 * page faults will create the large-page sptes.
 */
-   if ((change != KVM_MR_DELETE) &&
-   (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);

/*
 * Set up write protection and/or dirty logging for the new slot.

  After migration failure the score is 148, and the TLB miss statistics 
of the qemu process is:
  linux-sjrfac:/mnt/zhoujian # perf stat -e 
dTLB-load-misses,dTLB-loads,dTLB-store-misses,dTLB-stores,iTLB-load-misses,iTLB-loads 
-p 12432 sleep 10


 Performance counter stats for process id '12432':

 1,052,697  dTLB-load-misses  #0.19% of all 
dTLB cache hits   (50.45%)
   551,828,702  dTLB-loads 
   (50.46%)
   147,228  dTLB-store-misses 
   (16.55%)
60,427,834  dTLB-stores 
   (16.50%)
93,793  iTLB-load-misses  #7.43% of all 
iTLB cache hits   (16.67%)
 1,262,137  iTLB-loads 
   (33.33%)


  10.000709900 seconds time elapsed

  Regards,
  Jay Zhou


Regards,
Wanpeng Li



These are the steps:
==
  (1) the version of kmod is 4.4.11(with slightly modified) and the version of 
qemu is 2.6.0
 (with slightly modified), the kmod is applied with the following patch 
according to
 Paolo's advice:

diff --git a/source/x86/x86.c b/source/x86/x86.c
index 

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Wanpeng Li]

Hi Zhoujian,
2017-05-17 10:20 GMT+08:00 Zhoujian (jay) :
> Hi Wanpeng,
>
>> > On 11/05/2017 14:07, Zhoujian (jay) wrote:
>> >> -* Scan sptes if dirty logging has been stopped, dropping those
>> >> -* which can be collapsed into a single large-page spte.  Later
>> >> -* page faults will create the large-page sptes.
>> >> +* Reset each vcpu's mmu, then page faults will create the
>> large-page
>> >> +* sptes later.
>> >>  */
>> >> if ((change != KVM_MR_DELETE) &&
>> >> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
>> >> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
>> >> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
>>
>> This is an unlikely branch(unless guest live migration fails and continue
>> to run on the source machine) instead of hot path, do you have any
>> performance number for your real workloads?
>>
>
> Sorry to bother you again.
>
> Recently, I have tested the performance before migration and after migration 
> failure
> using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
> performance
> evaluation tool.
>
> These are the results:
> **
> Before migration the score is 153, and the TLB miss statistics of the 
> qemu process is:
> linux-sjrfac:/mnt/zhoujian # perf stat -e 
> dTLB-load-misses,dTLB-loads,dTLB-store-misses, \
> dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10
>
> Performance counter stats for process id '26463':
>
>698,938  dTLB-load-misses  #0.13% of all dTLB 
> cache hits   (50.46%)
>543,303,875  dTLB-loads
> (50.43%)
>199,597  dTLB-store-misses 
> (16.51%)
> 60,128,561  dTLB-stores   
> (16.67%)
> 69,986  iTLB-load-misses  #6.17% of all iTLB 
> cache hits   (16.67%)
>  1,134,097  iTLB-loads
> (33.33%)
>
>   10.000684064 seconds time elapsed
>
> After migration failure the score is 149, and the TLB miss statistics of 
> the qemu process is:
> linux-sjrfac:/mnt/zhoujian # perf stat -e 
> dTLB-load-misses,dTLB-loads,dTLB-store-misses, \
> dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10
>
> Performance counter stats for process id '26463':
>
>765,400  dTLB-load-misses  #0.14% of all dTLB 
> cache hits   (50.50%)
>540,972,144  dTLB-loads
> (50.47%)
>207,670  dTLB-store-misses 
> (16.50%)
> 58,363,787  dTLB-stores   
> (16.67%)
>109,772  iTLB-load-misses  #9.52% of all iTLB 
> cache hits   (16.67%)
>  1,152,784  iTLB-loads
> (33.32%)
>
>   10.000703078 seconds time elapsed
> **

Could you comment out the original "lazy collapse small sptes into
large sptes" codes in the function kvm_arch_commit_memory_region() and
post the results here?

Regards,
Wanpeng Li

>
> These are the steps:
> ==
>  (1) the version of kmod is 4.4.11(with slightly modified) and the version of 
> qemu is 2.6.0
> (with slightly modified), the kmod is applied with the following patch 
> according to
> Paolo's advice:
>
> diff --git a/source/x86/x86.c b/source/x86/x86.c
> index 054a7d3..75a4bb3 100644
> --- a/source/x86/x86.c
> +++ b/source/x86/x86.c
> @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
>  */
> if ((change != KVM_MR_DELETE) &&
> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
> +   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
> +   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
> +   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
> +   }
>
> /*
>  * Set up write protection and/or dirty logging for the new slot.
>
> (2) I started up a memory preoccupied 10G VM(suse11sp3), which means its "RES 
> column" in top is 10G,
> in order to set up the EPT table in advance.
> (3) And then, I run the test case 429.mcf of spec cpu2006 before migration 
> and after migration failure.
> The 429.mcf is a memory intensive workload, and the migration failure is 
> constructed deliberately
> with the following patch of qemu:
>
> diff --git a/migration/migration.c b/migration/migration.c
> index 5d725d0..88dfc59 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -625,6 +625,9 @@ static void process_incoming_migration_co(void *opaque)
>

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Zhoujian (jay)]

Hi Wanpeng,

> > On 11/05/2017 14:07, Zhoujian (jay) wrote:
> >> -* Scan sptes if dirty logging has been stopped, dropping those
> >> -* which can be collapsed into a single large-page spte.  Later
> >> -* page faults will create the large-page sptes.
> >> +* Reset each vcpu's mmu, then page faults will create the
> large-page
> >> +* sptes later.
> >>  */
> >> if ((change != KVM_MR_DELETE) &&
> >> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
> >> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
> >> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
> 
> This is an unlikely branch(unless guest live migration fails and continue
> to run on the source machine) instead of hot path, do you have any
> performance number for your real workloads?
> 

Sorry to bother you again.

Recently, I have tested the performance before migration and after migration 
failure
using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard 
performance
evaluation tool.

These are the results:
**
Before migration the score is 153, and the TLB miss statistics of the qemu 
process is:
linux-sjrfac:/mnt/zhoujian # perf stat -e 
dTLB-load-misses,dTLB-loads,dTLB-store-misses, \
dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10

Performance counter stats for process id '26463':

   698,938  dTLB-load-misses  #0.13% of all dTLB cache 
hits   (50.46%)
   543,303,875  dTLB-loads  
  (50.43%)
   199,597  dTLB-store-misses   
  (16.51%)
60,128,561  dTLB-stores 
  (16.67%)
69,986  iTLB-load-misses  #6.17% of all iTLB cache 
hits   (16.67%)
 1,134,097  iTLB-loads  
  (33.33%)

  10.000684064 seconds time elapsed

After migration failure the score is 149, and the TLB miss statistics of 
the qemu process is:
linux-sjrfac:/mnt/zhoujian # perf stat -e 
dTLB-load-misses,dTLB-loads,dTLB-store-misses, \
dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10

Performance counter stats for process id '26463':

   765,400  dTLB-load-misses  #0.14% of all dTLB cache 
hits   (50.50%)
   540,972,144  dTLB-loads  
  (50.47%)
   207,670  dTLB-store-misses   
  (16.50%)
58,363,787  dTLB-stores 
  (16.67%)
   109,772  iTLB-load-misses  #9.52% of all iTLB cache 
hits   (16.67%)
 1,152,784  iTLB-loads  
  (33.32%)

  10.000703078 seconds time elapsed
**

These are the steps:
==
 (1) the version of kmod is 4.4.11(with slightly modified) and the version of 
qemu is 2.6.0
(with slightly modified), the kmod is applied with the following patch 
according to
Paolo's advice:

diff --git a/source/x86/x86.c b/source/x86/x86.c
index 054a7d3..75a4bb3 100644
--- a/source/x86/x86.c
+++ b/source/x86/x86.c
@@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 */
if ((change != KVM_MR_DELETE) &&
(old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);
+   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
+   printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n");
+   kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
+   }
 
/*
 * Set up write protection and/or dirty logging for the new slot.

(2) I started up a memory preoccupied 10G VM(suse11sp3), which means its "RES 
column" in top is 10G,
in order to set up the EPT table in advance.
(3) And then, I run the test case 429.mcf of spec cpu2006 before migration and 
after migration failure.
The 429.mcf is a memory intensive workload, and the migration failure is 
constructed deliberately
with the following patch of qemu:

diff --git a/migration/migration.c b/migration/migration.c
index 5d725d0..88dfc59 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -625,6 +625,9 @@ static void process_incoming_migration_co(void *opaque)
   MIGRATION_STATUS_ACTIVE);
 ret = qemu_loadvm_state(f);
 
+// deliberately construct the migration failure
+exit(EXIT_FAILURE); 
+
 ps = postcopy_state_get();
 trace_process_incoming_migration_co_end(ret, ps);
 if (ps != POSTCOPY_INCOMING_NONE) {
==


Results of the score and TLB miss rate are almost the same, and I am confused.
May I ask which tool do you use to evaluate the performance?
And if my test steps are wrong, please 

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Hailiang Zhang]

On 2017/5/12 16:09, Xiao Guangrong wrote:


On 05/11/2017 08:24 PM, Paolo Bonzini wrote:


On 11/05/2017 14:07, Zhoujian (jay) wrote:

-* Scan sptes if dirty logging has been stopped, dropping those
-* which can be collapsed into a single large-page spte.  Later
-* page faults will create the large-page sptes.
+* Reset each vcpu's mmu, then page faults will create the large-page
+* sptes later.
   */
  if ((change != KVM_MR_DELETE) &&
  (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);
+   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
+   kvm_for_each_vcpu(i, vcpu, kvm)
+   kvm_mmu_reset_context(vcpu);

This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but
I am not sure it is enough.  I think that if you do not zap the SPTEs,
the page faults will use 4K SPTEs, not large ones (though I'd have to
check better; CCing Xiao and Wanpeng).

Yes, Paolo is right. kvm_mmu_reset_context() just reloads vCPU's
root page table, 4k mappings are still kept.

There are two issues reported:
- one is kvm_mmu_slot_apply_flags(), when enable dirty log tracking.

Its root cause is kvm_mmu_slot_remove_write_access() takes too much
time.

We can make the code adaptive to use the new fast-write-protect faculty
introduced by my patchset, i.e, if the number of pages contained in this
memslot is more than > TOTAL * FAST_WRITE_PROTECT_PAGE_PERCENTAGE, then
we use fast-write-protect instead.

- another one is kvm_mmu_zap_collapsible_sptes() when disable dirty
log tracking.

collapsible_sptes zaps 4k mappings to make memory-read happy, it is not
required by the semanteme of KVM_SET_USER_MEMORY_REGION and it is not
urgent for vCPU's running, it could be done in a separate thread and use
lock-break technology.


How about move the action of stopping dirty log into migrate_fd_cleanup() 
directly,
which is processed in main thread as BH after migration is completed?  It will 
not
has any side-effect even migration is failed, Or users cancel migration, No ?


Thanks!


.

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Xiao Guangrong]

On 05/11/2017 08:24 PM, Paolo Bonzini wrote:



On 11/05/2017 14:07, Zhoujian (jay) wrote:

-* Scan sptes if dirty logging has been stopped, dropping those
-* which can be collapsed into a single large-page spte.  Later
-* page faults will create the large-page sptes.
+* Reset each vcpu's mmu, then page faults will create the large-page
+* sptes later.
  */
 if ((change != KVM_MR_DELETE) &&
 (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);
+   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
+   kvm_for_each_vcpu(i, vcpu, kvm)
+   kvm_mmu_reset_context(vcpu);


This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but
I am not sure it is enough.  I think that if you do not zap the SPTEs,
the page faults will use 4K SPTEs, not large ones (though I'd have to
check better; CCing Xiao and Wanpeng).


Yes, Paolo is right. kvm_mmu_reset_context() just reloads vCPU's
root page table, 4k mappings are still kept.

There are two issues reported:
- one is kvm_mmu_slot_apply_flags(), when enable dirty log tracking.

  Its root cause is kvm_mmu_slot_remove_write_access() takes too much
  time.

  We can make the code adaptive to use the new fast-write-protect faculty
  introduced by my patchset, i.e, if the number of pages contained in this
  memslot is more than > TOTAL * FAST_WRITE_PROTECT_PAGE_PERCENTAGE, then
  we use fast-write-protect instead.

- another one is kvm_mmu_zap_collapsible_sptes() when disable dirty
  log tracking.

  collapsible_sptes zaps 4k mappings to make memory-read happy, it is not
  required by the semanteme of KVM_SET_USER_MEMORY_REGION and it is not
  urgent for vCPU's running, it could be done in a separate thread and use
  lock-break technology.

Thanks!

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Wanpeng Li]

2017-05-11 22:18 GMT+08:00 Zhoujian (jay) :
> Hi Wanpeng,
>
>> 2017-05-11 21:43 GMT+08:00 Wanpeng Li :
>> > 2017-05-11 20:24 GMT+08:00 Paolo Bonzini :
>> >>
>> >>
>> >> On 11/05/2017 14:07, Zhoujian (jay) wrote:
>> >>> -* Scan sptes if dirty logging has been stopped, dropping
>> those
>> >>> -* which can be collapsed into a single large-page spte.
>> Later
>> >>> -* page faults will create the large-page sptes.
>> >>> +* Reset each vcpu's mmu, then page faults will create the
>> large-page
>> >>> +* sptes later.
>> >>>  */
>> >>> if ((change != KVM_MR_DELETE) &&
>> >>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
>> >>> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
>> >>> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
>> >
>> > This is an unlikely branch(unless guest live migration fails and
>> > continue to run on the source machine) instead of hot path, do you
>> > have any performance number for your real workloads?
>>
>> I find the original discussion by google.
>> https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg04143.html
>> You will not go to this branch if the guest live migration successfully.
>
>  In our tests, this branch is taken when living migration is successful.
>  AFAIK, the kmod does not know whether living migration successful or not
>  when dealing with KVM_SET_USER_MEMORY_REGION ioctl. Do I miss something?

Original there is a bug which will not clear memslot dirty log flag
after live migration fails, a patch is submitted to fix it,
https://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg00794.html,
however, I can't remember whether the dirty log flag will be cleared
if live migration complete successfully at that time, but maybe not.
Paolo replied to the patch he has a better method. Then I'm too busy
and didn't follow the qemu patch for this fix any more, I just find
this commit is merged currently:
http://git.qemu.org/?p=qemu.git;a=commit;h=6f6a5ef3e429f92f987678ea8c396aab4dc6aa19.
This commit will clear memslot dirty log flag after live migration no
matter whether it is successful or not.

Regards,
Wanpeng Li

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Zhoujian (jay)]

Hi Wanpeng,

> 2017-05-11 21:43 GMT+08:00 Wanpeng Li :
> > 2017-05-11 20:24 GMT+08:00 Paolo Bonzini :
> >>
> >>
> >> On 11/05/2017 14:07, Zhoujian (jay) wrote:
> >>> -* Scan sptes if dirty logging has been stopped, dropping
> those
> >>> -* which can be collapsed into a single large-page spte.
> Later
> >>> -* page faults will create the large-page sptes.
> >>> +* Reset each vcpu's mmu, then page faults will create the
> large-page
> >>> +* sptes later.
> >>>  */
> >>> if ((change != KVM_MR_DELETE) &&
> >>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
> >>> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
> >>> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
> >
> > This is an unlikely branch(unless guest live migration fails and
> > continue to run on the source machine) instead of hot path, do you
> > have any performance number for your real workloads?
> 
> I find the original discussion by google.
> https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg04143.html
> You will not go to this branch if the guest live migration successfully.

 In our tests, this branch is taken when living migration is successful.
 AFAIK, the kmod does not know whether living migration successful or not
 when dealing with KVM_SET_USER_MEMORY_REGION ioctl. Do I miss something?

Regards,
Jay Zhou

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Zhoujian (jay)]

Hi all,

After applying the patch below, the time which
memory_global_dirty_log_stop() function takes is down to milliseconds
of a 4T memory guest, but I'm not sure whether this patch will trigger
other problems. Does this patch make sense?

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 464da93..fe26ee5 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8313,6 +8313,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
enum kvm_mr_change change)
 {
int nr_mmu_pages = 0;
+   int i;
+   struct kvm_vcpu *vcpu;
 
if (!kvm->arch.n_requested_mmu_pages)
nr_mmu_pages = kvm_mmu_calculate_mmu_pages(kvm);
@@ -8328,14 +8330,15 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 * in the source machine (for example if live migration fails), small
 * sptes will remain around and cause bad performance.
 *
-* Scan sptes if dirty logging has been stopped, dropping those
-* which can be collapsed into a single large-page spte.  Later
-* page faults will create the large-page sptes.
+* Reset each vcpu's mmu, then page faults will create the large-page
+* sptes later.
 */
if ((change != KVM_MR_DELETE) &&
(old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
-   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
-   kvm_mmu_zap_collapsible_sptes(kvm, new);
+   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
+   kvm_for_each_vcpu(i, vcpu, kvm)
+   kvm_mmu_reset_context(vcpu);
+   }
 
/*
 * Set up write protection and/or dirty logging for the new slot.

> * Yang Hongyang (yanghongy...@huawei.com) wrote:
> >
> >
> > On 2017/4/24 20:06, Juan Quintela wrote:
> > > Yang Hongyang  wrote:
> > >> Hi all,
> > >>
> > >> We found dirty log switch costs more then 13 seconds while
> > >> migrating a 4T memory guest, and dirty log switch is currently
> > >> protected by QEMU BQL. This causes guest freeze for a long time
> > >> when switching dirty log on, and the migration downtime is
> unacceptable.
> > >> Are there any chance to optimize the time cost for dirty log switch
> operation?
> > >> Or move the time consuming operation out of the QEMU BQL?
> > >
> > > Hi
> > >
> > > Could you specify what do you mean by dirty log switch?
> > > The one inside kvm?
> > > The merge between kvm one and migration bitmap?
> >
> > The call of the following functions:
> > memory_global_dirty_log_start/stop();
> 
> I suppose there's a few questions;
>   a) Do we actually need the BQL - and if so why
>   b) What actually takes 13s?  It's probably worth figuring out where it
> goes,  the whole bitmap is only 1GB isn't it even on a 4TB machine, and
> even the simplest way to fill that takes way less than 13s.
> 
> Dave
> 
> >
> > >
> > > Thanks, Juan.
> > >
> >
> > --
> > Thanks,
> > Yang
> --
> Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Regards,
Jay Zhou

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Wanpeng Li]

2017-05-11 21:43 GMT+08:00 Wanpeng Li :
> 2017-05-11 20:24 GMT+08:00 Paolo Bonzini :
>>
>>
>> On 11/05/2017 14:07, Zhoujian (jay) wrote:
>>> -* Scan sptes if dirty logging has been stopped, dropping those
>>> -* which can be collapsed into a single large-page spte.  Later
>>> -* page faults will create the large-page sptes.
>>> +* Reset each vcpu's mmu, then page faults will create the 
>>> large-page
>>> +* sptes later.
>>>  */
>>> if ((change != KVM_MR_DELETE) &&
>>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
>>> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
>>> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
>
> This is an unlikely branch(unless guest live migration fails and
> continue to run on the source machine) instead of hot path, do you
> have any performance number for your real workloads?

I find the original discussion by google.
https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg04143.html
You will not go to this branch if the guest live migration
successfully.

Regards,
Wanpeng Li

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Wanpeng Li]

2017-05-11 20:24 GMT+08:00 Paolo Bonzini :
>
>
> On 11/05/2017 14:07, Zhoujian (jay) wrote:
>> -* Scan sptes if dirty logging has been stopped, dropping those
>> -* which can be collapsed into a single large-page spte.  Later
>> -* page faults will create the large-page sptes.
>> +* Reset each vcpu's mmu, then page faults will create the large-page
>> +* sptes later.
>>  */
>> if ((change != KVM_MR_DELETE) &&
>> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
>> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
>> -   kvm_mmu_zap_collapsible_sptes(kvm, new);

This is an unlikely branch(unless guest live migration fails and
continue to run on the source machine) instead of hot path, do you
have any performance number for your real workloads?

Regards,
Wanpeng Li

>> +   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
>> +   kvm_for_each_vcpu(i, vcpu, kvm)
>> +   kvm_mmu_reset_context(vcpu);
>
> This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but
> I am not sure it is enough.  I think that if you do not zap the SPTEs,
> the page faults will use 4K SPTEs, not large ones (though I'd have to
> check better; CCing Xiao and Wanpeng).
>
> Paolo

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Paolo Bonzini]

On 11/05/2017 14:07, Zhoujian (jay) wrote:
> -* Scan sptes if dirty logging has been stopped, dropping those
> -* which can be collapsed into a single large-page spte.  Later
> -* page faults will create the large-page sptes.
> +* Reset each vcpu's mmu, then page faults will create the large-page
> +* sptes later.
>  */
> if ((change != KVM_MR_DELETE) &&
> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
> -   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
> -   kvm_mmu_zap_collapsible_sptes(kvm, new);
> +   !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
> +   kvm_for_each_vcpu(i, vcpu, kvm)
> +   kvm_mmu_reset_context(vcpu);

This should be "kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);" but
I am not sure it is enough.  I think that if you do not zap the SPTEs,
the page faults will use 4K SPTEs, not large ones (though I'd have to
check better; CCing Xiao and Wanpeng).

Paolo

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Yang Hongyang]

Hi Paolo, Dave,

On 2017/4/26 23:46, Paolo Bonzini wrote:
> 
> 
> On 24/04/2017 18:42, Dr. David Alan Gilbert wrote:
>> I suppose there's a few questions;
>>   a) Do we actually need the BQL - and if so why

Enable/disable dirty log tracking are operations on memory regions.
That's why they need to be in BQL I think.

>>   b) What actually takes 13s?  It's probably worth figuring
>> out where it goes,  the whole bitmap is only 1GB isn't it
>> even on a 4TB machine, and even the simplest way to fill
>> that takes way less than 13s.

I found two time consuming operations in KVM module,

- one is kvm_mmu_slot_apply_flags(), when enable dirty log tracking
kvm_vm_ioctl_set_memory_region
  |->kvm_set_memory_region
 |->__kvm_set_memory_region
|->kvm_arch_commit_memory_region
   |->kvm_mmu_slot_apply_flags
  ...

- the other is kvm_mmu_zap_collapsible_sptes(), when disable dirty log tracking
kvm_vm_ioctl_set_memory_region
  |->kvm_set_memory_region
 |->__kvm_set_memory_region
|->kvm_arch_commit_memory_region
   |->kvm_mmu_zap_collapsible_sptes
  ...

Any ideas that could optimize the time spending for these operations?

> 
> It's more likely that it is the migration_bitmap_sync immediately after
> that.

It's not, it's enable/disable dirty log tracking that costs time.

> 
> I think it is possible to move migration_bitmap_sync outside BQL.  It
> should be simpler to evaluate that after Juan's cleanups go in.
> 
> Paolo
> 

-- 
Thanks,
Yang

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Paolo Bonzini]

On 24/04/2017 18:42, Dr. David Alan Gilbert wrote:
> I suppose there's a few questions;
>   a) Do we actually need the BQL - and if so why
>   b) What actually takes 13s?  It's probably worth figuring
> out where it goes,  the whole bitmap is only 1GB isn't it
> even on a 4TB machine, and even the simplest way to fill
> that takes way less than 13s.

It's more likely that it is the migration_bitmap_sync immediately after
that.

I think it is possible to move migration_bitmap_sync outside BQL.  It
should be simpler to evaluate that after Juan's cleanups go in.

Paolo

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Dr. David Alan Gilbert]

* Yang Hongyang (yanghongy...@huawei.com) wrote:
> 
> 
> On 2017/4/24 20:06, Juan Quintela wrote:
> > Yang Hongyang  wrote:
> >> Hi all,
> >>
> >> We found dirty log switch costs more then 13 seconds while migrating
> >> a 4T memory guest, and dirty log switch is currently protected by QEMU
> >> BQL. This causes guest freeze for a long time when switching dirty log on,
> >> and the migration downtime is unacceptable.
> >> Are there any chance to optimize the time cost for dirty log switch 
> >> operation?
> >> Or move the time consuming operation out of the QEMU BQL?
> > 
> > Hi
> > 
> > Could you specify what do you mean by dirty log switch?
> > The one inside kvm?
> > The merge between kvm one and migration bitmap?
> 
> The call of the following functions:
> memory_global_dirty_log_start/stop();

I suppose there's a few questions;
  a) Do we actually need the BQL - and if so why
  b) What actually takes 13s?  It's probably worth figuring
out where it goes,  the whole bitmap is only 1GB isn't it
even on a 4TB machine, and even the simplest way to fill
that takes way less than 13s.

Dave

> 
> > 
> > Thanks, Juan.
> > 
> 
> -- 
> Thanks,
> Yang
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Yang Hongyang]

On 2017/4/24 20:06, Juan Quintela wrote:
> Yang Hongyang  wrote:
>> Hi all,
>>
>> We found dirty log switch costs more then 13 seconds while migrating
>> a 4T memory guest, and dirty log switch is currently protected by QEMU
>> BQL. This causes guest freeze for a long time when switching dirty log on,
>> and the migration downtime is unacceptable.
>> Are there any chance to optimize the time cost for dirty log switch 
>> operation?
>> Or move the time consuming operation out of the QEMU BQL?
> 
> Hi
> 
> Could you specify what do you mean by dirty log switch?
> The one inside kvm?
> The merge between kvm one and migration bitmap?

The call of the following functions:
memory_global_dirty_log_start/stop();

> 
> Thanks, Juan.
> 

-- 
Thanks,
Yang

Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Juan Quintela]

Yang Hongyang  wrote:
> Hi all,
>
> We found dirty log switch costs more then 13 seconds while migrating
> a 4T memory guest, and dirty log switch is currently protected by QEMU
> BQL. This causes guest freeze for a long time when switching dirty log on,
> and the migration downtime is unacceptable.
> Are there any chance to optimize the time cost for dirty log switch operation?
> Or move the time consuming operation out of the QEMU BQL?

Hi

Could you specify what do you mean by dirty log switch?
The one inside kvm?
The merge between kvm one and migration bitmap?

Thanks, Juan.

[Qemu-devel] About QEMU BQL and dirty log switch in Migration

[Yang Hongyang]

Hi all,

We found dirty log switch costs more then 13 seconds while migrating
a 4T memory guest, and dirty log switch is currently protected by QEMU
BQL. This causes guest freeze for a long time when switching dirty log on,
and the migration downtime is unacceptable.
Are there any chance to optimize the time cost for dirty log switch operation?
Or move the time consuming operation out of the QEMU BQL?

-- 
Thanks,
Yang