Re: [PATCH 0/7] KVM: MMU: fast write protect
On 05/30/2017 12:48 AM, Paolo Bonzini wrote: On 23/05/2017 04:23, Xiao Guangrong wrote: Ping... Sorry to disturb, just make this patchset not be missed. :) It won't. :) I'm going to look at it and the dirty page ring buffer this week. Ping.. :)
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 05/30/2017 12:48 AM, Paolo Bonzini wrote: On 23/05/2017 04:23, Xiao Guangrong wrote: Ping... Sorry to disturb, just make this patchset not be missed. :) It won't. :) I'm going to look at it and the dirty page ring buffer this week. Ping.. :)
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 23/05/2017 04:23, Xiao Guangrong wrote: > > Ping... > > Sorry to disturb, just make this patchset not be missed. :) It won't. :) I'm going to look at it and the dirty page ring buffer this week. Paolo
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 23/05/2017 04:23, Xiao Guangrong wrote: > > Ping... > > Sorry to disturb, just make this patchset not be missed. :) It won't. :) I'm going to look at it and the dirty page ring buffer this week. Paolo
Re: [PATCH 0/7] KVM: MMU: fast write protect
Ping... Sorry to disturb, just make this patchset not be missed. :) On 05/04/2017 03:06 PM, Paolo Bonzini wrote: On 04/05/2017 05:36, Xiao Guangrong wrote: Great. As there is no conflict between these two patchsets except dirty ring pages takes benefit from write-protect-all, i think they can be developed and iterated independently, right? I can certainly start reviewing this one. Paolo Or you prefer to merge dirty ring pages first then review the new version of this patchset later?
Re: [PATCH 0/7] KVM: MMU: fast write protect
Ping... Sorry to disturb, just make this patchset not be missed. :) On 05/04/2017 03:06 PM, Paolo Bonzini wrote: On 04/05/2017 05:36, Xiao Guangrong wrote: Great. As there is no conflict between these two patchsets except dirty ring pages takes benefit from write-protect-all, i think they can be developed and iterated independently, right? I can certainly start reviewing this one. Paolo Or you prefer to merge dirty ring pages first then review the new version of this patchset later?
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 04/05/2017 05:36, Xiao Guangrong wrote: > Great. > > As there is no conflict between these two patchsets except dirty > ring pages takes benefit from write-protect-all, i think they > can be developed and iterated independently, right? I can certainly start reviewing this one. Paolo > Or you prefer to merge dirty ring pages first then review the > new version of this patchset later?
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 04/05/2017 05:36, Xiao Guangrong wrote: > Great. > > As there is no conflict between these two patchsets except dirty > ring pages takes benefit from write-protect-all, i think they > can be developed and iterated independently, right? I can certainly start reviewing this one. Paolo > Or you prefer to merge dirty ring pages first then review the > new version of this patchset later?
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 05/03/2017 10:57 PM, Paolo Bonzini wrote: On 03/05/2017 16:50, Xiao Guangrong wrote: Furthermore, userspace has no knowledge about if PML is enable (it can be required from sysfs, but it is a good way in QEMU), so it is difficult for the usespace to know when to use write-protect-all. Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if PML is enabled? Yes, that's a good idea. Though it's a pity that, with PML, setting the dirty bit will still do the massive walk of the rmap. At least with reset_dirty_pages it's done a little bit at a time. Also, I wonder how the alternative write protection mechanism would affect performance of the dirty page ring buffer patches. You would do the write protection of all memory at the end of kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate ioctl, which is nice. On the other hand, checkpoints would be more frequent and most pages would be write-protected, so it would be more expensive to rebuild the shadow page tables... Yup, write-protect-all can improve reset_dirty_pages indeed, i will apply your idea after reset_dirty_pages is merged. However, we still prefer to have a separate ioctl for write-protect-all which cooperates with KVM_GET_DIRTY_LOG to improve live migration that should not always depend on checkpoint. Ok, I plan to merge the dirty ring pages early in 4.13 development. Great. As there is no conflict between these two patchsets except dirty ring pages takes benefit from write-protect-all, i think they can be developed and iterated independently, right? Or you prefer to merge dirty ring pages first then review the new version of this patchset later? Thanks!
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 05/03/2017 10:57 PM, Paolo Bonzini wrote: On 03/05/2017 16:50, Xiao Guangrong wrote: Furthermore, userspace has no knowledge about if PML is enable (it can be required from sysfs, but it is a good way in QEMU), so it is difficult for the usespace to know when to use write-protect-all. Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if PML is enabled? Yes, that's a good idea. Though it's a pity that, with PML, setting the dirty bit will still do the massive walk of the rmap. At least with reset_dirty_pages it's done a little bit at a time. Also, I wonder how the alternative write protection mechanism would affect performance of the dirty page ring buffer patches. You would do the write protection of all memory at the end of kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate ioctl, which is nice. On the other hand, checkpoints would be more frequent and most pages would be write-protected, so it would be more expensive to rebuild the shadow page tables... Yup, write-protect-all can improve reset_dirty_pages indeed, i will apply your idea after reset_dirty_pages is merged. However, we still prefer to have a separate ioctl for write-protect-all which cooperates with KVM_GET_DIRTY_LOG to improve live migration that should not always depend on checkpoint. Ok, I plan to merge the dirty ring pages early in 4.13 development. Great. As there is no conflict between these two patchsets except dirty ring pages takes benefit from write-protect-all, i think they can be developed and iterated independently, right? Or you prefer to merge dirty ring pages first then review the new version of this patchset later? Thanks!
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 03/05/2017 16:50, Xiao Guangrong wrote: > Furthermore, userspace has no knowledge about if PML is enable (it > can be required from sysfs, but it is a good way in QEMU), so it is > difficult for the usespace to know when to use write-protect-all. > Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if > PML is enabled? Yes, that's a good idea. Though it's a pity that, with PML, setting the dirty bit will still do the massive walk of the rmap. At least with reset_dirty_pages it's done a little bit at a time. >> Also, I wonder how the alternative write protection mechanism would >> affect performance of the dirty page ring buffer patches. You would do >> the write protection of all memory at the end of >> kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate >> ioctl, which is nice. On the other hand, checkpoints would be more >> frequent and most pages would be write-protected, so it would be more >> expensive to rebuild the shadow page tables... > > Yup, write-protect-all can improve reset_dirty_pages indeed, i will > apply your idea after reset_dirty_pages is merged. > > However, we still prefer to have a separate ioctl for write-protect-all > which cooperates with KVM_GET_DIRTY_LOG to improve live migration that > should not always depend on checkpoint. Ok, I plan to merge the dirty ring pages early in 4.13 development. Paolo
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 03/05/2017 16:50, Xiao Guangrong wrote: > Furthermore, userspace has no knowledge about if PML is enable (it > can be required from sysfs, but it is a good way in QEMU), so it is > difficult for the usespace to know when to use write-protect-all. > Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if > PML is enabled? Yes, that's a good idea. Though it's a pity that, with PML, setting the dirty bit will still do the massive walk of the rmap. At least with reset_dirty_pages it's done a little bit at a time. >> Also, I wonder how the alternative write protection mechanism would >> affect performance of the dirty page ring buffer patches. You would do >> the write protection of all memory at the end of >> kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate >> ioctl, which is nice. On the other hand, checkpoints would be more >> frequent and most pages would be write-protected, so it would be more >> expensive to rebuild the shadow page tables... > > Yup, write-protect-all can improve reset_dirty_pages indeed, i will > apply your idea after reset_dirty_pages is merged. > > However, we still prefer to have a separate ioctl for write-protect-all > which cooperates with KVM_GET_DIRTY_LOG to improve live migration that > should not always depend on checkpoint. Ok, I plan to merge the dirty ring pages early in 4.13 development. Paolo
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 05/03/2017 08:28 PM, Paolo Bonzini wrote:
So if I understand correctly this relies on userspace doing:
1) KVM_GET_DIRTY_LOG without write protect
2) KVM_WRITE_PROTECT_ALL_MEM
Writes may happen between 1 and 2; they are not represented in the live
dirty bitmap but it's okay because they are in the snapshot and will
only be used after 2. This is similar to what the dirty page ring
buffer patches do; in fact, the KVM_WRITE_PROTECT_ALL_MEM ioctl is very
similar to KVM_RESET_DIRTY_PAGES in those patches.
You are right. After 1) and 2), the page which has been modified either
in the bitmap returned to userspace or in the bitmap of memslot, i.e,
there is no dirty page lost.
On 03/05/2017 12:52, guangrong.x...@gmail.com wrote:
Comparing with the ordinary algorithm which
write protects last level sptes based on the rmap one by one,
it just simply updates the generation number to ask all vCPUs
to reload its root page table, particularly, it can be done out
of mmu-lock, so that it does not hurt vMMU's parallel.
This is clever.
For processors that have PML, write protecting is only done on large
pages and only for splitting purposes; not for dirty page tracking
process at 4k granularity. In this case, I think that you should do
nothing in the new write-protect-all ioctl?
Good point, thanks for you pointing it out.
Doing nothing in write-protect-all() is not acceptable as it breaks
its semantic. :(
Furthermore, userspace has no knowledge about if PML is enable (it
can be required from sysfs, but it is a good way in QEMU), so it is
difficult for the usespace to know when to use write-protect-all.
Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if
PML is enabled?
Also, I wonder how the alternative write protection mechanism would
affect performance of the dirty page ring buffer patches. You would do
the write protection of all memory at the end of
kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate
ioctl, which is nice. On the other hand, checkpoints would be more
frequent and most pages would be write-protected, so it would be more
expensive to rebuild the shadow page tables...
Yup, write-protect-all can improve reset_dirty_pages indeed, i will
apply your idea after reset_dirty_pages is merged.
However, we still prefer to have a separate ioctl for write-protect-all
which cooperates with KVM_GET_DIRTY_LOG to improve live migration that
should not always depend on checkpoint.
Thanks,
Paolo
@@ -490,6 +511,7 @@ static int kvm_physical_sync_dirty_bitmap(KVMMemoryListener
*kml,
memset(d.dirty_bitmap, 0, allocated_size);
d.slot = mem->slot | (kml->as_id << 16);
+d.flags = kvm_write_protect_all ? KVM_DIRTY_LOG_WITHOUT_WRITE_PROTECT
: 0;
if (kvm_vm_ioctl(s, KVM_GET_DIRTY_LOG, ) == -1) {
DPRINTF("ioctl failed %d\n", errno);
ret = -1;
How would this work when kvm_physical_sync_dirty_bitmap is called from
memory_region_sync_dirty_bitmap rather than
memory_region_global_dirty_log_sync?
You are right, we did not consider the full cases carefully, will fix it
when push it to QEMU formally.
Thank you, Paolo!
Re: [PATCH 0/7] KVM: MMU: fast write protect
On 05/03/2017 08:28 PM, Paolo Bonzini wrote:
So if I understand correctly this relies on userspace doing:
1) KVM_GET_DIRTY_LOG without write protect
2) KVM_WRITE_PROTECT_ALL_MEM
Writes may happen between 1 and 2; they are not represented in the live
dirty bitmap but it's okay because they are in the snapshot and will
only be used after 2. This is similar to what the dirty page ring
buffer patches do; in fact, the KVM_WRITE_PROTECT_ALL_MEM ioctl is very
similar to KVM_RESET_DIRTY_PAGES in those patches.
You are right. After 1) and 2), the page which has been modified either
in the bitmap returned to userspace or in the bitmap of memslot, i.e,
there is no dirty page lost.
On 03/05/2017 12:52, guangrong.x...@gmail.com wrote:
Comparing with the ordinary algorithm which
write protects last level sptes based on the rmap one by one,
it just simply updates the generation number to ask all vCPUs
to reload its root page table, particularly, it can be done out
of mmu-lock, so that it does not hurt vMMU's parallel.
This is clever.
For processors that have PML, write protecting is only done on large
pages and only for splitting purposes; not for dirty page tracking
process at 4k granularity. In this case, I think that you should do
nothing in the new write-protect-all ioctl?
Good point, thanks for you pointing it out.
Doing nothing in write-protect-all() is not acceptable as it breaks
its semantic. :(
Furthermore, userspace has no knowledge about if PML is enable (it
can be required from sysfs, but it is a good way in QEMU), so it is
difficult for the usespace to know when to use write-protect-all.
Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if
PML is enabled?
Also, I wonder how the alternative write protection mechanism would
affect performance of the dirty page ring buffer patches. You would do
the write protection of all memory at the end of
kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate
ioctl, which is nice. On the other hand, checkpoints would be more
frequent and most pages would be write-protected, so it would be more
expensive to rebuild the shadow page tables...
Yup, write-protect-all can improve reset_dirty_pages indeed, i will
apply your idea after reset_dirty_pages is merged.
However, we still prefer to have a separate ioctl for write-protect-all
which cooperates with KVM_GET_DIRTY_LOG to improve live migration that
should not always depend on checkpoint.
Thanks,
Paolo
@@ -490,6 +511,7 @@ static int kvm_physical_sync_dirty_bitmap(KVMMemoryListener
*kml,
memset(d.dirty_bitmap, 0, allocated_size);
d.slot = mem->slot | (kml->as_id << 16);
+d.flags = kvm_write_protect_all ? KVM_DIRTY_LOG_WITHOUT_WRITE_PROTECT
: 0;
if (kvm_vm_ioctl(s, KVM_GET_DIRTY_LOG, ) == -1) {
DPRINTF("ioctl failed %d\n", errno);
ret = -1;
How would this work when kvm_physical_sync_dirty_bitmap is called from
memory_region_sync_dirty_bitmap rather than
memory_region_global_dirty_log_sync?
You are right, we did not consider the full cases carefully, will fix it
when push it to QEMU formally.
Thank you, Paolo!
Re: [PATCH 0/7] KVM: MMU: fast write protect
So if I understand correctly this relies on userspace doing:
1) KVM_GET_DIRTY_LOG without write protect
2) KVM_WRITE_PROTECT_ALL_MEM
Writes may happen between 1 and 2; they are not represented in the live
dirty bitmap but it's okay because they are in the snapshot and will
only be used after 2. This is similar to what the dirty page ring
buffer patches do; in fact, the KVM_WRITE_PROTECT_ALL_MEM ioctl is very
similar to KVM_RESET_DIRTY_PAGES in those patches.
On 03/05/2017 12:52, guangrong.x...@gmail.com wrote:
> Comparing with the ordinary algorithm which
> write protects last level sptes based on the rmap one by one,
> it just simply updates the generation number to ask all vCPUs
> to reload its root page table, particularly, it can be done out
> of mmu-lock, so that it does not hurt vMMU's parallel.
This is clever.
For processors that have PML, write protecting is only done on large
pages and only for splitting purposes; not for dirty page tracking
process at 4k granularity. In this case, I think that you should do
nothing in the new write-protect-all ioctl?
Also, I wonder how the alternative write protection mechanism would
affect performance of the dirty page ring buffer patches. You would do
the write protection of all memory at the end of
kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate
ioctl, which is nice. On the other hand, checkpoints would be more
frequent and most pages would be write-protected, so it would be more
expensive to rebuild the shadow page tables...
Thanks,
Paolo
> @@ -490,6 +511,7 @@ static int
> kvm_physical_sync_dirty_bitmap(KVMMemoryListener *kml,
> memset(d.dirty_bitmap, 0, allocated_size);
>
> d.slot = mem->slot | (kml->as_id << 16);
> +d.flags = kvm_write_protect_all ?
> KVM_DIRTY_LOG_WITHOUT_WRITE_PROTECT : 0;
> if (kvm_vm_ioctl(s, KVM_GET_DIRTY_LOG, ) == -1) {
> DPRINTF("ioctl failed %d\n", errno);
> ret = -1;
How would this work when kvm_physical_sync_dirty_bitmap is called from
memory_region_sync_dirty_bitmap rather than
memory_region_global_dirty_log_sync?
Thanks,
Paolo
Re: [PATCH 0/7] KVM: MMU: fast write protect
So if I understand correctly this relies on userspace doing:
1) KVM_GET_DIRTY_LOG without write protect
2) KVM_WRITE_PROTECT_ALL_MEM
Writes may happen between 1 and 2; they are not represented in the live
dirty bitmap but it's okay because they are in the snapshot and will
only be used after 2. This is similar to what the dirty page ring
buffer patches do; in fact, the KVM_WRITE_PROTECT_ALL_MEM ioctl is very
similar to KVM_RESET_DIRTY_PAGES in those patches.
On 03/05/2017 12:52, guangrong.x...@gmail.com wrote:
> Comparing with the ordinary algorithm which
> write protects last level sptes based on the rmap one by one,
> it just simply updates the generation number to ask all vCPUs
> to reload its root page table, particularly, it can be done out
> of mmu-lock, so that it does not hurt vMMU's parallel.
This is clever.
For processors that have PML, write protecting is only done on large
pages and only for splitting purposes; not for dirty page tracking
process at 4k granularity. In this case, I think that you should do
nothing in the new write-protect-all ioctl?
Also, I wonder how the alternative write protection mechanism would
affect performance of the dirty page ring buffer patches. You would do
the write protection of all memory at the end of
kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate
ioctl, which is nice. On the other hand, checkpoints would be more
frequent and most pages would be write-protected, so it would be more
expensive to rebuild the shadow page tables...
Thanks,
Paolo
> @@ -490,6 +511,7 @@ static int
> kvm_physical_sync_dirty_bitmap(KVMMemoryListener *kml,
> memset(d.dirty_bitmap, 0, allocated_size);
>
> d.slot = mem->slot | (kml->as_id << 16);
> +d.flags = kvm_write_protect_all ?
> KVM_DIRTY_LOG_WITHOUT_WRITE_PROTECT : 0;
> if (kvm_vm_ioctl(s, KVM_GET_DIRTY_LOG, ) == -1) {
> DPRINTF("ioctl failed %d\n", errno);
> ret = -1;
How would this work when kvm_physical_sync_dirty_bitmap is called from
memory_region_sync_dirty_bitmap rather than
memory_region_global_dirty_log_sync?
Thanks,
Paolo
