Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Anthony Liguori wrote: Glauber Costa wrote: Anthony Liguori wrote: No, it can't. Because at the time qemu starts, no vcpu - thread id relationship exists at all. And we don't know when it will. Sure we do. The vcpu - thread id relationship is valid after kvm_init_ap() is called which is after machine init but before the select loop is entered for the first time. Therefore, if you start qemu with -S, then connect on the monitor, and do an info cpus, you could be guaranteed to be told the mapping. I missed that. This changes everything. I now completely agree with you. I'll post patches that expose the relationship, if it's better. The threads are *idle* at this point so there's no harm if they were started on the wrong CPU. You can now taskset to your hearts content and then when you're happy with placement, you can issue a 'cont' so that the VM actually starts running. I saw wrong because you can still taskset the initial creation guaranteeing that the threads are created on the right group of physical CPUs, you just can't specify the exact mapping until you start interacting with the monitor. Regards, Anthony Liguori Regards, Anthony Liguori - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Avi Kivity wrote: Glauber Costa wrote: Hi guys, Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask. However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created. The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities. Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead! A monitor interface would be more useful than a command line option, as it allows you to migrate the vcpus at runtime, and also control hotplugged cpus. For unmanaged use, taskset is probably sufficient to control affinity from the command line. Normally I encourage splitting patches, but this is a bit extreme. 1 and 3 are pointless without each other, 4 and 5, 7 and 8. Hope that doesn't interfere with any pay-per-patch contract. I'll post a new series that just expose the thread ids, so the management tools can use plain taskset. As for the split, there's nothing in my contract, but after all the x86 integration, it became a mania for me. Ingo made me this way. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Avi Kivity wrote: Affinity control is probably useful mostly for numa configurations, where you want to restrict virtual cpus to run on the cores closest to memory. However it may well be that the scheduler is already good enough to do this on its own. In that case, you need to use numactl to set a NUMA policy so it's pretty natural that you would also be using taskset. Assuming you're trying to keep a VM local to a particular node (we don't expose a virtual SRAT/SLIT table so that's all we can sanely do right now), it doesn't matter what CPU each VCPU thread lands on as long as they stay within the node. So taskset is perfectly capable to address this need today. In the brutal world of hypervisors, if your competitor has a feature, you must have it too. I often get asked about cpu pinning in kvm. And the answer to give is, of course, we support it through taskset :-) Regards, Anthony Liguori [I'd like to see how Xen implements swapping, though] - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
[kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Hi guys, Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask. However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created. The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities. Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead! - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Glauber Costa wrote: Hi guys, Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask. However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created. The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities. Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead! So why exactly is this useful? I have a hard time constructing a reasonable use-case in my mind for something like this. Regards, Anthony Liguori - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Anthony Liguori wrote: Glauber Costa wrote: Hi guys, Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask. However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created. The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities. Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead! So why exactly is this useful? I have a hard time constructing a reasonable use-case in my mind for something like this. My main interest is in management tools being able to specify pinning set ups at VM creation time. As I said, it can be done through tools like taskset, but then you'd have to know: * when are the threads created * which thread ids corresponds to each cpu And of course, for an amount of time, the threads will be running in a wrong cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address) - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Joerg Roedel wrote: On Tue, Mar 04, 2008 at 01:21:53PM -0300, Glauber Costa wrote: Hi guys, Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask. However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created. The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities. Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead! Cool, this goes into the same direction as I planned for KVM-NUMA support. Do you plan to extend vcpu pinning into that direction? I don't have any immediate plans, but it is surely interesting. If the patches (or something inspired in them) make it, there's something we can draw support for. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
On Tue, Mar 04, 2008 at 03:11:27PM -0300, Glauber Costa wrote: Joerg Roedel wrote: On Tue, Mar 04, 2008 at 01:21:53PM -0300, Glauber Costa wrote: Hi guys, Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask. However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created. The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities. Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead! Cool, this goes into the same direction as I planned for KVM-NUMA support. Do you plan to extend vcpu pinning into that direction? I don't have any immediate plans, but it is surely interesting. If the patches (or something inspired in them) make it, there's something we can draw support for. There are patches for HVM-NUMA support on Xen developed by André Przywara. I think they are easy to port to KVM. Maybe it is the better aproach than implementing simple vcpu pinning. Joerg -- | AMD Saxony Limited Liability Company Co. KG Operating | Wilschdorfer Landstr. 101, 01109 Dresden, Germany System| Register Court Dresden: HRA 4896 Research | General Partner authorized to represent: Center| AMD Saxony LLC (Wilmington, Delaware, US) | General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Glauber Costa wrote: My main interest is in management tools being able to specify pinning set ups at VM creation time. As I said, it can be done through tools like taskset, but then you'd have to know: * when are the threads created * which thread ids corresponds to each cpu And of course, for an amount of time, the threads will be running in a wrong cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address) A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality. My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it. Regards, Anthony Liguori - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
On Tue, Mar 04, 2008 at 01:28:24PM -0600, Anthony Liguori wrote: Glauber Costa wrote: My main interest is in management tools being able to specify pinning set ups at VM creation time. As I said, it can be done through tools like taskset, but then you'd have to know: * when are the threads created * which thread ids corresponds to each cpu And of course, for an amount of time, the threads will be running in a wrong cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address) A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality. I fully expected to have to run QEMU with -S and then use cont if I were todo CPU pinning from libvirt. The only info I'd need to get is the PID - vCPU mapping data. Then I can use regular Linux taskset capabilities from libvirt to assign the initial pCPU - vCPU mapping and finally run 'cont'. My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it. I agree that adding QEMU commands for stuff which Linux already has APIs and tools is a bad idea. QEMU/KVM is much nicer to manage than Xen, precisely because I can already use Linux APIs process management tools. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Daniel P. Berrange wrote: On Tue, Mar 04, 2008 at 01:28:24PM -0600, Anthony Liguori wrote: Glauber Costa wrote: My main interest is in management tools being able to specify pinning set ups at VM creation time. As I said, it can be done through tools like taskset, but then you'd have to know: * when are the threads created * which thread ids corresponds to each cpu And of course, for an amount of time, the threads will be running in a wrong cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address) A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality. I fully expected to have to run QEMU with -S and then use cont if I were todo CPU pinning from libvirt. The only info I'd need to get is the PID - vCPU mapping data. Then I can use regular Linux taskset capabilities from libvirt to assign the initial pCPU - vCPU mapping and finally run 'cont'. My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it. I agree that adding QEMU commands for stuff which Linux already has APIs and tools is a bad idea. QEMU/KVM is much nicer to manage than Xen, precisely because I can already use Linux APIs process management tools. I totally agree this is ideal, and I did not start this after thinking a little bit about this situation. The main point is that we don't know when the cpus are created, and it does not seem to me that we will without a considerable amount of work. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Anthony Liguori wrote: Glauber Costa wrote: My main interest is in management tools being able to specify pinning set ups at VM creation time. As I said, it can be done through tools like taskset, but then you'd have to know: * when are the threads created * which thread ids corresponds to each cpu And of course, for an amount of time, the threads will be running in a wrong cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address) A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality. No, it can't. Because at the time qemu starts, no vcpu - thread id relationship exists at all. And we don't know when it will. It would be a different story if there were some kind of api that could warn qemu My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. I agree with you that we should stick with linux tools, and that's why I didn't provide any kind of runtime setting via qemu monitor to do this (with the infrastructure, it would be trivial). taskset will do. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it. No, not just because xen does. I do however feel it useful, since starting a vm and then let it run unchanged is definitely an useful use case. And as I tried to show you, I can't see a good way to do that for pinning. Regards, Anthony Liguori - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Glauber Costa wrote: Anthony Liguori wrote: No, it can't. Because at the time qemu starts, no vcpu - thread id relationship exists at all. And we don't know when it will. Sure we do. The vcpu - thread id relationship is valid after kvm_init_ap() is called which is after machine init but before the select loop is entered for the first time. Therefore, if you start qemu with -S, then connect on the monitor, and do an info cpus, you could be guaranteed to be told the mapping. The threads are *idle* at this point so there's no harm if they were started on the wrong CPU. You can now taskset to your hearts content and then when you're happy with placement, you can issue a 'cont' so that the VM actually starts running. I saw wrong because you can still taskset the initial creation guaranteeing that the threads are created on the right group of physical CPUs, you just can't specify the exact mapping until you start interacting with the monitor. Regards, Anthony Liguori Regards, Anthony Liguori - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Glauber Costa wrote: Hi guys, Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask. However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created. The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities. Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead! A monitor interface would be more useful than a command line option, as it allows you to migrate the vcpus at runtime, and also control hotplugged cpus. For unmanaged use, taskset is probably sufficient to control affinity from the command line. Normally I encourage splitting patches, but this is a bit extreme. 1 and 3 are pointless without each other, 4 and 5, 7 and 8. Hope that doesn't interfere with any pay-per-patch contract. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Anthony Liguori wrote: Glauber Costa wrote: Anthony Liguori wrote: No, it can't. Because at the time qemu starts, no vcpu - thread id relationship exists at all. And we don't know when it will. Sure we do. The vcpu - thread id relationship is valid after kvm_init_ap() is called which is after machine init but before the select loop is entered for the first time. Therefore, if you start qemu with -S, then connect on the monitor, and do an info cpus, you could be guaranteed to be told the mapping. The threads are *idle* at this point so there's no harm if they were started on the wrong CPU. You can now taskset to your hearts content and then when you're happy with placement, you can issue a 'cont' so that the VM actually starts running. I saw wrong because you can still taskset the initial creation guaranteeing that the threads are created on the right group of physical CPUs, you just can't specify the exact mapping until you start interacting with the monitor. Good points. Initially I thought we ought to abstract the implementation and not expose the vcpu thread id, but I'm beginning to thing that due the wide variety of options (affinity, page migration, priority, cpu control groups) and the relative obscurity of the feature (which as you point out, isn't needed in the common case), we can export the thread id and let the management tools deal with it directly. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/8] RFC: vcpu pinning at qemu start
Anthony Liguori wrote: Glauber Costa wrote: My main interest is in management tools being able to specify pinning set ups at VM creation time. As I said, it can be done through tools like taskset, but then you'd have to know: * when are the threads created * which thread ids corresponds to each cpu And of course, for an amount of time, the threads will be running in a wrong cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address) A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality. Affinity control is probably useful mostly for numa configurations, where you want to restrict virtual cpus to run on the cores closest to memory. However it may well be that the scheduler is already good enough to do this on its own. My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it. In the brutal world of hypervisors, if your competitor has a feature, you must have it too. I often get asked about cpu pinning in kvm. [I'd like to see how Xen implements swapping, though] -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
