[libvirt] [RFC PATCH v1 0/4] Add cpu hotplug support to libvirt.
It seems that libvirt is not cpu hotplug aware. Please refer to the following problem. 1. At first, we have 2 cpus. # cat /cgroup/cpuset/cpuset.cpus 0-1 # cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus 0-1 2. And we have a vm1 with following configuration. cputune vcpupin vcpu='0' cpuset='1'/ hypervisorpin cpuset='1'/ /cputune 3. Offline cpu1. # echo 0 /sys/devices/system/cpu/cpu1/online # cat /sys/devices/system/cpu/cpu1/online 0 # cat /cgroup/cpuset/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/lxc/cpuset.cpus 0 4. Online cpu1. # echo 1 /sys/devices/system/cpu/cpu1/online # cat /sys/devices/system/cpu/cpu1/online 1 # cat /cgroup/cpuset/cpuset.cpus 0-1 # cat /cgroup/cpuset/libvirt/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/lxc/cpuset.cpus 0 Here，cgroup updated cpuset.cpus，but not for libvirt directory，and also qemu and lxc directory. vm1 cannot be started again. # virsh start vm1 error: Failed to start domain vm1 error: Unable to set cpuset.cpus: Permission denied And libvird gave the following errors. 2012-07-17 07:30:22.478+: 3118: error : qemuSetupCgroupVcpuPin:498 : Unable to set cpuset.cpus: Permission denied These patches resolves this problem by listening on the netlink for cpu hotplug event. When the netlink service gets the cpu hotplug event, it will attract the cpuid in the message, and add it into cpuset.cpus in: /cgroup/cpuset/libvirt /cgroup/cpuset/libvirt/qemu /cgroup/cpuset/libvirt/lxc Tang Chen (4): Add cpu hotplug handler for netlink service. Register cpu hotplug netlink handler for libvirtd. Register cpu hotplug netlink handler for qemu driver. Register cpu hotplug netlink handler for lxc driver. daemon/libvirtd.c | 11 +++ include/libvirt/virterror.h |2 + src/Makefile.am |1 + src/libvirt_private.syms|5 + src/lxc/lxc_driver.c|8 ++ src/qemu/qemu_driver.c |8 ++ src/util/cgroup.c |6 +- src/util/cgroup.h |4 + src/util/hotplug.c | 221 +++ src/util/hotplug.h | 32 +++ src/util/virterror.c|3 +- 11 files changed, 297 insertions(+), 4 deletions(-) create mode 100644 src/util/hotplug.c create mode 100644 src/util/hotplug.h -- 1.7.10.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v1 0/4] Add cpu hotplug support to libvirt.
On 09/03/2012 08:06 AM, Tang Chen wrote: It seems that libvirt is not cpu hotplug aware. Please refer to the following problem. 1. At first, we have 2 cpus. # cat /cgroup/cpuset/cpuset.cpus 0-1 # cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus 0-1 2. And we have a vm1 with following configuration. cputune vcpupin vcpu='0' cpuset='1'/ hypervisorpin cpuset='1'/ /cputune 3. Offline cpu1. # echo 0 /sys/devices/system/cpu/cpu1/online # cat /sys/devices/system/cpu/cpu1/online 0 # cat /cgroup/cpuset/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/lxc/cpuset.cpus 0 4. Online cpu1. # echo 1 /sys/devices/system/cpu/cpu1/online # cat /sys/devices/system/cpu/cpu1/online 1 # cat /cgroup/cpuset/cpuset.cpus 0-1 # cat /cgroup/cpuset/libvirt/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus 0 # cat /cgroup/cpuset/libvirt/lxc/cpuset.cpus 0 Here，cgroup updated cpuset.cpus，but not for libvirt directory，and also qemu and lxc directory. vm1 cannot be started again. # virsh start vm1 error: Failed to start domain vm1 error: Unable to set cpuset.cpus: Permission denied And libvird gave the following errors. 2012-07-17 07:30:22.478+: 3118: error : qemuSetupCgroupVcpuPin:498 : Unable to set cpuset.cpus: Permission denied These patches resolves this problem by listening on the netlink for cpu hotplug event. When the netlink service gets the cpu hotplug event, it will attract the cpuid in the message, and add it into cpuset.cpus in: /cgroup/cpuset/libvirt /cgroup/cpuset/libvirt/qemu /cgroup/cpuset/libvirt/lxc Hi, this approach requires that libvirtd keeps running through the entire lifecycle of a guest. That is something that cannot be safely assumed and therefore hotplug events can be missed. That means that libvirt must synchronize the hypervisors cpusets with the host's current CPU states. You could do that for instance when registering the callback. -- Mit freundlichen Grüßen/Kind Regards Viktor Mihajlovski IBM Deutschland Research Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v1 0/4] Add cpu hotplug support to libvirt.
On 09/03/2012 04:42 PM, Viktor Mihajlovski wrote: Hi, this approach requires that libvirtd keeps running through the entire lifecycle of a guest. That is something that cannot be safely assumed and therefore hotplug events can be missed. That means that libvirt must synchronize the hypervisors cpusets with the host's current CPU states. You could do that for instance when registering the callback. Yes, I will fix it soon in the next version. Thanks. :) -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
