Re: [libvirt] cgroup cpuset limited by default?
On Tue, Apr 17, 2012 at 11:09:33AM +0530, Srivatsa S. Bhat wrote: On 04/16/2012 09:43 PM, Daniel P. Berrange wrote: On Mon, Apr 16, 2012 at 06:00:22PM +0200, Marc-André Lureau wrote: Hi On Mon, Apr 16, 2012 at 2:32 PM, Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com wrote: On 04/16/2012 05:34 PM, Marc-André Lureau wrote: Did you happen to perform a suspend/resume or a hibernation/restore on your computer? (Or did you do CPU hotplug manually?) If yes, you might be seeing the problem reported at: https://bugzilla.redhat.com/show_bug.cgi?id=714271 yep, thank you very much, that seems to be the reason Thanks for the confirmation. The script below might work as a temporary workaround. This problem has existed for ages now. I wonder if there's a way we get get a userspace workaround implemented. IIRC, pm-utils has the ability to run arbitrary shell scripts upon restore from suspend/hibernate. We could put a temp hack in libvirt which resets the CPU affinity in the top level libvirt cgroup. That would at least make new VMs start with good affinity. To deal with existing running VMs, we would need to record existing affinity before suspend re-store it fully afterwards which is more complicated I don't think that would be all that complicated.. Below is a script that should do the trick, for all cases, including existing running VMs. This script is not at all specific to libvirt by the way (as the problem itself is not at all libvirt specific). It saves the cpuset configuration before suspend and restores it after resume, for all cpusets (not only the ones controlled by libvirt). Of course, it hooks onto the pm-utils mechanism, as you mentioned. Thanks for taking the time to write this. I have opened a BZ against systemd to see if they are willing to ship this script as a standard workaround for the kernel bug, since they are responsible for mounting the cpuset cgroups in Fedora 16 and later https://bugzilla.redhat.com/show_bug.cgi?id=813228 Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] cgroup cpuset limited by default?
On 04/17/2012 03:02 PM, Daniel P. Berrange wrote: On Tue, Apr 17, 2012 at 11:09:33AM +0530, Srivatsa S. Bhat wrote: On 04/16/2012 09:43 PM, Daniel P. Berrange wrote: On Mon, Apr 16, 2012 at 06:00:22PM +0200, Marc-André Lureau wrote: Hi On Mon, Apr 16, 2012 at 2:32 PM, Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com wrote: On 04/16/2012 05:34 PM, Marc-André Lureau wrote: Did you happen to perform a suspend/resume or a hibernation/restore on your computer? (Or did you do CPU hotplug manually?) If yes, you might be seeing the problem reported at: https://bugzilla.redhat.com/show_bug.cgi?id=714271 yep, thank you very much, that seems to be the reason Thanks for the confirmation. The script below might work as a temporary workaround. This problem has existed for ages now. I wonder if there's a way we get get a userspace workaround implemented. IIRC, pm-utils has the ability to run arbitrary shell scripts upon restore from suspend/hibernate. We could put a temp hack in libvirt which resets the CPU affinity in the top level libvirt cgroup. That would at least make new VMs start with good affinity. To deal with existing running VMs, we would need to record existing affinity before suspend re-store it fully afterwards which is more complicated I don't think that would be all that complicated.. Below is a script that should do the trick, for all cases, including existing running VMs. This script is not at all specific to libvirt by the way (as the problem itself is not at all libvirt specific). It saves the cpuset configuration before suspend and restores it after resume, for all cpusets (not only the ones controlled by libvirt). Of course, it hooks onto the pm-utils mechanism, as you mentioned. Thanks for taking the time to write this. I have opened a BZ against systemd to see if they are willing to ship this script as a standard workaround for the kernel bug, since they are responsible for mounting the cpuset cgroups in Fedora 16 and later https://bugzilla.redhat.com/show_bug.cgi?id=813228 Nice! Thanks :-) Regards, Srivatsa S. Bhat IBM Linux Technology Center -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] cgroup cpuset limited by default?
Hi, I noticed recently that my system VMs feel much slower, and that seems to be because of cpuset cgroup which constrain the VM on the first cpu. This doesn't happen with session VMs, since they are not ruled by cgroup. I run f17 with libvirt git, the VMs are created with virt-manager, using mostly default settings vcpu4/vcpu os type arch='x86_64' machine='pc-0.15'hvm/type boot dev='hd'/ /os features acpi/ apic/ pae/ /features clock offset='localtime'/ Is that intended or is it a bug? -- Marc-André Lureau -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] cgroup cpuset limited by default?
On 04/16/2012 05:34 PM, Marc-André Lureau wrote: Hi, I noticed recently that my system VMs feel much slower, and that seems to be because of cpuset cgroup which constrain the VM on the first cpu. This doesn't happen with session VMs, since they are not ruled by cgroup. I run f17 with libvirt git, the VMs are created with virt-manager, using mostly default settings vcpu4/vcpu os type arch='x86_64' machine='pc-0.15'hvm/type boot dev='hd'/ /os features acpi/ apic/ pae/ /features clock offset='localtime'/ Is that intended or is it a bug? Did you happen to perform a suspend/resume or a hibernation/restore on your computer? (Or did you do CPU hotplug manually?) If yes, you might be seeing the problem reported at: https://bugzilla.redhat.com/show_bug.cgi?id=714271 Unfortunately, as of now the kernel still doesn't handle this properly.. IOW, we don't have a kernel fix (yet). Regards, Srivatsa S. Bhat IBM Linux Technology Center -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] cgroup cpuset limited by default?
Hi On Mon, Apr 16, 2012 at 2:32 PM, Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com wrote: On 04/16/2012 05:34 PM, Marc-André Lureau wrote: Did you happen to perform a suspend/resume or a hibernation/restore on your computer? (Or did you do CPU hotplug manually?) If yes, you might be seeing the problem reported at: https://bugzilla.redhat.com/show_bug.cgi?id=714271 yep, thank you very much, that seems to be the reason Unfortunately, as of now the kernel still doesn't handle this properly.. IOW, we don't have a kernel fix (yet). That's sad to learn!, I am going to move all my VMs to session libvirt then :) thanks -- Marc-André Lureau -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] cgroup cpuset limited by default?
On Mon, Apr 16, 2012 at 06:00:22PM +0200, Marc-André Lureau wrote: Hi On Mon, Apr 16, 2012 at 2:32 PM, Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com wrote: On 04/16/2012 05:34 PM, Marc-André Lureau wrote: Did you happen to perform a suspend/resume or a hibernation/restore on your computer? (Or did you do CPU hotplug manually?) If yes, you might be seeing the problem reported at: https://bugzilla.redhat.com/show_bug.cgi?id=714271 yep, thank you very much, that seems to be the reason This problem has existed for ages now. I wonder if there's a way we get get a userspace workaround implemented. IIRC, pm-utils has the ability to run arbitrary shell scripts upon restore from suspend/hibernate. We could put a temp hack in libvirt which resets the CPU affinity in the top level libvirt cgroup. That would at least make new VMs start with good affinity. To deal with existing running VMs, we would need to record existing affinity before suspend re-store it fully afterwards which is more complicated Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] cgroup cpuset limited by default?
On 04/16/2012 09:43 PM, Daniel P. Berrange wrote:
On Mon, Apr 16, 2012 at 06:00:22PM +0200, Marc-André Lureau wrote:
Hi
On Mon, Apr 16, 2012 at 2:32 PM, Srivatsa S. Bhat
srivatsa.b...@linux.vnet.ibm.com wrote:
On 04/16/2012 05:34 PM, Marc-André Lureau wrote:
Did you happen to perform a suspend/resume or a hibernation/restore
on your computer? (Or did you do CPU hotplug manually?)
If yes, you might be seeing the problem reported at:
https://bugzilla.redhat.com/show_bug.cgi?id=714271
yep, thank you very much, that seems to be the reason
Thanks for the confirmation. The script below might work as a
temporary workaround.
This problem has existed for ages now. I wonder if there's a way we get get
a userspace workaround implemented.
IIRC, pm-utils has the ability to run arbitrary shell scripts upon
restore from suspend/hibernate. We could put a temp hack in libvirt
which resets the CPU affinity in the top level libvirt cgroup. That
would at least make new VMs start with good affinity. To deal with
existing running VMs, we would need to record existing affinity
before suspend re-store it fully afterwards which is more complicated
I don't think that would be all that complicated.. Below is a script
that should do the trick, for all cases, including existing running VMs.
This script is not at all specific to libvirt by the way (as the problem
itself is not at all libvirt specific). It saves the cpuset configuration
before suspend and restores it after resume, for all cpusets (not only
the ones controlled by libvirt). Of course, it hooks onto the pm-utils
mechanism, as you mentioned.
Regards,
Srivatsa S. Bhat
IBM Linux Technology Center
Note: Put this script in /usr/lib or lib64/pm-utils/sleep.d by prefixing
appropriate number (eg: /usr/lib64/pm-utils/sleep.d/02cpusets.sh). And then
give execute permissions to the script. It will save all the cpusets' cpu
configuration before suspend and restore them after resume. This is a
workaround until the problem gets fixed in the kernel.
cpusets.sh
#! /bin/bash
# Script to save and restore the cpusets' cpu configuration after a
# suspend/resume cycle.
# This is a workaround for a Linux kernel bug which results in cpusets
# being reduced to a single cpu (boot cpu) after a suspend/resume
# cycle.
#
# Author: Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com
# 16 March 2012
#
save_cpusets()
{
rm -f saved_cpusets.txt
# Check if cpusets are mounted separately
dir=`mount | grep cpuset`
if [ -z $dir ]; then
# Check if atleast cgroup is mounted
dir=`mount | grep cgroup`
if [ -z $dir ]; then
#Nothing to be done, as cpusets are not being used
exit 0
fi
fi
dir=`mount | grep cpuset | cut -d' ' -f3`
# Omit the root cpuset (using mindepth parameter) since it is read-only
# (and will be restored properly after resume by the kernel itself).
# Sorting is necessary because otherwise the hierarchy is not preserved
# and restore_cpusets() will have trouble restoring the cpusets.
# This is because of the rule that a child cpuset cannot have a cpu that
# is not present in its parent cpuset.
find $dir -mindepth 2 -name cpuset.cpus -type f -print | sort
saved_path.txt
# Delete saved_path.txt if it is NULL
val=`cat saved_path.txt`
if [ -z $val ]; then
rm -f saved_path.txt
exit 0
fi
for cpuset in `cat saved_path.txt`
do
cat $cpuset saved_cpusets.txt
done
paste saved_path.txt saved_cpusets.txt temp.txt
mv temp.txt saved_cpusets.txt
rm -f saved_path.txt
}
restore_cpusets()
{
if [ ! -f saved_cpusets.txt ]; then
# Nothing to be done
exit 0
fi
while read line
do
cpuset_path=`echo $line | cut -d' ' -f1`
value=`echo $line | cut -d' ' -f2`
echo $value $cpuset_path
done saved_cpusets.txt
rm -f saved_cpusets.txt
}
case $1 in
suspend|hibernate)
save_cpusets
;;
resume|thaw)
restore_cpusets
;;
*)
;;
esac
exit $?
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
