Re: KVM and the OOM-Killer
On Thu, May 13, 2010 at 01:20:31PM +0100, James Stevens wrote: > We have a KVM host with 48Gb of RAM and run about 20 KVM clients on it. > After some time - different time depending on the kernel version - the > VM host kernel will start OOM-Killing the VM clients, even when there is > lots of free RAM (>10Gb) and free SWAP (>34Gb). It seems going to a 64 bit kernel is what you want, but I thought it worth mentioning the available method to say "try not to OOM-kill *this* process": echo "-16" > /proc//oom_adj I go through some convolutions in starting KVM VMs so that this gets done automatically for them. It basically scores the process *way* down for consideration to be OOM-killed. Handy if you might accidentally run something on the KVM host that chews up all the available memory. It should then get killed instead of the VMs. You can also use '-17' to exclude the process from consideration entirely. See /Documentation/filesystems/proc.txt for details. -- - Athanasius = Athanasius(at)miggy.org / http://www.miggy.org/ Finger athan(at)fysh.org for PGP key "And it's me who is my enemy. Me who beats me up. Me who makes the monsters. Me who strips my confidence." Paula Cole - ME signature.asc Description: Digital signature
Re: Synchronized time with kvm_clock
On Mon, Apr 26, 2010 at 10:32:51AM -0400, John Buswell wrote: > You don't need to run ntp on each guest. You can enable rtc support in > the guest kernel and on the hypervisor. Run ntp client on the hypervisor > via cron, and use hwclock -w on the hypervisor after you run ntp, to > sync the hardware clock to the system clock (which is now updated by > ntpdate). On the guests, periodically run hwclock -s to set the system > clock from the hw clock. What a *horribly* hacky way to do it, meaning you'll get time warps all over the place, admittedly of short intervals if you run those cron jobs often enough. It seems much simpler to me to simply run ntpd in all the guests. It's not like the extra CPU or bandwidth is going to be a problem. At the very least you want to run ntpd, not ntpdate out of cron, in the hypervisor, and only use cron for those hwclock -w's. > This seems to work extremely well, the clocksource on the guests as > kvm_clock, and as long as you have the clocksource as hpet or acpi_pm on > the hypervisor, there doesn't seem to be any problems with keeping time. > > The only thing I've noticed is that when you reboot, the very first > guest will have the wrong time on boot, so the uptime is messed up. And I think many people would find this unacceptable. Really, I appreciate that "keep the time sync'd via ntpd on the hypervisor and have it passed accurately to the guests" has a certain elegant simplicity about it. But if you achieve the latter by periodically resyncing against what the guest sees as its hardware clock you've lost that elegance again. It really needs to 'just work' via KVM code in the guest kernel using the exact same time as the hypervisor kernel is supplying. -- - Athanasius = Athanasius(at)miggy.org / http://www.miggy.org/ Finger athan(at)fysh.org for PGP key "And it's me who is my enemy. Me who beats me up. Me who makes the monsters. Me who strips my confidence." Paula Cole - ME signature.asc Description: Digital signature
Re: Clocksource tsc unstable (delta = -4398046474878 ns)
On Tue, Mar 30, 2010 at 10:08:28AM +0200, Sebastian Hetze wrote: > On Mon, Mar 29, 2010 at 11:31:13AM +0100, Athanasius wrote: > > I didn't see any such messages but I've had a recent experience with > > the time on one KVM host leaping *forwards* approx. 5 and 2.5 hours in > > two separate incidents. Eerily the exact jumps, as best I can tell from > > logs are of 17592 and 8796 seconds, give or take a second or two. If > > you look at these as nanoseconds then that's 'exactly' 2^44 and 2^43 > > nanoseconds. > > What I've done that seems to have avoided this happening again is drop > > KVM_CLOCK kernel option from the kvm guests' kernel. > > To my understanding, kvm-clock is the best and most reliable clocksource > available, so I do not think it is a good idea to disable it. > > There is a lot of bit shift operation happening with the clocksources, > so there may be a real bug hidden somewhere in the code. > Somehow ntp adjustment is involved, can this cause such huge steps? > Im my case, I actually have NTP running in the guest. However, the > statistics show a pretty stable timing here. This is one thing thing to note, I *was* running ntpd in the affected guest (and rather obviously, I still am). If there's some bad interaction between KVM_CLOCK and ntpd it needs documenting in the first instance and preferably also fixing. -- - Athanasius = Athanasius(at)miggy.org / http://www.miggy.org/ Finger athan(at)fysh.org for PGP key "And it's me who is my enemy. Me who beats me up. Me who makes the monsters. Me who strips my confidence." Paula Cole - ME -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Clocksource tsc unstable (delta = -4398046474878 ns)
On Sun, Mar 28, 2010 at 01:46:35PM +0200, Sebastian Hetze wrote: > this message appeared in the KVM guest kern.log last night: > > Mar 27 22:35:30 guest kernel: [260041.559462] Clocksource tsc unstable (delta > = -4398046474878 ns) > > The guest is running a 2.6.31-20-generic-pae ubuntu kernel with > hrtimer-tune-hrtimer_interrupt-hang-logic.patch applied. > > If I understand things correct, in kernel/time/clocksource.c > clocksource_watchdog() checks all the > /sys/devices/system/clocksource/clocksource0/available_clocksource > every 0.5sec for an delta of more than 0.0625s. So the tsc must have > changed more than one hour within two subsequent calls of > clocksource_watchdog. No event in the host nor anything in the > guest gives reasonable cause for this step. > > However, the number 4398046474878 is only 36226 ns away from > 4*1024*1024*1024*1024 I didn't see any such messages but I've had a recent experience with the time on one KVM host leaping *forwards* approx. 5 and 2.5 hours in two separate incidents. Eerily the exact jumps, as best I can tell from logs are of 17592 and 8796 seconds, give or take a second or two. If you look at these as nanoseconds then that's 'exactly' 2^44 and 2^43 nanoseconds. What I've done that seems to have avoided this happening again is drop KVM_CLOCK kernel option from the kvm guests' kernel. This is with a Debian squeeze (testing) KVM host running 2.6.33 from vanilla sources and my own config. The guests are Debian lenny (stable) and were also running a 2.6.33 kernel from vanilla sources and my own (different, to match the virtual hardware in a KVM guest) config. Both systems/kernels are 64 bit. The base machine is a Dell R210 with an Intel Xeon X3450 quad-core CPU, with the hyper-threading enabled to give 8 visible CPUs in Linux. This only happened on one of the two guests, the much busier one (it does shell accounts, email, IMAP/POP3, a small news server and NFS serves web pages to the other guest which only runs apache2 and nagios3). It took around 2-3 days to see the problem both times. Without KVM_CLOCK it's been up and stable for well over a week now. Without KVM_CLOCK the only clocksource is acpi_pm and thus that is being used. I didn't test forcing that with a boot-time parameter and KVM_CLOCK still enabled. Given turning KVM_CLOCK off fixed my problem and the problem repeating itself causes all manner of trouble given how busy the machine is I'm not really willing to test alternative fixes. -- - Athanasius = Athanasius(at)miggy.org / http://www.miggy.org/ Finger athan(at)fysh.org for PGP key "And it's me who is my enemy. Me who beats me up. Me who makes the monsters. Me who strips my confidence." Paula Cole - ME signature.asc Description: Digital signature
