[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
>- trusty and vivid kernels with nohz=off have survived for a full day without any lockups. They both hung last night. So in summary: Neither nohz=off nor older kernels help here. This really seems to be a matter of luck/what's going on on the host system. ** Summary changed: - [arm64] lockups when idle if tickless (nohz=on) is used + [arm64] lockups some time after booting -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups some time after booting To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
Hi ! kvm cmdline : https://paste.ubuntu.com/17430254/ cpuinfo : https://paste.ubuntu.com/17430265/ dmesg: https://paste.ubuntu.com/17430277/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
I'd like to factor out if we are missing IRQs on the host and inside the VM, so can both be booted with kernel parameter: irqpoll unfortunately this can eat more cpu cycles, so I'm reluctant to ask for this to be used, but I'm wondering of the host or VM are occasionally missing timer wakeups -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
Colin Ian King [2016-06-17 10:50 -]: > I'm trying to get a reliable reproducer on a similarly sized aarch64 > host. Just so that I'm not missing anything, what is the entire command > line being used on the host to run the VM? I can't determine this. I asked Junien on IRC to put it here. > Also, what is /proc/cmdline on the VM? Aside from the "nohz=off" it's rather unsurprising: BOOT_IMAGE=/boot/vmlinuz-4.2.0-38-generic root=UUID=b98e4d93-8d8f-4349 -a6ce-b5a87cdb2edd ro nohz=off -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
And the /proc/cmdline info from the host would be of some use to see if anything special there is being used. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
I'm trying to get a reliable reproducer on a similarly sized aarch64 host. Just so that I'm not missing anything, what is the entire command line being used on the host to run the VM? Also, what is /proc/cmdline on the VM? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
Further notekeeping: - 4 dd's (xenial+nohz=off) has survived for half a day, then the instance crashed on something else. - trusty and vivid kernels with nohz=off have survived for a full day without any lockups. lxd on trusty kernel causes a lot of leaked "FREEZED/FREEZING" containers, but that's unrelated and does not happen with the vivid kernel. So it's unclear whether this combination is stable or the lockups are just reduced, or it just was lucky. - trusty kernel without the nohz option locked up a few minutes after reboot, without actually running any test. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
If 4 dd's work OK, it may be worth running a minimal sleep loop: while true; do sleep 0.5; done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
Hang still occurs with xenial kernel and one instance of nice -n 19 dd if=/dev/zero of=/dev/null bs=1024 & I have now rebooted and started four dd's, so that all four CPUs should remain busy constantly. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
FTR, running the trusty kernel on xenial userspace does not work: http://paste.ubuntu.com/17392362/ cking | pitti, syscall 384 on aarch64 is getrandom() and that does not exist on trusty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
> It may be worth trying nohz=off on the host as well Junien did that on the nova compute host, and no change. Processes in the instance still freeze. This is actually also consistent with the observation that this apparently does not happen with the trusty kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
Oh noes! I'm still getting "task * blocked for more than 120 seconds" hangs even with nohz=off :-( Is there another option which I could try? ** Changed in: auto-package-testing Status: Fix Released => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
It may be worth trying nohz=off on the host as well, just as an experiment to see if this also improves things. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
We've also been running into this issue on ScalingStack instances recently; I got this traceback which seems to strongly implicate nohz as the problem area: https://pastebin.canonical.com/158640/ Presently testing @pitti's workaround on a number of different sized instances to confirm. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1531768] Re: [arm64] lockups when idle if tickless (nohz=on) is used
That's great news! I'll try and figure out what the root cause is. Let me know if there are other issues. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: [arm64] lockups when idle if tickless (nohz=on) is used To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1531768/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
