[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-10-20 Thread Mike Krieger
Early results are promising on the no spin lock kernel...we're at 14 instances running it and no lockups (been running it for 3 days). We're going to see how the weekend goes and if successful, roll it out widely. -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-10-17 Thread Mike Krieger
Thanks Stefan, we just re-rotated the machine that locked up twice yesterday with your non-pv-spinlock kernel, we'll keep you updated on what happens. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-10-16 Thread Mike Krieger
Unfortunately, it looks like we're still freezing up with noautogroup enabled (set at boot using Grub). Booted with: Oct 17 00:07:45 localhost kernel: [0.00] Command line: root=UUID=3ad27d04-4ecf-493d-bb19-4710c3caf924 ro console=hvc0 noautogroup uname -a Linux

[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-10-12 Thread Mike Krieger
Just as a note, we run the same workload with no issues using Precise on HVMs, it only reproduces on PV in production, so your findings match our experience. Just double checking because I might have missed something--is there an Ubuntu based setup with auto groups off that doesn't freeze up? --

[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-08-18 Thread Mike Krieger
Hi Stefan, Yes, the same instance that froze (collected it after a reboot). - looking at the same instance type, does it happen on all of them sooner or later or are there exceptions? There is one of our instances of that type that is under the same load but hasn't frozen in weeks. Since

[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-08-18 Thread Mike Krieger
One more note--I tried stressing out the same instance using bonnie++ and some CPU burning yes processes, but it stood up okay. It does seem somehow related to throwing heavy read traffic at PostgreSQL on these instances. -- You received this bug notification because you are a member of Ubuntu

[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-08-16 Thread Mike Krieger
AcpiTables: sudo: unable to resolve host moonshine4-readslave-ssdtest2 AlsaDevices: total 0 crw-rw---T 1 root audio 116, 1 Aug 17 00:39 seq crw-rw---T 1 root audio 116, 33 Aug 17 00:39 timer AplayDevices: aplay: device_list:252: no soundcards found... ApportVersion: 2.0.1-0ubuntu12

[Bug 1011792] BootDmesg.txt

2012-08-16 Thread Mike Krieger
apport information ** Attachment added: BootDmesg.txt https://bugs.launchpad.net/bugs/1011792/+attachment/3264700/+files/BootDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1011792

[Bug 1011792] CurrentDmesg.txt

2012-08-16 Thread Mike Krieger
apport information ** Attachment added: CurrentDmesg.txt https://bugs.launchpad.net/bugs/1011792/+attachment/3264701/+files/CurrentDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1011792

[Bug 1011792] ProcCpuinfo.txt

2012-08-16 Thread Mike Krieger
apport information ** Attachment added: ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1011792/+attachment/3264702/+files/ProcCpuinfo.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1011792

[Bug 1011792] ProcInterrupts.txt

2012-08-16 Thread Mike Krieger
apport information ** Attachment added: ProcInterrupts.txt https://bugs.launchpad.net/bugs/1011792/+attachment/3264703/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1011792] ProcModules.txt

2012-08-16 Thread Mike Krieger
apport information ** Attachment added: ProcModules.txt https://bugs.launchpad.net/bugs/1011792/+attachment/3264704/+files/ProcModules.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1011792

[Bug 1011792] UdevDb.txt

2012-08-16 Thread Mike Krieger
apport information ** Attachment added: UdevDb.txt https://bugs.launchpad.net/bugs/1011792/+attachment/3264705/+files/UdevDb.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1011792 Title:

[Bug 1011792] UdevLog.txt

2012-08-16 Thread Mike Krieger
apport information ** Attachment added: UdevLog.txt https://bugs.launchpad.net/bugs/1011792/+attachment/3264706/+files/UdevLog.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1011792 Title:

[Bug 1011792] Re: Kernel lockup running 3.0.0 and 3.2.0 on multiple EC2 instance types

2012-08-16 Thread Mike Krieger
Hi Stefan, Thanks. We just tried with ami-7dae1b14 and it froze up within a few minutes of having traffic thrown at the instance. Are there any other diagnostics we can run either right before or after it freezes? Thanks. -- You received this bug notification because you are a member of Ubuntu