Re: System hangs on running kernbench
* Srivatsa Vaddagiri <[EMAIL PROTECTED]> wrote: > Basically, "make -s -j" workload hanged the machine, leading to lot of > OOM killings. This was on a 8-cpu machine with no swap space > configured and 4GB RAM. The same workload works "fine" (runs to > completion) on 2.6.22. while i agree that the 32msec was too low, i think the problem is that "make -s -j" is a workload that has no guarantee of "success" on that system. The box does not have enough RAM to service it and does not have enough swap to survive it. In make -j, jobs are started without any throttling whatsoever. _Any_ control mechanism within the kernel can act as an "accidental throttle": for example IO could artificially slow it down to reduce job rate and keep RAM usage below the critical level. Or a kernel bug could cause tasks to be delayed and thus let the make -j "succeed". Or some bad kernel inefficiency in sys_fork() could have this effect too. It is very important that we dont look at every random number that a system can produce as a "benchmark", we really have to consider what happens behind it. > I played with the scheduler tunables a bit and found that the problem > goes away if I set sched_granularity_ns to 100ms (default value 32ms). yep - 32msecs was too low, please try -rc1 too: i've increased the granularity limit so it should be larger than 32ms. Reduce CONFIG_HZ as well if you are on a more server-type system. > So my theory is this: 32ms preemption granularity is too low value for > any compile thread to make "usefull" progress. As a result of this > rapid context switch, job retiral rate slows down compared to job > arrival rate. This builds up job pressure on the system very quickly > (than may have happened with 100ms default granularity_ns or 2.6.22 > kernel), leading to OOM killings (and hang). By increasing the granularity the timings change - one can imagine workloads where _reducing_ the granularity would result in an effective throttling of the workload. I'm sure a workload could be constructed on the old scheduler too where its 100 msecs isnt enough either, only 200msecs would help. That thinking never ends - you cannot tune non-throttled workloads. We've got to be really careful about this. Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: System hangs on running kernbench
On Wed, Jul 18, 2007 at 01:26:48PM +0530, Dhaval Giani wrote: > Hi Andrew, > > I was running kernbench on top of 2.6.22-rc6-mm1 and I got a Hangcheck > alert (This is when kernbench reached make -j). > > Also make -j is hanging. [refer http://marc.info/?l=linux-kernel&m=118474574807055 for complete report of this bug] Ingo, Dhaval tracked the root cause of this problem to be in cfs (btw cfs patches weren't git-bisect safe). Basically, "make -s -j" workload hanged the machine, leading to lot of OOM killings. This was on a 8-cpu machine with no swap space configured and 4GB RAM. The same workload works "fine" (runs to completion) on 2.6.22. I played with the scheduler tunables a bit and found that the problem goes away if I set sched_granularity_ns to 100ms (default value 32ms). So my theory is this: 32ms preemption granularity is too low value for any compile thread to make "usefull" progress. As a result of this rapid context switch, job retiral rate slows down compared to job arrival rate. This builds up job pressure on the system very quickly (than may have happened with 100ms default granularity_ns or 2.6.22 kernel), leading to OOM killings (and hang). >From a user perspective, who is running with default granularity_ns value, this may be seen as a regression. Perhaps, these new tunables in cfs are something for users to become used to and tune it to approp setting for their system. It would have been nice for kernel to auto-tune the settings based on workload, but I guess that's harder. -- Regards, vatsa - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: System hangs on running kernbench
> (gdb) thread 6
> [Switching to thread 6 (process 6233)]#0 __do_softirq ()
> at kernel/softirq.c:231
> 231 if (pending & 1) {
> (gdb) bt
> #0 __do_softirq () at kernel/softirq.c:231
> #1 0xc012998b in do_softirq () at kernel/softirq.c:269
> #2 0xc0129a09 in irq_exit () at kernel/softirq.c:305
> #3 0xc0117443 in smp_apic_timer_interrupt (regs=Variable "regs" is not
> available.
> )
> at arch/i386/kernel/apic.c:592
> #4 0xc0105877 in apic_timer_interrupt () at include/asm/current.h:11
> #5 0xc0564480 in contig_page_data ()
> #6 0x0007 in ?? ()
> #7 0x0001 in ?? ()
> #8 0x in ?? ()
Looks interesting.
--
regards,
Dhaval
I would like to change the world but they don't give me the source code!
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: System hangs on running kernbench
On Wed, Jul 18, 2007 at 08:16:37PM +0530, Dhaval Giani wrote:
> Hi Andrew,
>
> On Wed, Jul 18, 2007 at 03:11:42PM +0530, Dhaval Giani wrote:
> > On Wed, Jul 18, 2007 at 01:07:00AM -0700, Andrew Morton wrote:
> > > On Wed, 18 Jul 2007 13:26:48 +0530 Dhaval Giani <[EMAIL PROTECTED]> wrote:
> > >
> > In the meantime I will go and check if it was there in 2.6.22-rc4-mm2
> >
>
> It is hanging with 2.6.22-rc4-mm2 as well as on the latest git on
> kernel.org (2.6.22-git10).
>
> I will get back to you with more information as soon as I have it.
Hi Andrew,
I've got a crash dump and stack traces. They are as follows (The trace
is on 2.6.22-git10)
(gdb) thread 1
[Switching to thread 1 (process 8096)]#0 delay_tsc (loops=1)
at include/asm/msr.h:64
64 {
(gdb) bt
#0 delay_tsc (loops=1) at include/asm/msr.h:64
#1 0xc0245130 in __delay (loops=Variable "loops" is not available.
) at arch/i386/lib/delay.c:74
#2 0xc0247115 in __spin_lock_debug (lock=0xc0564480)
at lib/spinlock_debug.c:111
#3 0xc02471cc in _raw_spin_lock (lock=0xc0564480) at lib/spinlock_debug.c:132
#4 0xc041ad3e in _spin_lock_irq (lock=0xc0564480) at kernel/spinlock.c:105
#5 0xc015ff2c in shrink_active_list (nr_pages=32, zone=0xc0563300,
sc=0xd65b3e60, priority=5) at mm/vmscan.c:926
#6 0xc01602a3 in shrink_zone (priority=5, zone=0xc0563300, sc=0xd65b3e60)
at mm/vmscan.c:1044
#7 0xc016036c in shrink_zones (priority=5, zones=0xc056584c, sc=0xd65b3e60)
at mm/vmscan.c:1101
#8 0xc0160488 in try_to_free_pages (zones=0xc056584c, order=Variable "order"
is not available.
)
at mm/vmscan.c:1153
#9 0xc015c190 in __alloc_pages (gfp_mask=688338, order=0, zonelist=0xc0565848)
at mm/page_alloc.c:1336
#10 0xc0165285 in do_anonymous_page (mm=0xe4498280, vma=0xd3afef3c,
address=3083890688, page_table=0xd65ca838, pmd=0xe3e33df0, write_access=1)
at include/linux/gfp.h:100
#11 0xc0165a58 in __handle_mm_fault (mm=0xe4498280, vma=0xd3afef3c,
address=3083890688, write_access=1) at mm/memory.c:2549
#12 0xc041c984 in do_page_fault (regs=0xd65b3fb8, error_code=6)
at include/linux/mm.h:776
#13 0xc041b37a in error_code () at include/linux/sched.h:13
#14 0x006c in ?? ()
#15 0x001b in ?? ()
#16 0x in ?? ()
(gdb) thread 2
[Switching to thread 2 (process 7371)]#0 __spin_lock_debug (lock=0xc0564480)
at include/asm/spinlock.h:88
88 {
(gdb) bt
#0 __spin_lock_debug (lock=0xc0564480) at include/asm/spinlock.h:88
#1 0xc02471cc in _raw_spin_lock (lock=0xc0564480) at lib/spinlock_debug.c:132
#2 0xc041ad3e in _spin_lock_irq (lock=0xc0564480) at kernel/spinlock.c:105
#3 0xc0160181 in shrink_active_list (nr_pages=32, zone=0xc0563300, sc=Variable
"sc" is not available.
)
at mm/vmscan.c:994
#4 0xc01602a3 in shrink_zone (priority=5, zone=0xc0563300, sc=0xeea67e60)
at mm/vmscan.c:1044
#5 0xc016036c in shrink_zones (priority=5, zones=0xc056584c, sc=0xeea67e60)
at mm/vmscan.c:1101
#6 0xc0160488 in try_to_free_pages (zones=0xc056584c, order=Variable "order"
is not available.
)
at mm/vmscan.c:1153
#7 0xc015c190 in __alloc_pages (gfp_mask=688338, order=0, zonelist=0xc0565848)
at mm/page_alloc.c:1336
#8 0xc0165285 in do_anonymous_page (mm=0xeeaabdc0, vma=0xf137c7ac,
address=3084021760, page_table=0xeeabf938, pmd=0xeeaafdf0, write_access=1)
at include/linux/gfp.h:100
#9 0xc0165a58 in __handle_mm_fault (mm=0xeeaabdc0, vma=0xf137c7ac,
address=3084021760, write_access=1) at mm/memory.c:2549
#10 0xc041c984 in do_page_fault (regs=0xeea67fb8, error_code=6)
at include/linux/mm.h:776
#11 0xc041b37a in error_code () at include/linux/sched.h:13
#12 0x002c in ?? ()
#13 0x000b in ?? ()
#14 0x in ?? ()
(gdb) thread 3
[Switching to thread 3 (process 8392)]#0 __spin_lock_debug (lock=0xc0564480)
at include/asm/spinlock.h:88
88 {
(gdb) bt
#0 __spin_lock_debug (lock=0xc0564480) at include/asm/spinlock.h:88
#1 0xc02471cc in _raw_spin_lock (lock=0xc0564480) at lib/spinlock_debug.c:132
#2 0xc041ad3e in _spin_lock_irq (lock=0xc0564480) at kernel/spinlock.c:105
#3 0xc016000f in shrink_active_list (nr_pages=32, zone=0xc0563300, sc=Variable
"sc" is not available.
)
at mm/vmscan.c:950
#4 0xc01602a3 in shrink_zone (priority=5, zone=0xc0563300, sc=0xf733de60)
at mm/vmscan.c:1044
#5 0xc016036c in shrink_zones (priority=5, zones=0xc056584c, sc=0xf733de60)
at mm/vmscan.c:1101
#6 0xc0160488 in try_to_free_pages (zones=0xc056584c, order=Variable "order"
is not available.
)
at mm/vmscan.c:1153
#7 0xc015c190 in __alloc_pages (gfp_mask=688338, order=0, zonelist=0xc0565848)
at mm/page_alloc.c:1336
#8 0xc0165285 in do_anonymous_page (mm=0xf3054940, vma=0xf60496a4,
address=135188540, page_table=0xd275c768, pmd=0xf317d200, write_access=1)
at include/linux/gfp.h:100
#9 0xc0165a58 in __handle_mm_fault (mm=0xf3054940, vma=0xf60496a4,
address=135188540, write_access=1) at mm/memory.c:2549
#10 0xc041c984 in do_page_fault (regs=0xf733dfb8, e
Re: System hangs on running kernbench
Hi Chuck, On Wed, Jul 18, 2007 at 11:08:18AM -0400, Chuck Ebbert wrote: > On 07/18/2007 05:41 AM, Dhaval Giani wrote: > > On Wed, Jul 18, 2007 at 01:07:00AM -0700, Andrew Morton wrote: > >> On Wed, 18 Jul 2007 13:26:48 +0530 Dhaval Giani <[EMAIL PROTECTED]> wrote: > >> > Softlockup is broken in 2.6.22. > The system does not hang on 2.6.22. It is however still hanging with yesterday's git plus Ingo's softlockup patch. -- regards, Dhaval I would like to change the world but they don't give me the source code! - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: System hangs on running kernbench
On 07/18/2007 05:41 AM, Dhaval Giani wrote:
> On Wed, Jul 18, 2007 at 01:07:00AM -0700, Andrew Morton wrote:
>> On Wed, 18 Jul 2007 13:26:48 +0530 Dhaval Giani <[EMAIL PROTECTED]> wrote:
>>
>>> I was running kernbench on top of 2.6.22-rc6-mm1 and I got a Hangcheck
>>> alert (This is when kernbench reached make -j).
>> hm, never had a report of that before. It's the first time I've seen
>> hangcheck produce anything useful, frankly.
>>
>> Was the softlockup detector not enabled?
>
> I notice CONFIG_DETECT_SOFTLOCKUP=y
>>> Also make -j is hanging.
>> Please try to capture the full sysrq-T output when it is hung.
>
> Available at http://dhaval.giani.googlepages.com/sysrq-t-trace.bz2
>
> In the meantime I will go and check if it was there in 2.6.22-rc4-mm2
>
Softlockup is broken in 2.6.22.
===
Subject: fix the softlockup watchdog to actually work
From: Ingo Molnar <[EMAIL PROTECTED]>
this Xen related commit:
commit 966812dc98e6a7fcdf759cbfa0efab77500a8868
Author: Jeremy Fitzhardinge <[EMAIL PROTECTED]>
Date: Tue May 8 00:28:02 2007 -0700
Ignore stolen time in the softlockup watchdog
broke the softlockup watchdog to never report any lockups. (!)
print_timestamp defaults to 0, this makes the following condition
always true:
if (print_timestamp < (touch_timestamp + 1) ||
and we'll in essence never report soft lockups.
apparently the functionality of the soft lockup watchdog was never
actually tested with that patch applied ...
[this is -stable material too.]
Signed-off-by: Ingo Molnar <[EMAIL PROTECTED]>
---
kernel/softlockup.c |7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
Index: linux/kernel/softlockup.c
===
--- linux.orig/kernel/softlockup.c
+++ linux/kernel/softlockup.c
@@ -79,10 +79,11 @@ void softlockup_tick(void)
print_timestamp = per_cpu(print_timestamp, this_cpu);
/* report at most once a second */
- if (print_timestamp < (touch_timestamp + 1) ||
- did_panic ||
- !per_cpu(watchdog_task, this_cpu))
+ if ((print_timestamp >= touch_timestamp &&
+ print_timestamp < (touch_timestamp + 1)) ||
+ did_panic || !per_cpu(watchdog_task, this_cpu)) {
return;
+ }
/* do not print during early bootup: */
if (unlikely(system_state != SYSTEM_RUNNING)) {
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: System hangs on running kernbench
Hi Andrew, On Wed, Jul 18, 2007 at 03:11:42PM +0530, Dhaval Giani wrote: > On Wed, Jul 18, 2007 at 01:07:00AM -0700, Andrew Morton wrote: > > On Wed, 18 Jul 2007 13:26:48 +0530 Dhaval Giani <[EMAIL PROTECTED]> wrote: > > > In the meantime I will go and check if it was there in 2.6.22-rc4-mm2 > It is hanging with 2.6.22-rc4-mm2 as well as on the latest git on kernel.org (2.6.22-git10). I will get back to you with more information as soon as I have it. Thanks -- regards, Dhaval I would like to change the world but they don't give me the source code! - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: System hangs on running kernbench
On Wed, Jul 18, 2007 at 01:07:00AM -0700, Andrew Morton wrote: > On Wed, 18 Jul 2007 13:26:48 +0530 Dhaval Giani <[EMAIL PROTECTED]> wrote: > > > I was running kernbench on top of 2.6.22-rc6-mm1 and I got a Hangcheck > > alert (This is when kernbench reached make -j). > > hm, never had a report of that before. It's the first time I've seen > hangcheck produce anything useful, frankly. > > Was the softlockup detector not enabled? I notice CONFIG_DETECT_SOFTLOCKUP=y > > > Also make -j is hanging. > > Please try to capture the full sysrq-T output when it is hung. Available at http://dhaval.giani.googlepages.com/sysrq-t-trace.bz2 In the meantime I will go and check if it was there in 2.6.22-rc4-mm2 -- regards, Dhaval I would like to change the world but they don't give me the source code! - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: System hangs on running kernbench
On Wed, 18 Jul 2007 13:26:48 +0530 Dhaval Giani <[EMAIL PROTECTED]> wrote: > I was running kernbench on top of 2.6.22-rc6-mm1 and I got a Hangcheck > alert (This is when kernbench reached make -j). hm, never had a report of that before. It's the first time I've seen hangcheck produce anything useful, frankly. Was the softlockup detector not enabled? > Also make -j is hanging. Please try to capture the full sysrq-T output when it is hung. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
System hangs on running kernbench
Hi Andrew, I was running kernbench on top of 2.6.22-rc6-mm1 and I got a Hangcheck alert (This is when kernbench reached make -j). Also make -j is hanging. (Also checked, it doesn't hang on 2.6.22) This is my /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) MP CPU 2.50GHz stepping: 5 cpu MHz : 2500.000 cache size : 1024 KB physical id : 0 siblings: 2 core id : 0 cpu cores : 1 fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts sync_rdtsc cid xtpr bogomips: 4982.85 clflush size: 64 processor : 1 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) MP CPU 2.50GHz stepping: 5 cpu MHz : 2500.000 cache size : 1024 KB physical id : 0 siblings: 2 core id : 0 cpu cores : 1 fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts sync_rdtsc cid xtpr bogomips: 4976.29 clflush size: 64 processor : 2 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) MP CPU 2.50GHz stepping: 5 cpu MHz : 2500.000 cache size : 1024 KB physical id : 1 siblings: 2 core id : 0 cpu cores : 1 fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts sync_rdtsc cid xtpr bogomips: 4976.25 clflush size: 64 processor : 3 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) MP CPU 2.50GHz stepping: 5 cpu MHz : 2500.000 cache size : 1024 KB physical id : 1 siblings: 2 core id : 0 cpu cores : 1 fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts sync_rdtsc cid xtpr bogomips: 4976.00 clflush size: 64 processor : 4 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) MP CPU 2.50GHz stepping: 5 cpu MHz : 2500.000 cache size : 1024 KB physical id : 2 siblings: 2 core id : 0 cpu cores : 1 fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts sync_rdtsc cid xtpr bogomips: 4976.19 clflush size: 64 processor : 5 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) MP CPU 2.50GHz stepping: 5 cpu MHz : 2500.000 cache size : 1024 KB physical id : 2 siblings: 2 core id : 0 cpu cores : 1 fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts sync_rdtsc cid xtpr bogomips: 4976.50 clflush size: 64 processor : 6 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) MP CPU 2.50GHz stepping: 5 cpu MHz : 2500.000 cache size : 1024 KB physical id : 3 siblings: 2 core id : 0 cpu cores : 1 fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts sync_rdtsc cid xtpr bogomips: 4976.37 clflush size
