LTP Results for 2.6.x and 2.4.x
Here's an updated summary of our LTP regression test runs against the 2.6.x and 2.4.x kernels on RedHat 9.0: http://developer.osdl.org/bryce/ltp/ Briefly, here are the numbers for the most recent kernels: Patch Name TestReq# LTP Ver CPUPASS FAIL WARN BROK --- linux-2.6.10 299759 20041105 2-way 2196 6 2 6 patch-2.6.10-rc3 299166 20041007 2-way 2199 6 2 6 patch-2.6.10-rc2 298746 20041007 2-way 2198 8 2 6 patch-2.6.10-rc1 298400 20041007 2-way 2198 6 2 6 Patch Name TestReq# LTP CPUPASS FAIL WARN BROK --- patch-2.4.29-rc3 300054 20041105 2-way 2210 3 2 3 patch-2.4.29-rc1 299873 20041105 2-way 2210 3 2 3 patch-2.4.29-pre2 299601 20041105 2-way 2210 3 2 3 patch-2.4.29-pre1 298976 20041007 2-way 2210 3 2 3 linux-2.4.28 298851 20041007 2-way 2210 3 2 3 A summary and a detailed report of the current failures on 2.6.10 is available at: http://khack.osdl.org/299759/results/FAIL_summary.txt http://developer.osdl.org/bryce/ltp/failrpt_299759_2.6.10.txt I've run into some issues with patch-2.6.11-rc1 and the latest LTP, but will post numbers when I've sorted those out. Bryce On Fri, 1 Oct 2004, Linus Torvalds wrote: > On Fri, 1 Oct 2004, Bryce Harrington wrote: > > > > madvise02 7 FAIL : madvise failed with wrong errno, expected > > errno = 22, got errno = 12 : Cannot allocate > > memory > >See: ltp/testcases/kernel/syscalls/madvise/madvise02.c > > Are you running this test on a 64-bit kernel with a 32-bit test > environment? This failure _looks_ that way, apparently because the > compatibility layer doesn't sign-extend "len". And quite frankly, > sign-extending it would be silly, although I think it would make the test > happy. > > Linus > - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Kernel Panic with LTP on 2.6.11-rc1 (was Re: LTP Results for 2.6.x and 2.4.x)
On Tue, 18 Jan 2005, Bryce Harrington wrote: > Here's an updated summary of our LTP regression test runs against the > 2.6.x and 2.4.x kernels on RedHat 9.0: > > http://developer.osdl.org/bryce/ltp/ > > I've run into some issues with patch-2.6.11-rc1 and the latest LTP, but > will post numbers when I've sorted those out. Okay, it looks like there is a regression of Linux 2.6.11-rc1 on LTP. I've run a bunch of tests to narrow down and rule things out: * Other tests (e.g. open_posix) run ok on this kernel * This version of LTP is working fine on other kernels * Not a hardware problem; same issue occurs on all of our 2-ways * Not distro-dependent; problem occurs for me on RH 9.0, and SuSE 9.0 and 9.2 Here is the output from the test causing the panic: <<>> tag=gf13 stime=110659 cmdline="mkfifo gffifo18; growfiles -b -W gf13 -e 1 -u -i 0 -L 30 -I r -r 1-4096 gffifo18" contacts="" analysis=exit initiation_status="ok" <<>> growfiles(gf13): 17094 DEBUG1 Using random seed of 1106350453 Kernel panic - not syncing: Out of memory and no killable processes... The full output are available at these links: FAIL LTP 2.6.11-rc1 SuSE 9.0 2-way http://khack.osdl.org/stp/300213/ FAIL LTP 2.6.11-rc1 SuSE 9.2 2-way http://khack.osdl.org/stp/300219/ FAIL LTP 2.6.11-rc1 SuSE 9.2 1-way http://khack.osdl.org/stp/300209/ OK LTP 2.6.10 SuSE 9.2 2-way http://khack.osdl.org/stp/300230 OK LTP 2.6.10 SuSE 9.0 2-way http://khack.osdl.org/stp/300229 OK LTP 2.6.10 RH 9.02-way http://khack.osdl.org/stp/300228 OK OPTS 2.6.11-rc1 RH 9.22-way http://khack.osdl.org/stp/300227 These are all run on the December version of LTP (20041203). growfiles.c is in ltp-full-20041203/testcases/kernel/fs/doio. Bryce - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Dev] Re: Kernel Panic with LTP on 2.6.11-rc1 (was Re: LTP Results for 2.6.x and 2.4.x)
On Fri, 21 Jan 2005, Chris Wright wrote: > * Andrew Morton ([EMAIL PROTECTED]) wrote: > > Bryce Harrington <[EMAIL PROTECTED]> wrote: > > I am unable to find the oops trace amongst all that stuff. Help? > > > > (It would have been handy to include it in the bug report, actually) > > Yes, it would. Or at least some better granularity leading up to the > problem. I ran growfiles locally on 2.6.11-rc-bk and didn't have any > problem. Could you strace growfiles and see what it was doing when it > killed the machine? Okay, I'll set up another run and try collecting that info. Is there any other data that would be useful to collect while I'm at it? Bryce - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [LTP] Re: [Dev] Re: Kernel Panic with LTP on 2.6.11-rc1 (was Re: LTP Results for 2.6.x and 2.4.x)
On Fri, 21 Jan 2005, Bryce Harrington wrote: > On Fri, 21 Jan 2005, Chris Wright wrote: > > * Andrew Morton ([EMAIL PROTECTED]) wrote: > > > Bryce Harrington <[EMAIL PROTECTED]> wrote: > > > I am unable to find the oops trace amongst all that stuff. Help? > > > > > > (It would have been handy to include it in the bug report, actually) > > > > Yes, it would. Or at least some better granularity leading up to the > > problem. I ran growfiles locally on 2.6.11-rc-bk and didn't have any > > problem. Could you strace growfiles and see what it was doing when it > > killed the machine? > > Okay, I'll set up another run and try collecting that info. Is there > any other data that would be useful to collect while I'm at it? Well, I'm not having much luck. strace isn't installed on the system (and is giving errors when trying to compile it). Also, the ssh session (and sshd) quits whenever I try running the following growfiles command manually, so I'm having trouble replicating the kernel panic manually. # growfiles -W gf14 -b -e 1 -u -i 0 -L 20 -w -l -C 1 -T 10 glseek19 glseek19.2 Anyway, if anyone wants to investigate this further, I can provide access to the machine (email me). Otherwise, I'm probably just going to wait for -rc2 and see if the problem's still there. Thanks, Bryce - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Kernel 2.6.11-rc1/2 goes Postal on LTP
On Fri, 21 Jan 2005, Chris Wright wrote: > * Bryce Harrington ([EMAIL PROTECTED]) wrote: > > Well, I'm not having much luck. strace isn't installed on the system > > (and is giving errors when trying to compile it). Also, the ssh session > > (and sshd) quits whenever I try running the following growfiles command > > manually, so I'm having trouble replicating the kernel panic manually. > > Sounds very much like oom killer gone nuts. > > > # growfiles -W gf14 -b -e 1 -u -i 0 -L 20 -w -l -C 1 -T 10 glseek19 > > glseek19.2 > > > > Anyway, if anyone wants to investigate this further, I can provide > > access to the machine (email me). Otherwise, I'm probably just going to > > wait for -rc2 and see if the problem's still there. > > Wait no longer, it's here ;-) Hmm, still the kernel is going nutso. LTP and everything else on the system is getting killed, including the test manager process. Below's a bit more info scraped from the console. This is from a run on RH 9.0. It looks like LTP got through 722 of its 2270 tests before the kernel goes postal on it. This time it got through all the growfile commands before it died (see bottom of this post), however it looks like the same growfile cmd I reported earlier is the one causing the problem. When I run: mkfifo gffifo18; strace growfiles -b -W gf13 -e 1 -u -i 0 -L 30 -I r -r 1-4096 gffifo18 &> /tmp/growfiles_strace_log.txt The following happens: * I get this strace log: http://developer.osdl.org/bryce/growfiles_strace_log.txt * The ssh session dies and returns to login prompt * A bunch of stuff similar to below is spewed to console Bryce Console === ...snip... Memory: 905212k/917504k available (2211k kernel code, 11840k reserved, 871k data, 192k init, 0k highmem) ...snip... HighMem per-cpu: empty Free pages:3828kB (0kB HighMem) Active:341 inactive:297 dirty:0 writeback:1 unstable:0 free:957 slab:1591 mapped:614 pagetables:172 DMA free:68kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes protections[]: 0 0 0 Normal free:3760kB min:3756kB low:4692kB high:5632kB active:1364kB inactive:1188kB present:901120kB pages_scanned:1571 all_unreclaimable? no protections[]: 0 0 0 HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB Normal: 6*4kB 5*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3760kB HighMem: empty Swap cache: add 4541, delete 4457, find 33/60, race 0+0 Out of Memory: Killed process 2103 (jserver). oom-killer: gfp_mask=0x1d2 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 Normal per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 HighMem per-cpu: empty Free pages:3764kB (0kB HighMem) Active:360 inactive:267 dirty:0 writeback:0 unstable:0 free:941 slab:1520 mapped:614 pagetables:167 DMA free:68kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes protections[]: 0 0 0 Normal free:3696kB min:3756kB low:4692kB high:5632kB active:1440kB inactive:1068kB present:901120kB pages_scanned:3461 all_unreclaimable? yes protections[]: 0 0 0 HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB Normal: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB HighMem: empty Swap cache: add 4994, delete 4918, find 106/200, race 0+0 Out of Memory: Killed process 2052 (ntpd). oom-killer: gfp_mask=0xd2 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 Normal per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 HighMem per-cpu: empty Free pages:3764kB (0kB HighMem) Active:0 inactive:13 dirty:0 writeback:0 unstable:0 free:941 slab:1482 mapped:0 pagetables:162 DMA free:68kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes protections[]: 0 0 0 Normal free:3696kB min:3756kB low:4692kB high:5632kB active:0kB inactive:52kB present:901120kB pages_scanned:134 all_unreclaimable? yes protections[]: 0 0 0 HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unrecla
Re: [torvalds@osdl.org: Re: Memory leak in 2.6.11-rc1?]
Hi Chris,
I applied the patch and reran the test on a RH 9.0 system. LTP is
continuing past where it failed before, and processes are not getting
killed, so I assume the OOM killer is no longer getting activated.
There is a new behavior, though. Now the test is hanging indefinitely
on the nptl01 test. I am assuming that since it passed the spot that it
had failed before, that this is an unrelated issue.
Bryce
On Mon, 24 Jan 2005, Chris Wright wrote:
> Any chance you could re-try with this patch applied?
>
> - Forwarded message from Linus Torvalds <[EMAIL PROTECTED]> -
>
> Date: Mon, 24 Jan 2005 14:35:47 -0800 (PST)
> From: Linus Torvalds <[EMAIL PROTECTED]>
> To: Andrew Morton <[EMAIL PROTECTED]>
> cc: Jens Axboe <[EMAIL PROTECTED]>, [EMAIL PROTECTED], [EMAIL PROTECTED],
>linux-kernel@vger.kernel.org, [EMAIL PROTECTED]
> Subject: Re: Memory leak in 2.6.11-rc1?
>
>
>
> On Mon, 24 Jan 2005, Andrew Morton wrote:
> >
> > Would indicate that the new pipe code is leaking.
>
> Duh. It's the pipe merging.
>
> Linus
>
>
> --- 1.40/fs/pipe.c2005-01-15 12:01:16 -08:00
> +++ edited/fs/pipe.c 2005-01-24 14:35:09 -08:00
> @@ -630,13 +630,13 @@
> struct pipe_inode_info *info = inode->i_pipe;
>
> inode->i_pipe = NULL;
> - if (info->tmp_page)
> - __free_page(info->tmp_page);
> for (i = 0; i < PIPE_BUFFERS; i++) {
> struct pipe_buffer *buf = info->bufs + i;
> if (buf->ops)
> buf->ops->release(info, buf);
> }
> + if (info->tmp_page)
> + __free_page(info->tmp_page);
> kfree(info);
> }
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [EMAIL PROTECTED]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
> - End forwarded message -
>
>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [torvalds@osdl.org: Re: Memory leak in 2.6.11-rc1?]
On Mon, 24 Jan 2005, Andrew Morton wrote: > Bryce Harrington <[EMAIL PROTECTED]> wrote: > > > > There is a new behavior, though. Now the test is hanging indefinitely > > on the nptl01 test. I am assuming that since it passed the spot that it > > had failed before, that this is an unrelated issue. > > I'm finding that nptl01 somtimes gets stuck and sometimes works OK. Try > running it by hand a few times. > > It could be an application bug or a kernel bug. Okay, so it doesn't sound like anything to be worried about. Thanks, Bryce - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
