Re: crashme fault

2007-09-17 Thread Randy Dunlap
On Mon, 17 Sep 2007 07:53:50 -0700 (PDT) Linus Torvalds wrote:

> On Mon, 17 Sep 2007, Randy Dunlap wrote:
> > 
> > OK, I haven't done the microcode update yet.  I ran crashme overnight
> > with your newer patch and it crashed:
> 
> Well, duh.
> 
> That's because I forgot to do the "error_code & PF_USER" => 
> "user_mode_vm(regs)" thing in the most common case - the 
> "bad_area_nosemaphore" if-statement.
> 
> But thinking more about it, it's actually just easier and more 
> straightforward to just take the same approach that my original hack did, 
> namely to just set PF_USER if the register state implies it was in user  
> mode.
> 
> So ignore that patch. You're better off with my original one that also 
> gave debugging info, and if we decide that this really looks like a CPU 
> buglet, the patch to actually commit would be a simplified version of that 
> (appended, just FYI).

[switched email address due to some unknown problems]


Reverted that patch.  Loaded new microcode update file.
Ran test for 3 hours without a fault.

It all seems rather inconclusive to me, still, the new patch (below)
makes sense.

> Also, I changed my mind on the microcode update - if you have a newer 
> microcode, go ahead and try it, because if that fixes the issue, then we 
> can really just close this as a "unimportant CPU bug" with the patch 
> below.
> 
>   Linus
> 
> ---
>  arch/x86_64/mm/fault.c |7 +++
>  1 files changed, 7 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86_64/mm/fault.c b/arch/x86_64/mm/fault.c
> index 327c9f2..54816ad 100644
> --- a/arch/x86_64/mm/fault.c
> +++ b/arch/x86_64/mm/fault.c
> @@ -374,6 +374,13 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs 
> *regs,
>   if (unlikely(in_atomic() || !mm))
>   goto bad_area_nosemaphore;
>  
> + /*
> +  * User-mode registers count as a user access even for any
> +  * potential system fault or CPU buglet.
> +  */
> + if (user_mode_vm(regs))
> + error_code |= PF_USER;
> +
>   again:
>   /* When running in the kernel we expect faults to occur only to
>* addresses in user space.  All other faults represent errors in the
> -

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-17 Thread Linus Torvalds


On Mon, 17 Sep 2007, Randy Dunlap wrote:
> 
> OK, I haven't done the microcode update yet.  I ran crashme overnight
> with your newer patch and it crashed:

Well, duh.

That's because I forgot to do the "error_code & PF_USER" => 
"user_mode_vm(regs)" thing in the most common case - the 
"bad_area_nosemaphore" if-statement.

But thinking more about it, it's actually just easier and more 
straightforward to just take the same approach that my original hack did, 
namely to just set PF_USER if the register state implies it was in user  
mode.

So ignore that patch. You're better off with my original one that also 
gave debugging info, and if we decide that this really looks like a CPU 
buglet, the patch to actually commit would be a simplified version of that 
(appended, just FYI).

Also, I changed my mind on the microcode update - if you have a newer 
microcode, go ahead and try it, because if that fixes the issue, then we 
can really just close this as a "unimportant CPU bug" with the patch 
below.

Linus

---
 arch/x86_64/mm/fault.c |7 +++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/arch/x86_64/mm/fault.c b/arch/x86_64/mm/fault.c
index 327c9f2..54816ad 100644
--- a/arch/x86_64/mm/fault.c
+++ b/arch/x86_64/mm/fault.c
@@ -374,6 +374,13 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs 
*regs,
if (unlikely(in_atomic() || !mm))
goto bad_area_nosemaphore;
 
+   /*
+* User-mode registers count as a user access even for any
+* potential system fault or CPU buglet.
+*/
+   if (user_mode_vm(regs))
+   error_code |= PF_USER;
+
  again:
/* When running in the kernel we expect faults to occur only to
 * addresses in user space.  All other faults represent errors in the
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-17 Thread Randy Dunlap

Linus Torvalds wrote:


On Sun, 16 Sep 2007, Randy Dunlap wrote:

I'll test this overnight on 2.6.23-rc6-git2 since that was failing.

I haven't been able to reproduce the fault on 2.6.21 after several
hours of testing.

I'll also test a microcode update to see if it helps.


Before you do the microcode update, try to see if you can bisect the place 
between 2.6.21->22 that seems to start it. Even if you don't get all the 
way, if you are confident enough about the "no error" case to be able to 
bisect it down by doing a few reboots, it will at least cut down the set 
of possible commits by roughly a factor of 2^ events, so 
even "just" a series of 4-5 bisect things might give us more of a clue.


OK, I haven't done the microcode update yet.  I ran crashme overnight
with your newer patch and it crashed:

[14254.327676] Unable to handle kernel paging request at ff021eaf RIP: 
[14254.332299]  [<00504225>]
[14254.338084] PGD d8542067 PUD 0 
[14254.341271] Oops:  [1] SMP 
[14254.39] CPU 3 
[14254.346484] Modules linked in: loop

[14254.350001] Pid: 28565, comm: crashme Not tainted 2.6.23-rc6-git2 #2
[14254.356349] RIP: 0033:[<00504225>]  [<00504225>]
[14254.362376] RSP: 002b:7fff5afccbf8  EFLAGS: 00010656
[14254.367685] RAX: 5afccbf8 RBX: 2abd4fbf5c00 RCX: 2abd4fc88b37
[14254.374812] RDX: 00504220 RSI:  RDI: 000a
[14254.381939] RBP: 7fff5afccc00 R08: 7fff5afccb50 R09: 
[14254.389068] R10:  R11: 0202 R12: 
[14254.396195] R13: 7fff5afccdf0 R14:  R15: 
[14254.403324] FS:  2abd4fe276d0() GS:81011fc751c0() 
knlGS:
[14254.411403] CS:  0010 DS:  ES:  CR0: 8005003b
[14254.417144] CR2: ff021eaf CR3: d85f6000 CR4: 06e0
[14254.424273] DR0:  DR1:  DR2: 
[14254.431400] DR3:  DR6: 4ff0 DR7: 0400
[14254.438528] Process crashme (pid: 28565, threadinfo 8100d897, task 
8100d8628820)
[14254.446953] 
[14254.448443] RIP  [<00504225>]

[14254.452124]  RSP <7fff5afccbf8>
[14254.455614] CR2: ff021eaf
[14254.459244] Kernel panic - not syncing: Fatal exception


Of course, if it's somewhat random and timing-dependent, bisection can be 
hard (the "2^n" thing is very efficient, but it also means that a *single* 
wrong answer will totally invalidate the result, so if something isn't 
entirely reproducible, bisection often fails!)


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Linus Torvalds


On Sun, 16 Sep 2007, Randy Dunlap wrote:
> 
> I'll test this overnight on 2.6.23-rc6-git2 since that was failing.
> 
> I haven't been able to reproduce the fault on 2.6.21 after several
> hours of testing.
> 
> I'll also test a microcode update to see if it helps.

Before you do the microcode update, try to see if you can bisect the place 
between 2.6.21->22 that seems to start it. Even if you don't get all the 
way, if you are confident enough about the "no error" case to be able to 
bisect it down by doing a few reboots, it will at least cut down the set 
of possible commits by roughly a factor of 2^ events, so 
even "just" a series of 4-5 bisect things might give us more of a clue.

Of course, if it's somewhat random and timing-dependent, bisection can be 
hard (the "2^n" thing is very efficient, but it also means that a *single* 
wrong answer will totally invalidate the result, so if something isn't 
entirely reproducible, bisection often fails!)

Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Randy Dunlap
On Sun, 16 Sep 2007 11:12:23 -0700 (PDT) Linus Torvalds wrote:

> 
> 
> On Sun, 16 Sep 2007, Linus Torvalds wrote:
> > 
> > I'm really starting to suspect some early EM64T bug, and I also suspect 
> > that it's harmless but that we should just do the trivial patch to say "if 
> > the register state is in user mode, we don't care if the CPU says it was a 
> > kernel access".
> 
> Namely something like this..
> 
> The basic idea is that it's pointless to test for "error_code & PF_USER" 
> to decide whether we should oops or kill the process: the *real* issue is 
> whether we *can* kill the process or not. And that depends not on whether 
> the CPU claimed it was a user access or not, but on whether the register 
> state we'd return to is user-mode or not!
> 
> So anything that decides whether it should send a signal or do to the 
> "no_context" Ooops path should use "user_mode_vm(regs)" (yeah, I realize 
> that the "_vm" part is unnecessary on x86-64, but it doesn't hurt either, 
> and all of the issues are the same on 32/64-bit) which tests the right 
> thing.
> 
> Now, normally the USER bit in the error code should be the exact same 
> thing, except for
> 
>  - Some CPU bug (microcode issue, whatever) where some complex fault 
>situation sets the wrong error code.
> 
>  - user space accesses that caused a system page fault (ie a page fault 
>while handling another trap - possibly due to lazy page table setup 
>and having the LDT or some other CPU data structure in vmalloc space)
> 
> Now, the vmalloc space accesses should be handled separately anyway, so I 
> really wonder if it's some subtle CPU bug (I can't reproduce any problems 
> on my Core 2 Duo), but the point is that I think this patch really is 
> conceptually a real fix regardless, even if it _shouldn't_ matter.
> 
> Comments?
> 
> Randy, this replaces the hacky patch I sent, but also shuts up about the 
> odd thing you're hitting, so for testing your case further this may not be 
> the right thing. However, it would be nice to hear whether this just makes 
> "crashme" work properly for you without any side effects..

I'll test this overnight on 2.6.23-rc6-git2 since that was failing.

I haven't been able to reproduce the fault on 2.6.21 after several
hours of testing.

I'll also test a microcode update to see if it helps.

---
~Randy
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Andi Kleen
On Sun, Sep 16, 2007 at 10:14:46AM -0700, Linus Torvalds wrote:
> 
> 
> On Sun, 16 Sep 2007, Randy Dunlap wrote:
> > 
> > I'll apply this patch today, but I haven't done so yet (for the 2
> > bug reports below).
> 
> Actually, it's probably better that you don't change your situation 
> unnecessarily, in case the bug goes away.

I checked the Prescott Specification Update, but didn't seen anything
that could cause such strange faults.

I'm trying to reproduce it here.

-Andi

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Linus Torvalds


On Sun, 16 Sep 2007, Linus Torvalds wrote:
> 
> I'm really starting to suspect some early EM64T bug, and I also suspect 
> that it's harmless but that we should just do the trivial patch to say "if 
> the register state is in user mode, we don't care if the CPU says it was a 
> kernel access".

Namely something like this..

The basic idea is that it's pointless to test for "error_code & PF_USER" 
to decide whether we should oops or kill the process: the *real* issue is 
whether we *can* kill the process or not. And that depends not on whether 
the CPU claimed it was a user access or not, but on whether the register 
state we'd return to is user-mode or not!

So anything that decides whether it should send a signal or do to the 
"no_context" Ooops path should use "user_mode_vm(regs)" (yeah, I realize 
that the "_vm" part is unnecessary on x86-64, but it doesn't hurt either, 
and all of the issues are the same on 32/64-bit) which tests the right 
thing.

Now, normally the USER bit in the error code should be the exact same 
thing, except for

 - Some CPU bug (microcode issue, whatever) where some complex fault 
   situation sets the wrong error code.

 - user space accesses that caused a system page fault (ie a page fault 
   while handling another trap - possibly due to lazy page table setup 
   and having the LDT or some other CPU data structure in vmalloc space)

Now, the vmalloc space accesses should be handled separately anyway, so I 
really wonder if it's some subtle CPU bug (I can't reproduce any problems 
on my Core 2 Duo), but the point is that I think this patch really is 
conceptually a real fix regardless, even if it _shouldn't_ matter.

Comments?

Randy, this replaces the hacky patch I sent, but also shuts up about the 
odd thing you're hitting, so for testing your case further this may not be 
the right thing. However, it would be nice to hear whether this just makes 
"crashme" work properly for you without any side effects..

Linus


 arch/x86_64/mm/fault.c |9 -
 1 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/arch/x86_64/mm/fault.c b/arch/x86_64/mm/fault.c
index 327c9f2..75270a0 100644
--- a/arch/x86_64/mm/fault.c
+++ b/arch/x86_64/mm/fault.c
@@ -87,7 +87,7 @@ static noinline int is_prefetch(struct pt_regs *regs, 
unsigned long addr,
instr = (unsigned char __user *)convert_rip_to_linear(current, regs);
max_instr = instr + 15;
 
-   if (user_mode(regs) && instr >= (unsigned char *)TASK_SIZE)
+   if (user_mode_vm(regs) && instr >= (unsigned char *)TASK_SIZE)
return 0;
 
while (scan_more && instr < max_instr) { 
@@ -320,7 +320,6 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs 
*regs,
 
info.si_code = SEGV_MAPERR;
 
-
/*
 * We fault-in kernel-space virtual memory on-demand. The
 * 'reference' page table is init_mm.pgd.
@@ -404,7 +403,7 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs 
*regs,
goto good_area;
if (!(vma->vm_flags & VM_GROWSDOWN))
goto bad_area;
-   if (error_code & 4) {
+   if (error_code & PF_USER) {
/* Allow userspace just enough access below the stack pointer
 * to let the 'enter' instruction work.
 */
@@ -558,7 +557,7 @@ out_of_memory:
goto again;
}
printk("VM: killing process %s\n", tsk->comm);
-   if (error_code & 4)
+   if (user_mode_vm(regs))
do_group_exit(SIGKILL);
goto no_context;
 
@@ -566,7 +565,7 @@ do_sigbus:
up_read(&mm->mmap_sem);
 
/* Kernel mode? Handle exceptions or die */
-   if (!(error_code & PF_USER))
+   if (!user_mode_vm(regs))
goto no_context;
 
tsk->thread.cr2 = address;
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Linus Torvalds


On Sun, 16 Sep 2007, Randy Dunlap wrote:
> 
> I'll apply this patch today, but I haven't done so yet (for the 2
> bug reports below).

Actually, it's probably better that you don't change your situation 
unnecessarily, in case the bug goes away.

Since you are triggering the problem even *without* the patch, whatever 
crashme is doing now for you is the interesting case.

I suspect NX just isn't active for you, so the mprotect doesn't matter, 
but if it *does* matter, that's interesting in itself.

> Then I'll back up more, prior to 2.6.22 (which got the Huh? message and 
> kept going).

Ahh, ok, so this is not a new thing, and it happened in 2.6.22 too.

I'm really starting to suspect some early EM64T bug, and I also suspect 
that it's harmless but that we should just do the trivial patch to say "if 
the register state is in user mode, we don't care if the CPU says it was a 
kernel access".

Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Randy Dunlap
On Sat, 15 Sep 2007 17:34:54 -0700 (PDT) Linus Torvalds wrote:

> 
> 
> On Sat, 15 Sep 2007, Randy Dunlap wrote:
> > Command: ./crashme +2000 666 1000 1:00:00 1
> 
> Ok, that's close to what I was testing (one of the examples from the 
> crashme docs).
> 
> > > The original gjc crashme doesn't even do a "mprotect(PROT_EXEC)" by 
> > > default
> > > (nor does it even compile on a modern unix), so it's not going to do
> > > anything. I hacked it up, and it appears to work ok for me, but I'm not at
> > > all confident that I'm even close to recreating what you are doing.
> > 
> > So I'm missing some mprotect() call also?
> > Oops.
> 
> Yeah, by default, it won't do a mprotect(), so as far as I can tell, 
> you're always going to hit NX on a writable data page. I'm running with 
> the appended patch.

I'll apply this patch today, but I haven't done so yet (for the 2
bug reports below).  Then I'll back up more, prior to 2.6.22 (which
got the Huh? message and kept going).

> That said, I didn't actually *check* that the mprotect makes any 
> difference, and maybe you have one of the early EM64T chips that didn't 
> have NX support at all. Or maybe it's dodgy, and is actually the whole 
> *reason* for the page fault problem..

It's a bit old.  I wouldn't be surprised if it doesn't support NX.

> I don't have any P4's capable of 64-bit operation (nor do I really want 
> any ;)

ack that.


~~~ 2.6.23-rc3 ~
[21769.161490] Unable to handle kernel paging request at ff016abe RIP:
[21769.166119]  [<0050522f>]
[21769.171912] PGD d5572067 PUD 0
[21769.175100] Oops:  [1] SMP
[21769.178278] CPU 3
[21769.180312] Modules linked in: loop
[21769.183830] Pid: 8823, comm: crashme Not tainted 2.6.23-rc3 #1
[21769.189657] RIP: 0033:[<0050522f>]  [<0050522f>]
[21769.195684] RSP: 002b:7fffe9528118  EFLAGS: 00010202
[21769.200993] RAX: c169bc00 RBX:  RCX: 2b88c172eb37
[21769.208121] RDX: 005051e0 RSI:  RDI: 000a
[21769.215249] RBP: 7fffe9528120 R08: 7fffe9528070 R09: 
[21769.222377] R10: 0008 R11: 0216 R12: 
[21769.229505] R13: 7fffe9528310 R14:  R15: 
[21769.236632] FS:  2b88c18cd6d0() GS:81011fc751c0() 
knlGS:
[21769.244711] CS:  0010 DS:  ES:  CR0: 8005003b
[21769.250453] CR2: ff016abe CR3: d792f000 CR4: 06e0
[21769.257581] DR0:  DR1:  DR2: 
[21769.264709] DR3:  DR6: 4ff0 DR7: 0400
[21769.271837] Process crashme (pid: 8823, threadinfo 8100d5656000, task 
81011810)
[21769.280175]
[21769.281673] RIP  [<0050522f>]
[21769.285355]  RSP <7fffe9528118>
[21769.288845] CR2: ff016abe
[21769.292455] Kernel panic - not syncing: Fatal exception
[21769.297699] Rebooting in 30 seconds..

~~~ 2.6.22 
Sep 15 17:15:03 caor-test8 kernel: [18447.172949] kernel mode page fault from 
user space? Huh?  
Sep 15 17:15:03 caor-test8 kernel: [18447.178268] 
Sep 15 17:15:03 caor-test8 kernel: [18447.179773] Modules linked in: loop
Sep 15 17:15:03 caor-test8 kernel: [18447.183306] Pid: 30819, comm: crashme Not 
tainted 2.6.22 #2 
Sep 15 17:15:03 caor-test8 kernel: [18447.183311] RIP: 
0033:[<00507163>]  [<00507163>]
Sep 15 17:15:03 caor-test8 kernel: [18447.183318] RSP: 002b:7fff10da99b0  
EFLAGS: 00010202  
Sep 15 17:15:03 caor-test8 kernel: [18447.183320] RAX:  RBX: 
2b4b99e18c00 RCX: 2b4b99eabb37
Sep 15 17:15:03 caor-test8 kernel: [18447.183323] RDX: 00507160 RSI: 
00401476 RDI: 000a
Sep 15 17:15:03 caor-test8 kernel: [18447.183326] RBP: 7fff10da99b0 R08: 
7fff10da9900 R09: 
Sep 15 17:15:03 caor-test8 kernel: [18447.183328] R10: 0008 R11: 
0212 R12: 
Sep 15 17:15:03 caor-test8 kernel: [18447.183330] R13: 7fff10da9ba0 R14: 
 R15: 
Sep 15 17:15:03 caor-test8 kernel: [18447.18] FS:  2b4b9a04a6d0() 
GS:80721000() knlGS:
Sep 15 17:15:03 caor-test8 kernel: [18447.183336] CS:  0010 DS:  ES:  
CR0: 8005003b
Sep 15 17:15:03 caor-test8 kernel: [18447.183339] CR2: ff019fd5 CR3: 
cacb CR4: 06e0
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Randy Dunlap
On Sun, 16 Sep 2007 17:53:21 +0200 Andrea Arcangeli wrote:

> On Wed, Sep 12, 2007 at 10:21:51PM -0700, Randy Dunlap wrote:
> > I run almost-daily kernel testing.  I haven't seen 'crashme' cause a
> > kernel fault until today, and now I've seen it twice on 2.6.23-rc6-git2,
> 
> Did the room temperature change in the server room? ;) Those early

I rarely go there, but when I do, it's usually very cool.
But I have no way of tracking the room temp there.

> EM64T P4 core based are fast but they suck a whole lot of power. It
> seems a bitflip in a cpu register. You can use cpufreq to set low
> frequency (they should support it, and they shouldn't use more than
> 100w per core that way) to lower the temp, and see if the problem goes
> away. If it's a software issue it will hopefully be still reproducible
> with ~2ghz instead of 3.4ghz.


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-16 Thread Andrea Arcangeli
On Wed, Sep 12, 2007 at 10:21:51PM -0700, Randy Dunlap wrote:
> I run almost-daily kernel testing.  I haven't seen 'crashme' cause a
> kernel fault until today, and now I've seen it twice on 2.6.23-rc6-git2,

Did the room temperature change in the server room? ;) Those early
EM64T P4 core based are fast but they suck a whole lot of power. It
seems a bitflip in a cpu register. You can use cpufreq to set low
frequency (they should support it, and they shouldn't use more than
100w per core that way) to lower the temp, and see if the problem goes
away. If it's a software issue it will hopefully be still reproducible
with ~2ghz instead of 3.4ghz.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Andi Kleen
On Sat, Sep 15, 2007 at 03:47:19PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 15 Sep 2007, Linus Torvalds wrote:
> > 
> > So regardless of whether we want to trust "user_mode(regs)" more than 
> > "error_code & PF_USER", it would definitely be very interesting if you can 
> > give a good "this is where it started happening".
> 
> Also, can you point to good crashme sources, and give the arguments you 

LTP (http://ltp.sourceforge.net) should have a modern crashme.

-Andi

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Linus Torvalds


On Sat, 15 Sep 2007, Randy Dunlap wrote:
> Command: ./crashme +2000 666 1000 1:00:00 1

Ok, that's close to what I was testing (one of the examples from the 
crashme docs).

> > The original gjc crashme doesn't even do a "mprotect(PROT_EXEC)" by default
> > (nor does it even compile on a modern unix), so it's not going to do
> > anything. I hacked it up, and it appears to work ok for me, but I'm not at
> > all confident that I'm even close to recreating what you are doing.
> 
> So I'm missing some mprotect() call also?
> Oops.

Yeah, by default, it won't do a mprotect(), so as far as I can tell, 
you're always going to hit NX on a writable data page. I'm running with 
the appended patch.

That said, I didn't actually *check* that the mprotect makes any 
difference, and maybe you have one of the early EM64T chips that didn't 
have NX support at all. Or maybe it's dodgy, and is actually the whole 
*reason* for the page fault problem..

I don't have any P4's capable of 64-bit operation (nor do I really want 
any ;)

Linus

---
diff --git a/crashme.c b/crashme.c
index 5e7e663..19a2fbb 100644
--- a/crashme.c
+++ b/crashme.c
@@ -1,3 +1,5 @@
+#define pyr
+#define PAGSIZ 4096
 /* crashme: Create a string of random bytes and then jump to it.
 crashme [+][.inc]   [nsub] [verboseness] */
 
@@ -143,7 +145,6 @@ a script.
 #include 
 #include 
 #include 
-#include 
 #define strchr index
 #endif
 
@@ -215,7 +216,7 @@ unsigned char *bad_malloc(n)
 {unsigned char *data;
  data = (unsigned char *) malloc(n);
 #ifdef pyr
- if (mprotect(((int)data/PAGSIZ)*PAGSIZ, (n/PAGSIZ+1)*PAGSIZ,
+ if (mprotect(((long)data/PAGSIZ)*PAGSIZ, (n/PAGSIZ+1)*PAGSIZ,
  PROT_READ|PROT_WRITE|PROT_EXEC))
perror("mprotect");
 #endif
@@ -259,11 +260,8 @@ void my_signal(sig, func)
  signal(sig, func);
 #else
  struct sigaction act;
+ memset(&act, 0, sizeof(act));
  act.sa_handler = func;
- act.sa_mask = 0;
-#ifdef linux
- act.sa_restorer = 0;
-#endif /* linux */
  act.sa_flags = SA_NOMASK;
 #ifdef SA_RESTART
  act.sa_flags |= SA_RESTART;
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Randy Dunlap

Linus Torvalds wrote:


On Sat, 15 Sep 2007, Linus Torvalds wrote:
So regardless of whether we want to trust "user_mode(regs)" more than 
"error_code & PF_USER", it would definitely be very interesting if you can 
give a good "this is where it started happening".


Also, can you point to good crashme sources, and give the arguments you 
used to run it when it crashed?


I use http://people.delphiforums.com/gjc/crashme.tgz from
http://people.delphiforums.com/gjc/crashme.html, plus one small patch:

--- gjc-crashme.orig/crashme.c
+++ gjc-crashme/crashme.c
@@ -260,7 +260,8 @@ void my_signal(sig, func)
#else
 struct sigaction act;
 act.sa_handler = func;
- act.sa_mask = 0;
+ //act.sa_mask = 0;
+ sigemptyset(&act.sa_mask);
#ifdef linux
 act.sa_restorer = 0;
#endif /* linux */


Command: ./crashme +2000 666 1000 1:00:00 1

The original gjc crashme doesn't even do a "mprotect(PROT_EXEC)" by 
default (nor does it even compile on a modern unix), so it's not going to 
do anything. I hacked it up, and it appears to work ok for me, but I'm not 
at all confident that I'm even close to recreating what you are doing.


So I'm missing some mprotect() call also?
Oops.

(It probably goes without saying that I've not reproduced the oops on my 
Core 2 Duo. Lots of #GP and #PF errors, but nothing interesting.


Linus


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Linus Torvalds


On Sat, 15 Sep 2007, Linus Torvalds wrote:
> 
> So regardless of whether we want to trust "user_mode(regs)" more than 
> "error_code & PF_USER", it would definitely be very interesting if you can 
> give a good "this is where it started happening".

Also, can you point to good crashme sources, and give the arguments you 
used to run it when it crashed?

The original gjc crashme doesn't even do a "mprotect(PROT_EXEC)" by 
default (nor does it even compile on a modern unix), so it's not going to 
do anything. I hacked it up, and it appears to work ok for me, but I'm not 
at all confident that I'm even close to recreating what you are doing.

(It probably goes without saying that I've not reproduced the oops on my 
Core 2 Duo. Lots of #GP and #PF errors, but nothing interesting.

Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Linus Torvalds


On Sat, 15 Sep 2007, Linus Torvalds wrote:
> 
> Here's a really *stupid* patch (and untested too, btw) to see if it gets 
> easier to debug when you don't oops, just print the register state 
> instead.

Side note - while thinking about this, I'm wondering whether maybe that 
"stupid" patch might not actually be the right thing to do.

The fact is, the "user access" bit in the error code is in many ways a 
totally *different* issue than the "user_mode(regs)" test. We may end up 
having a system page fault for some action that was initiated by user 
space: eg taking a page fault on a segment load. And it may well be marked 
as a "system access" in the error code, even though the instruction that 
triggered it was all user mode, and even though we could just terminate 
the program rather than oops.

That said, since the kernel isn't paging itself out, user mode generally 
should not be able to generate those kinds of system access page faults. 
Our segment tables should all be in core, and so I would still in 
*practice* always expect the register state and the error code to match in 
the "usermode'ness".

So regardless of whether we want to trust "user_mode(regs)" more than 
"error_code & PF_USER", it would definitely be very interesting if you can 
give a good "this is where it started happening".

Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Randy Dunlap

Linus Torvalds wrote:


On Sat, 15 Sep 2007, Randy Dunlap wrote:

Had another on recent last night (probably not helpful):


At least the original "crashme" would write its random number seeds to a 
logfile each time (and I made it fsync it in some versions), which meant 
that once a crash happened, you could re-produce it immediately (if it was 
reproducible at all, of course).


Does your crashme have something like that?


I tell it the "random" seed to use.  I can also sets its debug level,
but when I did that yesterday, it never faulted, so I lowered it again,
them boom.  Could be coincidence.


All your crashes look basically identical - I don't think there is 
anything new in this one, they're all the same issue. What CPU do you have 
- vendor, stepping, version etc - and has something else than the kernel 
changed in your setup lately?


Just kernel changes.  CPU is dual Pentium Xeon + HT:

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 15
model   : 3
model name  :   Intel(R) Xeon(TM) CPU 3.40GHz
stepping: 4
cpu MHz : 3400.227
cache size  : 1024 KB
physical id : 0
siblings: 2
core id : 0
cpu cores   : 1
fpu : yes
fpu_exception   : yes
cpuid level : 5
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor 
ds_cpl est t
m2 cid xtpr
bogomips: 6805.96
clflush size: 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 15
model   : 3
model name  :   Intel(R) Xeon(TM) CPU 3.40GHz
stepping: 4
cpu MHz : 3400.227
cache size  : 1024 KB
physical id : 3
siblings: 2
core id : 0
cpu cores   : 1
fpu : yes
fpu_exception   : yes
cpuid level : 5
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor 
ds_cpl est t
m2 cid xtpr
bogomips: 6800.28
clflush size: 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 15
model   : 3
model name  :   Intel(R) Xeon(TM) CPU 3.40GHz
stepping: 4
cpu MHz : 3400.227
cache size  : 1024 KB
physical id : 0
siblings: 2
core id : 0
cpu cores   : 1
fpu : yes
fpu_exception   : yes
cpuid level : 5
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor 
ds_cpl est t
m2 cid xtpr
bogomips: 6800.72
clflush size: 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 15
model   : 3
model name  :   Intel(R) Xeon(TM) CPU 3.40GHz
stepping: 4
cpu MHz : 3400.227
cache size  : 1024 KB
physical id : 3
siblings: 2
core id : 0
cpu cores   : 1
fpu : yes
fpu_exception   : yes
cpuid level : 5
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor 
ds_cpl est t
m2 cid xtpr
bogomips: 6800.57
clflush size: 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:



As mentioned, the crash does look like a user-level crash got reported as 
a kernel page fault, and while a CPU bug sounds incredibly unlikely, this 
does have the smell of something strange like a fault in the middle of an 
"iretq" or "sysretq", where part of the CPU state has already been 
restored - which would explain why rip/cs is user space - but some part of 
the CPU is still in kernel mode - which would explain the incorrect page 
fault error code.


Here's a really *stupid* patch (and untested too, btw) to see if it gets 
easier to debug when you don't oops, just print the register state 
instead.


Will add this patch.


(It might be interesting to also do something like

force_sig_specific(SIGSTOP, current);

to then be able to more easily attach to the process that had problems, 
and debug it in user space to see what was going on..)


Linus
---
diff --git a/arch/x86_64/mm/fault.c b/arch/x86_64/mm/fault.c
index 327c9f2..1b81392 100644
--- a/arch/x86_64/mm/fault.c
+++ b/arch/x86_64/mm/fault.c
@@ -320,6 +320,11 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs 
*regs,
 
 	info.si_code 

Re: crashme fault

2007-09-15 Thread Linus Torvalds


On Sat, 15 Sep 2007, Randy Dunlap wrote:
> 
> Had another on recent last night (probably not helpful):

At least the original "crashme" would write its random number seeds to a 
logfile each time (and I made it fsync it in some versions), which meant 
that once a crash happened, you could re-produce it immediately (if it was 
reproducible at all, of course).

Does your crashme have something like that?

All your crashes look basically identical - I don't think there is 
anything new in this one, they're all the same issue. What CPU do you have 
- vendor, stepping, version etc - and has something else than the kernel 
changed in your setup lately?

As mentioned, the crash does look like a user-level crash got reported as 
a kernel page fault, and while a CPU bug sounds incredibly unlikely, this 
does have the smell of something strange like a fault in the middle of an 
"iretq" or "sysretq", where part of the CPU state has already been 
restored - which would explain why rip/cs is user space - but some part of 
the CPU is still in kernel mode - which would explain the incorrect page 
fault error code.

Here's a really *stupid* patch (and untested too, btw) to see if it gets 
easier to debug when you don't oops, just print the register state 
instead.

(It might be interesting to also do something like

force_sig_specific(SIGSTOP, current);

to then be able to more easily attach to the process that had problems, 
and debug it in user space to see what was going on..)

Linus
---
diff --git a/arch/x86_64/mm/fault.c b/arch/x86_64/mm/fault.c
index 327c9f2..1b81392 100644
--- a/arch/x86_64/mm/fault.c
+++ b/arch/x86_64/mm/fault.c
@@ -320,6 +320,11 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs 
*regs,
 
info.si_code = SEGV_MAPERR;
 
+   if (!(error_code & PF_USER) && user_mode(regs)) {
+   printk("kernel mode page fault from user space? Huh?\n");
+   __show_regs(regs);
+   error_code |= PF_USER;
+   }
 
/*
 * We fault-in kernel-space virtual memory on-demand. The
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Randy Dunlap

Andi Kleen wrote:

Andi, anything comes to mind?


No, unfortunately not. There weren't any changes to entry.S recently
that could corrupt the error code as far as I remember. Also cannot think of 
something else.


A version where it started happening would be useful.


I'll begin testing older kernels then.


Had another on recent last night (probably not helpful):

[58169.607164] Unable to handle kernel paging request at ff018915 RIP:
[58169.611784]  [<00506986>]
[58169.617570] PGD d4380067 PUD 0
[58169.620757] Oops:  [1] SMP
[58169.623943] CPU 2
[58169.625970] Modules linked in: loop
[58169.629496] Pid: 26735, comm: crashme Not tainted 2.6.23-rc6-git4 #1
[58169.635842] RIP: 0033:[<00506986>]  [<00506986>]
[58169.641861] RSP: 002b:7fff14d3f9a0  EFLAGS: 00010247
[58169.647171] RAX: 00506909 RBX: 2b8695e82c00 RCX: 2b8695f15b37
[58169.654298] RDX:  RSI: 00401476 RDI: 000a
[58169.661426] RBP: 7fff14d3f9a0 R08: 7fff14d3f8f0 R09: 
[58169.668553] R10: 0008 R11: 0212 R12: 
[58169.675680] R13: 7fff14d3fb90 R14:  R15: 
[58169.682808] FS:  2b86960b46d0() GS:81011fc75840() 
knlGS:
[58169.690889] CS:  0010 DS:  ES:  CR0: 80050033
[58169.696630] CR2: ff018915 CR3: d882 CR4: 06e0
[58169.703757] DR0:  DR1:  DR2: 
[58169.710885] DR3:  DR6: 4ff0 DR7: 0400
[58169.718014] Process crashme (pid: 26735, threadinfo 8100cd176000, task 
81011fffe040)
[58169.726439]
[58169.727937] RIP  [<00506986>]
[58169.731609]  RSP <7fff14d3f9a0>
[58169.735100] CR2: ff018915
[58169.738710] Kernel panic - not syncing: Fatal exception
[58169.743953] Rebooting in 30 seconds..

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-15 Thread Andi Kleen
> Andi, anything comes to mind?

No, unfortunately not. There weren't any changes to entry.S recently
that could corrupt the error code as far as I remember. Also cannot think of 
something else.

A version where it started happening would be useful.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-14 Thread Randy Dunlap
On Fri, 14 Sep 2007 22:05:17 -0700 Randy Dunlap wrote:

> On Fri, 14 Sep 2007 21:28:12 -0700 (PDT) Linus Torvalds wrote:
> 
> > On Wed, 12 Sep 2007, Randy Dunlap wrote:
> > >
> > > I run almost-daily kernel testing.  I haven't seen 'crashme' cause a
> > > kernel fault until today, and now I've seen it twice on 2.6.23-rc6-git2,
> > > x86_64.  After the first fault, I ran 'crashme' about 10 more times
> > > to get the second fault (usually for 10 minutes, one time for 30
> > > minutes).
> > 
> > Interesting. If this is reproducible for you, can you try to narrow down 
> > (with bit-bisect) roughly when it started.
> 
> It doesn't happen quite on demand.  Only 2 times in several hours
> of testing of the last few days.
> 
> > > There is very little helpful info.  RIP is strange, e.g.: 0051b446
> > > The call stack is not printed.  No kernel symbols are printed,
> > > even though I have CONFIG_KALLSYMS{_ALL}=y.
> > 
> > It looks like it's a page fault as a result of a *user*space* access, and 
> > most likely your machine would continue happily, except you have 
> > "panic_on_oops" set, so when the oops happens, it shuts the system down.
> 
> I see.
> 
> > Now, the reason I say it looks like a user space access is that you have
> > 
> > RIP: 0033:[<00510eea>]
> > RSP: 002b:7fffc9a8ec10
> > 
> > which are all user space segments. So the register contents clearly say 
> > "page fault in user space".
> > 
> > However, what makes the kernel oops (rather than just send a SIGSEGV) is 
> > that the page fault "error code" is zero (that's the number that is 
> > printed out just after the "Oops" string). For a normal user space access, 
> > you should have bit #2 set in the error code.
> > 
> > So the kernel thinks it's a kernel page fault, because the page fault 
> > error code says so. But everything else seems to indicate that it's really 
> > user mode.. It would be very interesting to hear when this started 
> > happening.
> > 
> > Even if you cannot bisect it down all the way, since you say that you do 
> > almost daily kernel testing, is this really new to 2.6.23-rc6-git2, and 
> > 2.6.23-rc6-git1 was fine? 
> 
> I'll check back thru my test logs.

I looked thru several hundred test logs and didn't find any other
occurrences of it.  Doesn't prove much.
I'll continue to run the test most of this weekend.


> > Andi, anything comes to mind?
> > 
> > Linus
> > 
> > ---
> > > [ 7487.208128] Unable to handle kernel paging request at ff019b53 
> > > RIP:
> > > [ 7487.212752]  [<00510eea>]
> > > [ 7487.218537] PGD 10c1a2067 PUD 0
> > > [ 7487.221811] Oops:  [1] SMP
> > > [ 7487.224989] CPU 2
> > > [ 7487.227024] Modules linked in: loop
> > > [ 7487.230550] Pid: 19139, comm: crashme Not tainted 2.6.23-rc6-git2 #1
> > > [ 7487.236896] RIP: 0033:[<00510eea>]  [<00510eea>]
> > > [ 7487.242925] RSP: 002b:7fffc9a8ec10  EFLAGS: 00010e83
> > > [ 7487.248234] RAX: 8c4a RBX: 004014f1 RCX: 
> > > 2b20e11c8b37
> > > [ 7487.255361] RDX: 00510ee0 RSI:  RDI: 
> > > 000a
> > > [ 7487.262489] RBP: 7fffc9a8ec10 R08: 7fffc9a8eb60 R09: 
> > > 
> > > [ 7487.269616] R10: 0008 R11: 0612 R12: 
> > > 
> > > [ 7487.276743] R13: 7fffc9a8ee00 R14:  R15: 
> > > 
> > > [ 7487.283871] FS:  2b20e13676d0() GS:81011fc75840() 
> > > knlGS:
> > > [ 7487.291952] CS:  0010 DS:  ES:  CR0: 8005003b
> > > [ 7487.297693] CR2: ff019b53 CR3: 5be6b000 CR4: 
> > > 06e0
> > > [ 7487.304821] DR0:  DR1:  DR2: 
> > > 
> > > [ 7487.311949] DR3:  DR6: 0ff0 DR7: 
> > > 0400
> > > [ 7487.319076] Process crashme (pid: 19139, threadinfo 81010683, 
> > > task 810102cf5040)
> > > [ 7487.327511]
> > > [ 7487.329009] RIP  [<00510eea>]
> > > [ 7487.332690]  RSP <7fffc9a8ec10>
> > > [ 7487.336180] CR2: ff019b53
> > > [ 7487.339810] Kernel panic - not syncing: Fatal exception
> > -
> 
> ---
> ~Randy
> *** Remember to use Documentation/SubmitChecklist when testing your code ***
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [EMAIL PROTECTED]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-14 Thread Randy Dunlap
On Fri, 14 Sep 2007 21:28:12 -0700 (PDT) Linus Torvalds wrote:

> On Wed, 12 Sep 2007, Randy Dunlap wrote:
> >
> > I run almost-daily kernel testing.  I haven't seen 'crashme' cause a
> > kernel fault until today, and now I've seen it twice on 2.6.23-rc6-git2,
> > x86_64.  After the first fault, I ran 'crashme' about 10 more times
> > to get the second fault (usually for 10 minutes, one time for 30
> > minutes).
> 
> Interesting. If this is reproducible for you, can you try to narrow down 
> (with bit-bisect) roughly when it started.

It doesn't happen quite on demand.  Only 2 times in several hours
of testing of the last few days.

> > There is very little helpful info.  RIP is strange, e.g.: 0051b446
> > The call stack is not printed.  No kernel symbols are printed,
> > even though I have CONFIG_KALLSYMS{_ALL}=y.
> 
> It looks like it's a page fault as a result of a *user*space* access, and 
> most likely your machine would continue happily, except you have 
> "panic_on_oops" set, so when the oops happens, it shuts the system down.

I see.

> Now, the reason I say it looks like a user space access is that you have
> 
>   RIP: 0033:[<00510eea>]
>   RSP: 002b:7fffc9a8ec10
> 
> which are all user space segments. So the register contents clearly say 
> "page fault in user space".
> 
> However, what makes the kernel oops (rather than just send a SIGSEGV) is 
> that the page fault "error code" is zero (that's the number that is 
> printed out just after the "Oops" string). For a normal user space access, 
> you should have bit #2 set in the error code.
> 
> So the kernel thinks it's a kernel page fault, because the page fault 
> error code says so. But everything else seems to indicate that it's really 
> user mode.. It would be very interesting to hear when this started 
> happening.
> 
> Even if you cannot bisect it down all the way, since you say that you do 
> almost daily kernel testing, is this really new to 2.6.23-rc6-git2, and 
> 2.6.23-rc6-git1 was fine? 

I'll check back thru my test logs.

> Andi, anything comes to mind?
> 
>   Linus
> 
> ---
> > [ 7487.208128] Unable to handle kernel paging request at ff019b53 
> > RIP:
> > [ 7487.212752]  [<00510eea>]
> > [ 7487.218537] PGD 10c1a2067 PUD 0
> > [ 7487.221811] Oops:  [1] SMP
> > [ 7487.224989] CPU 2
> > [ 7487.227024] Modules linked in: loop
> > [ 7487.230550] Pid: 19139, comm: crashme Not tainted 2.6.23-rc6-git2 #1
> > [ 7487.236896] RIP: 0033:[<00510eea>]  [<00510eea>]
> > [ 7487.242925] RSP: 002b:7fffc9a8ec10  EFLAGS: 00010e83
> > [ 7487.248234] RAX: 8c4a RBX: 004014f1 RCX: 
> > 2b20e11c8b37
> > [ 7487.255361] RDX: 00510ee0 RSI:  RDI: 
> > 000a
> > [ 7487.262489] RBP: 7fffc9a8ec10 R08: 7fffc9a8eb60 R09: 
> > 
> > [ 7487.269616] R10: 0008 R11: 0612 R12: 
> > 
> > [ 7487.276743] R13: 7fffc9a8ee00 R14:  R15: 
> > 
> > [ 7487.283871] FS:  2b20e13676d0() GS:81011fc75840() 
> > knlGS:
> > [ 7487.291952] CS:  0010 DS:  ES:  CR0: 8005003b
> > [ 7487.297693] CR2: ff019b53 CR3: 5be6b000 CR4: 
> > 06e0
> > [ 7487.304821] DR0:  DR1:  DR2: 
> > 
> > [ 7487.311949] DR3:  DR6: 0ff0 DR7: 
> > 0400
> > [ 7487.319076] Process crashme (pid: 19139, threadinfo 81010683, 
> > task 810102cf5040)
> > [ 7487.327511]
> > [ 7487.329009] RIP  [<00510eea>]
> > [ 7487.332690]  RSP <7fffc9a8ec10>
> > [ 7487.336180] CR2: ff019b53
> > [ 7487.339810] Kernel panic - not syncing: Fatal exception
> -

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: crashme fault

2007-09-14 Thread Linus Torvalds


On Wed, 12 Sep 2007, Randy Dunlap wrote:
>
> I run almost-daily kernel testing.  I haven't seen 'crashme' cause a
> kernel fault until today, and now I've seen it twice on 2.6.23-rc6-git2,
> x86_64.  After the first fault, I ran 'crashme' about 10 more times
> to get the second fault (usually for 10 minutes, one time for 30
> minutes).

Interesting. If this is reproducible for you, can you try to narrow down 
(with bit-bisect) roughly when it started.

> There is very little helpful info.  RIP is strange, e.g.: 0051b446
> The call stack is not printed.  No kernel symbols are printed,
> even though I have CONFIG_KALLSYMS{_ALL}=y.

It looks like it's a page fault as a result of a *user*space* access, and 
most likely your machine would continue happily, except you have 
"panic_on_oops" set, so when the oops happens, it shuts the system down.

Now, the reason I say it looks like a user space access is that you have

RIP: 0033:[<00510eea>]
RSP: 002b:7fffc9a8ec10

which are all user space segments. So the register contents clearly say 
"page fault in user space".

However, what makes the kernel oops (rather than just send a SIGSEGV) is 
that the page fault "error code" is zero (that's the number that is 
printed out just after the "Oops" string). For a normal user space access, 
you should have bit #2 set in the error code.

So the kernel thinks it's a kernel page fault, because the page fault 
error code says so. But everything else seems to indicate that it's really 
user mode.. It would be very interesting to hear when this started 
happening.

Even if you cannot bisect it down all the way, since you say that you do 
almost daily kernel testing, is this really new to 2.6.23-rc6-git2, and 
2.6.23-rc6-git1 was fine? 

Andi, anything comes to mind?

Linus

---
> [ 7487.208128] Unable to handle kernel paging request at ff019b53 RIP:
> [ 7487.212752]  [<00510eea>]
> [ 7487.218537] PGD 10c1a2067 PUD 0
> [ 7487.221811] Oops:  [1] SMP
> [ 7487.224989] CPU 2
> [ 7487.227024] Modules linked in: loop
> [ 7487.230550] Pid: 19139, comm: crashme Not tainted 2.6.23-rc6-git2 #1
> [ 7487.236896] RIP: 0033:[<00510eea>]  [<00510eea>]
> [ 7487.242925] RSP: 002b:7fffc9a8ec10  EFLAGS: 00010e83
> [ 7487.248234] RAX: 8c4a RBX: 004014f1 RCX: 
> 2b20e11c8b37
> [ 7487.255361] RDX: 00510ee0 RSI:  RDI: 
> 000a
> [ 7487.262489] RBP: 7fffc9a8ec10 R08: 7fffc9a8eb60 R09: 
> 
> [ 7487.269616] R10: 0008 R11: 0612 R12: 
> 
> [ 7487.276743] R13: 7fffc9a8ee00 R14:  R15: 
> 
> [ 7487.283871] FS:  2b20e13676d0() GS:81011fc75840() 
> knlGS:
> [ 7487.291952] CS:  0010 DS:  ES:  CR0: 8005003b
> [ 7487.297693] CR2: ff019b53 CR3: 5be6b000 CR4: 
> 06e0
> [ 7487.304821] DR0:  DR1:  DR2: 
> 
> [ 7487.311949] DR3:  DR6: 0ff0 DR7: 
> 0400
> [ 7487.319076] Process crashme (pid: 19139, threadinfo 81010683, task 
> 810102cf5040)
> [ 7487.327511]
> [ 7487.329009] RIP  [<00510eea>]
> [ 7487.332690]  RSP <7fffc9a8ec10>
> [ 7487.336180] CR2: ff019b53
> [ 7487.339810] Kernel panic - not syncing: Fatal exception
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/