Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-07 Thread Benjamin Herrenschmidt 
On Wed, 2009-10-07 at 14:26 +0530, Sachin Sant wrote:

> As Ben suggested, i changed LOAD_REG_IMMEDIATE() to LOAD_REG_ADDR()
> as follows.
> 
> -   LOAD_REG_IMMEDIATE(r4,ftrace_return_to_handler)
> +   LOAD_REG_ADDR(r4,ftrace_return_to_handler)
> 
> With this change compile time warnings about bad relocations
> related to ftrace are gone.

You also need to make sure r2 is properly loaded with the kernel TOC
pointer, something like ld r2,PACAKTOC(r13) before the LOAD_REG_ADDR.

Cheers,
Ben.

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-07 Thread Sachin Sant 

Steven Rostedt wrote:

On Tue, 2009-10-06 at 07:20 +1100, Benjamin Herrenschmidt wrote:
  

On Mon, 2009-10-05 at 09:25 -0400, Steven Rostedt wrote:


Sachin, can you give me more details on how you built that kernel ? (or
give them again in case I missed them the first time around :-), ie,
what toolchain, options, etc... or even better, give me remote access to
the build host ?
  

Ok, got access and had a quick look... seems to be a toolchain problem
to me. I'll investigate more tomorrow.


Hi Ben,

Any more word on this issue? 
  

Didn't you fix it using a TOC access ?

Unless I'm confusing things, I think the problem is the usage
of LOAD_REG_IMMEDIATE which generates relocs that we don't support
when CONFIG_RELOCATABLE is set.

I've merged a patch that post-processes the kernel now, to check
for such relocs so at least you should be warned at build time.



I thought we had two issues. One was the use of the relocs that did
cause issues. But then there was still crashes reported after that.
IIRC. I'm still suffering jetlag, so my memory is not that fresh about
  

As Ben suggested, i changed LOAD_REG_IMMEDIATE() to LOAD_REG_ADDR()
as follows.

-   LOAD_REG_IMMEDIATE(r4,ftrace_return_to_handler)
+   LOAD_REG_ADDR(r4,ftrace_return_to_handler)

With this change compile time warnings about bad relocations
related to ftrace are gone.

Before the change :
WARNING: 6 bad relocations
c0008f1a R_PPC64_ADDR16_HIGHEST  __ksymtab+0x00742110
c0008f1e R_PPC64_ADDR16_HIGHER  __ksymtab+0x00742110
c0008f26 R_PPC64_ADDR16_HI  __ksymtab+0x00742110
c0008f2a R_PPC64_ADDR16_LO  __ksymtab+0x00742110
c085e118 R_PPC64_ADDR64__crc_per_cpu__softirq_work_list
c08662d0 R_PPC64_ADDR64__crc_simple_prepare_write

After the change :
WARNING: 2 bad relocations
c085e118 R_PPC64_ADDR64__crc_per_cpu__softirq_work_list
c08662d0 R_PPC64_ADDR64__crc_simple_prepare_write

But i still run into oops while using ftrace function_graph.

Thanks
-Sachin



--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

diff -Naurp a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
--- a/arch/powerpc/kernel/entry_64.S	2009-10-06 15:31:29.0 +0530
+++ b/arch/powerpc/kernel/entry_64.S	2009-10-06 15:34:00.0 +0530
@@ -1038,7 +1038,7 @@ _GLOBAL(mod_return_to_handler)
 	 * We are in a module using the module's TOC.
 	 * Switch to our TOC to run inside the core kernel.
 	 */
-	LOAD_REG_IMMEDIATE(r4,ftrace_return_to_handler)
+	LOAD_REG_ADDR(r4,ftrace_return_to_handler)
 	ld	r2, 8(r4)
 
 	bl	.ftrace_return_to_handler
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-05 Thread Steven Rostedt 
On Tue, 2009-10-06 at 07:20 +1100, Benjamin Herrenschmidt wrote:
> On Mon, 2009-10-05 at 09:25 -0400, Steven Rostedt wrote:
> > > > Sachin, can you give me more details on how you built that kernel ? (or
> > > > give them again in case I missed them the first time around :-), ie,
> > > > what toolchain, options, etc... or even better, give me remote access to
> > > > the build host ?
> > > 
> > > Ok, got access and had a quick look... seems to be a toolchain problem
> > > to me. I'll investigate more tomorrow.
> > 
> > Hi Ben,
> > 
> > Any more word on this issue? 
> 
> Didn't you fix it using a TOC access ?
> 
> Unless I'm confusing things, I think the problem is the usage
> of LOAD_REG_IMMEDIATE which generates relocs that we don't support
> when CONFIG_RELOCATABLE is set.
> 
> I've merged a patch that post-processes the kernel now, to check
> for such relocs so at least you should be warned at build time.

I thought we had two issues. One was the use of the relocs that did
cause issues. But then there was still crashes reported after that.
IIRC. I'm still suffering jetlag, so my memory is not that fresh about
this.

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-05 Thread Benjamin Herrenschmidt 
On Mon, 2009-10-05 at 09:25 -0400, Steven Rostedt wrote:
> > > Sachin, can you give me more details on how you built that kernel ? (or
> > > give them again in case I missed them the first time around :-), ie,
> > > what toolchain, options, etc... or even better, give me remote access to
> > > the build host ?
> > 
> > Ok, got access and had a quick look... seems to be a toolchain problem
> > to me. I'll investigate more tomorrow.
> 
> Hi Ben,
> 
> Any more word on this issue? 

Didn't you fix it using a TOC access ?

Unless I'm confusing things, I think the problem is the usage
of LOAD_REG_IMMEDIATE which generates relocs that we don't support
when CONFIG_RELOCATABLE is set.

I've merged a patch that post-processes the kernel now, to check
for such relocs so at least you should be warned at build time.

Cheers,
Ben.

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-05 Thread Steven Rostedt 
On Mon, 2009-09-14 at 18:00 +1000, Benjamin Herrenschmidt wrote:
> On Mon, 2009-09-14 at 06:25 +1000, Benjamin Herrenschmidt wrote:
> > > > .../...
> > > > 
> > > > > Something is totally messed up here.
> > > > 
> > > > Could it be that we don't handle R_PPC64_ADDR16_* relocs in
> > > > arch/powerpc/kernel/modules/module_64.c ?
> > > > 
> > > > Sachin, do you see a bunch of "Unknown ADD relocation" in your dmesg ?
> > > 
> > > Ben,
> > > 
> > > The thing is, this is kernel proper. This code is in entry_64.S not in
> > > the module code.
> > 
> > Argh... indeed.
> > 
> > Sachin, can you give me more details on how you built that kernel ? (or
> > give them again in case I missed them the first time around :-), ie,
> > what toolchain, options, etc... or even better, give me remote access to
> > the build host ?
> 
> Ok, got access and had a quick look... seems to be a toolchain problem
> to me. I'll investigate more tomorrow.

Hi Ben,

Any more word on this issue?

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-14 Thread Benjamin Herrenschmidt 
On Mon, 2009-09-14 at 06:25 +1000, Benjamin Herrenschmidt wrote:
> > > .../...
> > > 
> > > > Something is totally messed up here.
> > > 
> > > Could it be that we don't handle R_PPC64_ADDR16_* relocs in
> > > arch/powerpc/kernel/modules/module_64.c ?
> > > 
> > > Sachin, do you see a bunch of "Unknown ADD relocation" in your dmesg ?
> > 
> > Ben,
> > 
> > The thing is, this is kernel proper. This code is in entry_64.S not in
> > the module code.
> 
> Argh... indeed.
> 
> Sachin, can you give me more details on how you built that kernel ? (or
> give them again in case I missed them the first time around :-), ie,
> what toolchain, options, etc... or even better, give me remote access to
> the build host ?

Ok, got access and had a quick look... seems to be a toolchain problem
to me. I'll investigate more tomorrow.

Cheers,
Ben.


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-13 Thread Benjamin Herrenschmidt 

> > .../...
> > 
> > > Something is totally messed up here.
> > 
> > Could it be that we don't handle R_PPC64_ADDR16_* relocs in
> > arch/powerpc/kernel/modules/module_64.c ?
> > 
> > Sachin, do you see a bunch of "Unknown ADD relocation" in your dmesg ?
> 
> Ben,
> 
> The thing is, this is kernel proper. This code is in entry_64.S not in
> the module code.

Argh... indeed.

Sachin, can you give me more details on how you built that kernel ? (or
give them again in case I missed them the first time around :-), ie,
what toolchain, options, etc... or even better, give me remote access to
the build host ?

Cheers,
Ben.

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-13 Thread Steven Rostedt 
On Sun, 2009-09-13 at 14:37 +1000, Benjamin Herrenschmidt wrote:
> On Sun, 2009-09-13 at 00:07 -0400, Steven Rostedt wrote:
> >982: R_PPC64_ADDR16_HIGHEST
> > ftrace_return_to_handler
> >  984:   60 84 00 00 ori r4,r4,0
> > 986: R_PPC64_ADDR16_HIGHER
> > ftrace_return_to_handler
> >  988:   78 84 07 c6 rldicr  r4,r4,32,31
> >  98c:   64 84 00 00 orisr4,r4,0
> > 98e: R_PPC64_ADDR16_HI
> > ftrace_return_to_handler
> >  990:   60 84 00 00 ori r4,r4,0
> > 992: R_PPC64_ADDR16_LO
> > ftrace_return_to_handler
> >  994:   e8 44 00 08 ld  r2,8(r4)
> >  998:   48 00 00 01 bl  998 <.mod_return_to_handler+0x30>
> > 998:
> > R_PPC64_REL24  .ftrace_return_to_handler
> >  99c:   60 00 00 00 nop
> >  9a0:   7c 68 03 a6 mtlrr3
> 
> .../...
> 
> > Something is totally messed up here.
> 
> Could it be that we don't handle R_PPC64_ADDR16_* relocs in
> arch/powerpc/kernel/modules/module_64.c ?
> 
> Sachin, do you see a bunch of "Unknown ADD relocation" in your dmesg ?

Ben,

The thing is, this is kernel proper. This code is in entry_64.S not in
the module code.

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-12 Thread Benjamin Herrenschmidt 
On Sun, 2009-09-13 at 00:07 -0400, Steven Rostedt wrote:
>982: R_PPC64_ADDR16_HIGHEST
> ftrace_return_to_handler
>  984:   60 84 00 00 ori r4,r4,0
> 986: R_PPC64_ADDR16_HIGHER
> ftrace_return_to_handler
>  988:   78 84 07 c6 rldicr  r4,r4,32,31
>  98c:   64 84 00 00 orisr4,r4,0
> 98e: R_PPC64_ADDR16_HI
> ftrace_return_to_handler
>  990:   60 84 00 00 ori r4,r4,0
> 992: R_PPC64_ADDR16_LO
> ftrace_return_to_handler
>  994:   e8 44 00 08 ld  r2,8(r4)
>  998:   48 00 00 01 bl  998 <.mod_return_to_handler+0x30>
> 998:
> R_PPC64_REL24  .ftrace_return_to_handler
>  99c:   60 00 00 00 nop
>  9a0:   7c 68 03 a6 mtlrr3

.../...

> Something is totally messed up here.

Could it be that we don't handle R_PPC64_ADDR16_* relocs in
arch/powerpc/kernel/modules/module_64.c ?

Sachin, do you see a bunch of "Unknown ADD relocation" in your dmesg ?

Ben.


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-12 Thread Steven Rostedt 
On Wed, 2009-09-09 at 11:57 +0530, Sachin Sant wrote:
> Steven Rostedt wrote:
> > I'm going through old email, and I found this. Do you still see this
> > error. I don't recall seeing it myself.
> >   
> I can still recreate this with 31-rc9. When i enable tracing
> with function_graph i notice the following oops. This happens
> only once. Later if i try to enable/disable tracing i don't
> get this oops message. This behavior is observed only with
> function_graph. Other tracers work fine.
> 
> Oops: Kernel access of bad area, sig: 11 [#1]
> SMP NR_CPUS=1024 NUMA pSeries
> Modules linked in: ipv6 fuse loop dm_mod sr_mod ehea ibmveth sg cdrom sd_mod 
> crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
> NIP: c0008f30 LR: c0008f04 CTR: 800f6d68
> REGS: c0003e98f560 TRAP: 0300   Not tainted  (2.6.31-rc9)
> MSR: 80009032   CR: 24000422  XER: 0020
> DAR: 0008, DSISR: 4000
> TASK = c0003e953b20[2483] 'irqbalance' THREAD: c0003e98c000 CPU: 1
> GPR00: c0008f04 c0003e98f7e0 d117ed38 
> GPR04:  6600 10bf 
> GPR08:  80010021bb40 00ff 80010021bb60
> GPR12: 0002 c1032800  effdff68
> GPR16: 0fffa39fd6a0 0fffa39e6c38 c0003ebe9c38 f000
> GPR20: c0002a6cf980 c0003e98fdf8 c0003e98fba8 0fffa174
> GPR24: f000 80010300 ffe0 0009
> GPR28: c0003db4 0002 d117da78 c0003e98f850
> NIP [c0008f30] .mod_return_to_handler+0x2c/0x64
> LR [c0008f04] .mod_return_to_handler+0x0/0x64
> Call Trace:
> [c0003e98f7e0] [c0002a6cf980] 0xc0002a6cf980 (unreliable)
> [c0003e98f850] [c0008f04] .mod_return_to_handler+0x0/0x64
> [c0003e98f900] [c0008f04] .mod_return_to_handler+0x0/0x64
> [c0003e98f9a0] [c0008f04] .mod_return_to_handler+0x0/0x64
> [c0003e98fa30] [c0008ed0] .return_to_handler+0x0/0x34 
> (.bad_page_fault+0xc8/0xe8)
> [c0003e98fb30] [c0008ed0] .return_to_handler+0x0/0x34 
> (handle_page_fault+0x3c/0x5c)
> [c0003e98fc20] [c0008ed0] .return_to_handler+0x0/0x34 
> (.ehea_h_query_ehea_port+0x74/0x9c [ehea])
> [c0003e98fcd0] [c0008ed0] .return_to_handler+0x0/0x34 
> (.ehea_get_stats+0xa0/0x1d0 [ehea])
> [c0003e98fd80] [c0008ed0] .return_to_handler+0x0/0x34 
> (.dev_get_stats+0x50/0xec)
> [c0003e98fe30] [c0008ed0] .return_to_handler+0x0/0x34 
> (.dev_seq_show+0x5c/0x140)
> Instruction dump:
> 4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
> 6084 788407c6 6484 6084  48126375 6000 7c6803a6
> ---[ end trace bb43efc994aed790 ]---

I'm looking at your back dump and this really bothers me. I did a
objdump -dr arch/powerpc/kernel/entry_64.o and this is what I have:

0968 <.mod_return_to_handler>:
 968:   f8 81 ff e0 std r4,-32(r1)
 96c:   f8 61 ff e8 std r3,-24(r1)
 970:   f8 41 ff f0 std r2,-16(r1)
 974:   fb e1 ff f8 std r31,-8(r1)
 978:   7c 3f 0b 78 mr  r31,r1
 97c:   f8 21 ff 91 stdur1,-112(r1)
 980:   3c 80 00 00 lis r4,0
982: R_PPC64_ADDR16_HIGHEST ftrace_return_to_handler
 984:   60 84 00 00 ori r4,r4,0
986: R_PPC64_ADDR16_HIGHER  ftrace_return_to_handler
 988:   78 84 07 c6 rldicr  r4,r4,32,31
 98c:   64 84 00 00 orisr4,r4,0
98e: R_PPC64_ADDR16_HI  ftrace_return_to_handler
 990:   60 84 00 00 ori r4,r4,0
992: R_PPC64_ADDR16_LO  ftrace_return_to_handler
 994:   e8 44 00 08 ld  r2,8(r4)
 998:   48 00 00 01 bl  998 <.mod_return_to_handler+0x30>
998: R_PPC64_REL24  .ftrace_return_to_handler
 99c:   60 00 00 00 nop
 9a0:   7c 68 03 a6 mtlrr3


The bug happened at mod_return_to_handler+0x2c which is 994 above. Your
reg dump shows r4 is 0, and worse yet, looking at the code:

4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
6084 788407c6 6484 6084  48126375 6000
7c6803a6

The 6484 6084 shows that the linker never resolved the address
to ftrace_return_to_handle??

Something is totally messed up here.

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-11 Thread Steven Rostedt 
On Thu, 2009-09-10 at 11:02 +0530, Sachin Sant wrote:
> Steven Rostedt wrote:
> > Ah, seems the bug happens to be in the module handling. Does the call
> > back always have .mod_return_to_handler?
> >   
> Yes. Every time it ends up in .mod_return_to_handler

Hmm, I still can not reproduce it, and I've confirmed that I
hit .mod_return_to_handler too.

Could you apply the below patch. This wont fix anything, but it will at
least make the trace back show the real functions that were called.

Thanks,

-- Steve

diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 892a9f2..a2e1b15 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1014,9 +1014,13 @@ void show_stack(struct task_struct *tsk, unsigned long 
*stack)
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
int curr_frame = current->curr_ret_stack;
extern void return_to_handler(void);
-   unsigned long addr = (unsigned long)return_to_handler;
+   unsigned long rth = (unsigned long)return_to_handler;
+   unsigned long mrth = -1;
 #ifdef CONFIG_PPC64
-   addr = *(unsigned long*)addr;
+   extern void mod_return_to_handler(void);
+   rth = *(unsigned long *)rth;
+   mrth = (unsigned long)mod_return_to_handler;
+   mrth = *(unsigned long *)mrth;
 #endif
 #endif
 
@@ -1042,7 +1046,7 @@ void show_stack(struct task_struct *tsk, unsigned long 
*stack)
if (!firstframe || ip != lr) {
printk("["REG"] ["REG"] %pS", sp, ip, (void *)ip);
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
-   if (ip == addr && curr_frame >= 0) {
+   if ((ip == rth || ip == mrth) && curr_frame >= 0) {
printk(" (%pS)",
   (void 
*)current->ret_stack[curr_frame].ret);
curr_frame--;


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-10 Thread Sachin Sant 

Steven Rostedt wrote:

On Thu, 2009-09-10 at 11:02 +0530, Sachin Sant wrote:
  

Steven Rostedt wrote:


Ah, seems the bug happens to be in the module handling. Does the call
back always have .mod_return_to_handler?
  
  

Yes. Every time it ends up in .mod_return_to_handler



BTW, do you have CONFIG_IRQSTACK set?
  

Yes. That option is enabled.

CONFIG_DEBUGGER=y
CONFIG_IRQSTACKS=y
# CONFIG_VIRQ_DEBUG is not set

thanks
-Sachin

--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-10 Thread Steven Rostedt 
On Thu, 2009-09-10 at 11:02 +0530, Sachin Sant wrote:
> Steven Rostedt wrote:
> > Ah, seems the bug happens to be in the module handling. Does the call
> > back always have .mod_return_to_handler?
> >   
> Yes. Every time it ends up in .mod_return_to_handler

BTW, do you have CONFIG_IRQSTACK set?

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-09 Thread Sachin Sant 

Steven Rostedt wrote:

Ah, seems the bug happens to be in the module handling. Does the call
back always have .mod_return_to_handler?
  

Yes. Every time it ends up in .mod_return_to_handler

Thanks
-Sachin


This doesn't surprise me any. The module code is a bit harry, and
function graph does some crazy crap with it.

-- Steve
  



--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-09 Thread Steven Rostedt 
On Wed, 2009-09-09 at 11:57 +0530, Sachin Sant wrote:
> Steven Rostedt wrote:
> > I'm going through old email, and I found this. Do you still see this
> > error. I don't recall seeing it myself.
> >   
> I can still recreate this with 31-rc9. When i enable tracing
> with function_graph i notice the following oops. This happens
> only once. Later if i try to enable/disable tracing i don't
> get this oops message. This behavior is observed only with
> function_graph. Other tracers work fine.
> 
> Oops: Kernel access of bad area, sig: 11 [#1]
> SMP NR_CPUS=1024 NUMA pSeries
> Modules linked in: ipv6 fuse loop dm_mod sr_mod ehea ibmveth sg cdrom sd_mod 
> crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
> NIP: c0008f30 LR: c0008f04 CTR: 800f6d68
> REGS: c0003e98f560 TRAP: 0300   Not tainted  (2.6.31-rc9)
> MSR: 80009032   CR: 24000422  XER: 0020
> DAR: 0008, DSISR: 4000
> TASK = c0003e953b20[2483] 'irqbalance' THREAD: c0003e98c000 CPU: 1
> GPR00: c0008f04 c0003e98f7e0 d117ed38 
> GPR04:  6600 10bf 
> GPR08:  80010021bb40 00ff 80010021bb60
> GPR12: 0002 c1032800  effdff68
> GPR16: 0fffa39fd6a0 0fffa39e6c38 c0003ebe9c38 f000
> GPR20: c0002a6cf980 c0003e98fdf8 c0003e98fba8 0fffa174
> GPR24: f000 80010300 ffe0 0009
> GPR28: c0003db4 0002 d117da78 c0003e98f850
> NIP [c0008f30] .mod_return_to_handler+0x2c/0x64
> LR [c0008f04] .mod_return_to_handler+0x0/0x64
> Call Trace:
> [c0003e98f7e0] [c0002a6cf980] 0xc0002a6cf980 (unreliable)
> [c0003e98f850] [c0008f04] .mod_return_to_handler+0x0/0x64
> [c0003e98f900] [c0008f04] .mod_return_to_handler+0x0/0x64
> [c0003e98f9a0] [c0008f04] .mod_return_to_handler+0x0/0x64

Ah, seems the bug happens to be in the module handling. Does the call
back always have .mod_return_to_handler?

This doesn't surprise me any. The module code is a bit harry, and
function graph does some crazy crap with it.

-- Steve

> [c0003e98fa30] [c0008ed0] .return_to_handler+0x0/0x34 
> (.bad_page_fault+0xc8/0xe8)
> [c0003e98fb30] [c0008ed0] .return_to_handler+0x0/0x34 
> (handle_page_fault+0x3c/0x5c)
> [c0003e98fc20] [c0008ed0] .return_to_handler+0x0/0x34 
> (.ehea_h_query_ehea_port+0x74/0x9c [ehea])
> [c0003e98fcd0] [c0008ed0] .return_to_handler+0x0/0x34 
> (.ehea_get_stats+0xa0/0x1d0 [ehea])
> [c0003e98fd80] [c0008ed0] .return_to_handler+0x0/0x34 
> (.dev_get_stats+0x50/0xec)
> [c0003e98fe30] [c0008ed0] .return_to_handler+0x0/0x34 
> (.dev_seq_show+0x5c/0x140)
> Instruction dump:
> 4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
> 6084 788407c6 6484 6084  48126375 6000 7c6803a6
> ---[ end trace bb43efc994aed790 ]---
> 
> function_graph traces are recorded and can be retrieved using
> /sys/kernel/debug/tracing/trace.
> 
> 1)   3.936 us|}
> 1)   |.release_console_sem() {
> 1)   0.594 us|  ._spin_lock_irqsave();
> 1)   0.560 us|  ._call_console_drivers();
> 1)   0.580 us|  ._call_console_drivers();
> 1)   0.582 us|  ._spin_lock_irqsave();
> 1)   |  .up() {
> 1)   0.592 us|._spin_lock_irqsave();
> 1)   0.556 us|._spin_unlock_irqrestore();
> 1)   2.842 us|  }
> 1)   0.588 us|  ._spin_unlock_irqrestore();
> 1)   9.750 us|}
> 1) + 75.274 us   |  }
> 1)   |  .die() {
> 1)   |.oops_enter() {
> 
> Thanks
> -Sachin
> 

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-08 Thread Sachin Sant 

Steven Rostedt wrote:

I'm going through old email, and I found this. Do you still see this
error. I don't recall seeing it myself.
  

I can still recreate this with 31-rc9. When i enable tracing
with function_graph i notice the following oops. This happens
only once. Later if i try to enable/disable tracing i don't
get this oops message. This behavior is observed only with
function_graph. Other tracers work fine.

Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=1024 NUMA pSeries
Modules linked in: ipv6 fuse loop dm_mod sr_mod ehea ibmveth sg cdrom sd_mod 
crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
NIP: c0008f30 LR: c0008f04 CTR: 800f6d68
REGS: c0003e98f560 TRAP: 0300   Not tainted  (2.6.31-rc9)
MSR: 80009032   CR: 24000422  XER: 0020
DAR: 0008, DSISR: 4000
TASK = c0003e953b20[2483] 'irqbalance' THREAD: c0003e98c000 CPU: 1
GPR00: c0008f04 c0003e98f7e0 d117ed38 
GPR04:  6600 10bf 
GPR08:  80010021bb40 00ff 80010021bb60
GPR12: 0002 c1032800  effdff68
GPR16: 0fffa39fd6a0 0fffa39e6c38 c0003ebe9c38 f000
GPR20: c0002a6cf980 c0003e98fdf8 c0003e98fba8 0fffa174
GPR24: f000 80010300 ffe0 0009
GPR28: c0003db4 0002 d117da78 c0003e98f850
NIP [c0008f30] .mod_return_to_handler+0x2c/0x64
LR [c0008f04] .mod_return_to_handler+0x0/0x64
Call Trace:
[c0003e98f7e0] [c0002a6cf980] 0xc0002a6cf980 (unreliable)
[c0003e98f850] [c0008f04] .mod_return_to_handler+0x0/0x64
[c0003e98f900] [c0008f04] .mod_return_to_handler+0x0/0x64
[c0003e98f9a0] [c0008f04] .mod_return_to_handler+0x0/0x64
[c0003e98fa30] [c0008ed0] .return_to_handler+0x0/0x34 
(.bad_page_fault+0xc8/0xe8)
[c0003e98fb30] [c0008ed0] .return_to_handler+0x0/0x34 
(handle_page_fault+0x3c/0x5c)
[c0003e98fc20] [c0008ed0] .return_to_handler+0x0/0x34 
(.ehea_h_query_ehea_port+0x74/0x9c [ehea])
[c0003e98fcd0] [c0008ed0] .return_to_handler+0x0/0x34 
(.ehea_get_stats+0xa0/0x1d0 [ehea])
[c0003e98fd80] [c0008ed0] .return_to_handler+0x0/0x34 
(.dev_get_stats+0x50/0xec)
[c0003e98fe30] [c0008ed0] .return_to_handler+0x0/0x34 
(.dev_seq_show+0x5c/0x140)
Instruction dump:
4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
6084 788407c6 6484 6084  48126375 6000 7c6803a6
---[ end trace bb43efc994aed790 ]---

function_graph traces are recorded and can be retrieved using
/sys/kernel/debug/tracing/trace.

1)   3.936 us|}
1)   |.release_console_sem() {
1)   0.594 us|  ._spin_lock_irqsave();
1)   0.560 us|  ._call_console_drivers();
1)   0.580 us|  ._call_console_drivers();
1)   0.582 us|  ._spin_lock_irqsave();
1)   |  .up() {
1)   0.592 us|._spin_lock_irqsave();
1)   0.556 us|._spin_unlock_irqrestore();
1)   2.842 us|  }
1)   0.588 us|  ._spin_unlock_irqrestore();
1)   9.750 us|}
1) + 75.274 us   |  }
1)   |  .die() {
1)   |.oops_enter() {

Thanks
-Sachin

--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-08 Thread Steven Rostedt 
On Mon, 2009-08-03 at 16:10 +0530, Sachin Sant wrote:
> Steven Rostedt wrote:
> > Thanks,
> >
> > I've seen issues with my PPC box and function graph, but the bugs were
> > also caused by other changes. I'll boot up my PPC64 box and see if
> > I see the same issues you have.
> >   
> Hi Steven,
> 
> I can still recreate this issue with 2.6.31-rc5. Let me know
> if i can provide any information to find a solution for this.

Hi Sachin,

I'm going through old email, and I found this. Do you still see this
error. I don't recall seeing it myself.

Thanks,

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-08-03 Thread Sachin Sant 

Steven Rostedt wrote:

Thanks,

I've seen issues with my PPC box and function graph, but the bugs were
also caused by other changes. I'll boot up my PPC64 box and see if
I see the same issues you have.
  

Hi Steven,

I can still recreate this issue with 2.6.31-rc5. Let me know
if i can provide any information to find a solution for this.

Thanks
-Sachin


--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-07-14 Thread Steven Rostedt 

On Tue, 14 Jul 2009, Sachin Sant wrote:

> While enabling function_graph tracer on a Power6 box, machine
> crashed with following trace. Kernel version is 2.6.31-rc3.

Thanks,

I've seen issues with my PPC box and function graph, but the bugs were
also caused by other changes. I'll boot up my PPC64 box and see if
I see the same issues you have.

-- Steve

> 
> :/debug/tracing # echo function_graph > current_tracer :/debug/tracing # cat
> current_tracer function_graph
> :/debug/tracing # echo 1 > tracing_enabled
> 
> cpu 0x0: Vector: 300 (Data Access) at [c0003eb86de0]
>pc: c0008f30: .mod_return_to_handler+0x2c/0x64
>lr: c0008f04: .mod_return_to_handler+0x0/0x64
>sp: c0003eb87060
>   msr: 80009032
>   dar: 8
> dsisr: 4000
>  current = 0xc0003e83c080
>  paca= 0xc0ff2400
>pid   = 2700, comm = sshd
> enter ? for help
> [c0003eb870d0] c0008f04 .mod_return_to_handler+0x0/0x64
> [c0003eb871a0] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87290] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87330] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb873e0] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87470] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87500] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87640] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87730] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87830] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb878d0] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87a00] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87b30] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87cd0] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87d80] c0008ed0 .return_to_handler+0x0/0x34
> [c0003eb87e30] c0008ed0 .return_to_handler+0x0/0x34
> --- Exception: c01 (System Call) at 0fffb640a8f8
> SP (fffe5b9b960) is in userspace
> 0:mon> di c0008f04
> c0008f04  f881ffe0  std r4,-32(r1)
> c0008f08  f861ffe8  std r3,-24(r1)
> c0008f0c  f841fff0  std r2,-16(r1)
> c0008f10  fbe1fff8  std r31,-8(r1)
> c0008f14  7c3f0b78  mr  r31,r1
> c0008f18  f821ff91  stdur1,-112(r1)
> c0008f1c  3c80  lis r4,0
> c0008f20  6084  ori r4,r4,0
> c0008f24  788407c6  rldicr  r4,r4,32,31
> c0008f28  6484  orisr4,r4,0
> c0008f2c  6084  ori r4,r4,0
> c0008f30  e8440008  ld  r2,8(r4)  ^ PC points to this
> ^^^
> c0008f34  48123431  bl  c012c364#
> .ftrace_return_to_handler+0x0/0x110
> c0008f38  6000  nop
> c0008f3c  7c6803a6  mtlrr3
> c0008f40  e821  ld  r1,0(r1)
> 0:xmon> r
> R00 = c0008f04   R16 = 0fffb741d6a0
> R01 = c0003ea4b7e0   R17 = 0fffb7406c38
> R02 = d10bec80   R18 = c0003de41838
> R03 =    R19 = f000
> R04 =    R20 = c0003cfc8c80
> R05 = 6600   R21 = c0003ea4bdf8
> R06 = 10bf   R22 = c0003ea4bba8
> R07 =    R23 = 0fff8eb6
> R08 =    R24 = f000
> R09 = 80010021c740   R25 = 80010300
> R10 = 00ff   R26 = ffe0
> R11 = 80010021c760   R27 = 0009
> R12 = 0002   R28 = c0006f85
> R13 = c0ff2400   R29 = 0002
> R14 =    R30 = d10bd9c8
> R15 = effdff68   R31 = c0003ea4b850
> pc  = c0008f30 .mod_return_to_handler+0x2c/0x64
> lr  = c0008f04 .mod_return_to_handler+0x0/0x64
> msr = 80009032   cr  = 24000442
> ctr = 800f6d68   xer = 0020   trap =  300
> dar = 0008   dsisr = 4000
> 0:xmon>
> 
> Following tracers are supported by the kernel.
> :/debug/tracing # cat available_tracers function_graph function sched_switch
> nop
> 
> Other tracers function and sched_switch work fine. Having problem
> only with function_graph.
> 
> Have attached the .config.
> 
> Thanks
> -Sachin
> 
> 
> -- 
> 
> -
> Sachin Sant
> IBM Linux Technology Center
> India Systems and Technology Labs
> Bangalore, India
> -
> 
> 
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev