Hi,
We have seen the following kernel panic, happened during loading a kernel
module:
[ 536.107430] Unable to handle kernel paging request for data at address
0xd76a907c
[ 536.114922] Faulting instruction address: 0xc0000770
[ 536.119891] Oops: Kernel access of bad area, sig: 11 [#1]
[ 536.125291] CCEP MPC8541E
[ 536.127908] Modules linked in: pppoe(+) nf_conntrack_ipv6 ...
[ 536.155705] NIP: c0000770 LR: c0000770 CTR: d76ab0d4
[ 536.160674] REGS: d76a8f24 TRAP: 0300 Not tainted (2.6.33-ccep)
[ 536.166857] MSR: 00021000 <ME,CE> CR: 24000482 XER: 20000000
[ 536.172718] DEAR: d76a907c, ESR: 00800000
[ 536.176728] TASK = cbd7f9f0[972] 'insmod' THREAD: cbeb2000
[ 536.182041] GPR00: 83cbfff8 d76a8fd4 cbd7f9f0 00000000 83cbfff8 00000000
cbeb3e1e 00000000
[ 536.190438] GPR08: 0000327b d76aa000 24000482 d76a8fd4 cbd7fc08 10019b04
100c2e5c 00000000
[ 536.198836] GPR16: 00000000 1009bafc 100c44a4 100c2ed4 100c0000 100d1e60
100d1ca8 100017ab
[ 536.207235] GPR24: 100017ae 10001936 c0343ae8 10012018 00000000 834bffe8
836bffec 838bfff0
[ 536.215819] NIP [c0000770] InstructionStorage+0xb0/0xc0
[ 536.221048] LR [c0000770] InstructionStorage+0xb0/0xc0
[ 536.226188] Call Trace:
[ 536.228630] Instruction dump:
[ 536.231600] 90eb002c 910b0030 7cbe0aa6 90ab00b8 7d846378 38a00000
39400401 914b00b0
[ 536.239386] 3d400002 614a1002 512a0420 4800d6f5 <c000e43c> c000e65c
60000000 60000000
[ 536.247348] Kernel panic - not syncing: Fatal exception in interrupt
[ 536.253704] Call Trace:
[ 536.256149] Rebooting in 10 seconds..
The system crashes inside the return of the init entry point of the kernel
module.
I've found the following root cause:
(1) The system has a high number of NAT rules configured, which created a
bigger vmalloc area.
I've checked this by looking at /proc/vmallocinfo.
(2) The kernel module ELF file contains the separate section .init.text for
the init entry point,
which is marked with __init, as usual.
(3) The kernel module ELF file contains the function prologue and epilogue
in the .text section.
(4) The epilogue is also called from the init entry point, in order to
return to the caller.
It is intended to restore the non-volatile registers from the stack
and to jump to the caller.
(5) Because of (1), it is not more possible to jump by a relative branch
instruction. The distance is too big.
Instead, the trampoline method is applied, which allows longer jumps
via register.
(please see see do_plt_call() in arch/powerpc/kernel/module_32.c)
(6) Unfortunately, the trampoline code (do_plt_call()) is using register
r11 to setup the jump.
It looks like the prologue and epilogue are using also the register
r11, in order to point to the previous stack frame.
This is a conflict !!! The trampoline code is damaging the content of
r11.
According to the current EABI definitions, the register r11 has got a dedicated
function (pointer to previous stack frame).
In the following, there are parts of the prologue/epilogue shown, which are
generated by the compiler:
...
00000084 <_rest32gpr_28>:
84: 83 8b ff f0 lwz r28,-16(r11)
00000088 <_rest32gpr_29>:
88: 83 ab ff f4 lwz r29,-12(r11)
0000008c <_rest32gpr_30>:
8c: 83 cb ff f8 lwz r30,-8(r11)
00000090 <_rest32gpr_31>:
90: 83 eb ff fc lwz r31,-4(r11)
94: 4e 80 00 20 blr
00000098 <_rest32gpr_14_x>:
98: 81 cb ff b8 lwz r14,-72(r11)
...
I'd suggest to use register r12 instead of r11 in the trampoline generation
code, in do_plt_call() (arch/powerpc/kernel/module_32.c).
I'm using kernel 2.6.33, but I think it is also relevant for the current kernel
release.
Below, there is the complete debug sessions, showing more the details.
Thanks
Steffen Rumler
--
0xd54990c0: addi r11,r1,48
0xd54990c4: mr r3,r29
0xd54990c8: b 0xd5499100 <-- going to return from the init entry
point
(gdb) bt
#0 0xd5499100 in ?? ()
#1 0xd54990ac in ?? ()
#2 0xc0001db0 in do_one_initcall (fn=0, wait=1131130) at init/main.c:719
#3 0xc0059e50 in sys_init_module (umod=<value optimized out>, ...
#4 0xc000e038 in syscall_dotrace_cont () at
arch/powerpc/kernel/entry_32.S:331
Backtrace stopped: frame did not save the PC
(gdb) info reg r1
r1 0xcbbdbed0 3418210000
(gdb) x/2x 0xcbbdbed0
0xcbbdbed0: 0xcbbdbf00 0xd54990ac
(gdb) x/2x 0xcbbdbf00
0xcbbdbf00: 0xcbbdbf20 0xc0001db0
(gdb) info reg r11
r11 0xcbbdbf00 3418210048
--> the stack is OK here
--> r11 is OK, pointing to the previous stack frame
0xd5499100: lis r11,-10381 <-- this is the trampoline code
using/changing r11 (do_plt_call()).
0xd5499104: addi r11,r11,-3888
0xd5499108: mtctr r11
(gdb) info reg r11
r11 0xd772f0d0 3614634192
<-- r11 is now damaged !!!
0xd549910c: bctr
0xd772f0d0: lwz r28,-16(r11) <-- the epilogue, using damaged r11 as
stack pointer
0xd772f0d4: lwz r29,-12(r11)
0xd772f0d8: lwz r30,-8(r11)
0xd772f0dc: lwz r0,4(r11)
0xd772f0e0: lwz r31,-4(r11)
0xd772f0e4: mtlr r0
0xd772f0e8: mr r1,r11
(gdb) info reg lr
lr 0x83abfff4 0x83abfff4
0xd772f0ec: blr
(gdb) stepi
Program received signal SIGSTOP, Stopped (signal).
---> the kernel panic !!!
_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev
