Re: [Xenomai-core] [PowerPC]Badness at mmu_context_nohash

2011-04-28 Thread Jean-Michel Hautbois
2011/4/27 Philippe Gerum r...@xenomai.org:
 On Wed, 2011-04-27 at 20:42 +0200, Jean-Michel Hautbois wrote:
 Hi list,

 I am currently using a Xenomai port on a linux 2.6.35.11 linux kernel
 and the adeos-ipipe-2.6.35.7-powerpc-2.12-01.patch.
 I am facing a scheduling issue on a P2020 (dual core PowerPC), and I
 get the following message :

 Badness at arch/powerpc/mm/mmu_context_nohash.c:209
 NIP: c0018d20 LR: c039b94c CTR: c00343e4
 REGS: ecfadce0 TRAP: 0700   Tainted: G        W    (2.6.35.11)
 MSR: 00021000 ME,CE  CR: 24000488  XER: 
 TASK = ec5220d0[496] 'sipaq' THREAD: ecfac000 CPU: 1
 GPR00: 0001 ecfadd90 ec5220d0 ec5df340 ec58a700   
 0003
 GPR08: c04a2d98 0007 c04a2d98 0067e000 0002f385 1007f1f8 c04a5b40 
 ecfac040
 GPR16: c04a5b40 c04deb80 c04a2120 c04a2d98 c04a5b40 c04d008c ecfac000 
 00029000
 GPR24: c04d c04d1e6c 0001 ec58a700 eceaf390 c04d1e78 c0b23b40 
 ec5df340
 NIP [c0018d20] switch_mmu_context+0x80/0x438
 LR [c039b94c] schedule+0x774/0x7dc
 Call Trace:
 [ecfadd90] [44000484] 0x44000484 (unreliable)
 [ecfadde0] [c039b94c] schedule+0x774/0x7dc
 [ecfade50] [c039cb98] do_nanosleep+0xc8/0x114
 [ecfade80] [c0059bf8] hrtimer_nanosleep+0xd8/0x158
 [ecfadf10] [c0059d48] sys_nanosleep+0xd0/0xd4
 [ecfadf40] [c0013c0c] ret_from_syscall+0x0/0x3c
 --- Exception: c01 at 0xffa6cc4
    LR = 0xffa6cb0
 Instruction dump:
 40a2fff0 4c00012c 2f80 409e0128 813b018c 2f83 39290001 913b018c
 419e0020 8003018c 7c34 5400d97e 0f00 8123018c 3929 9123018c

 Do you have a clue on how to start debugging it ?

 Yes, but that can't be easily summarized here. In short, we have a
 serious problem with the sharing of the MMU context between the Linux
 and Xenomai schedulers in the SMP case on powerpc.

OK, good to know that it is a known issue. If there is a thread with
some thoughts about it, I am interested ;).

 It is happening quite randomly... :).

 Does disabling CONFIG_XENO_HW_UNLOCKED_SWITCH clear this issue?


Well, yes and no. It starts well, but when booting the kernel I get :

Badness at kernel/lockdep.c:2327
NIP: c006e554 LR: c006e53c CTR: 000186a0
REGS: effe9e00 TRAP: 0700   Not tainted  (2.6.35.11)
MSR: 00021000 ME,CE  CR: 24242022  XER: 
TASK = c0508398[0] 'swapper' THREAD: c052e000 CPU: 0
GPR00:  effe9eb0 c0508398 0001 80021000 ea50 0060 0003
GPR08: c0501e80 c053 c051 0001 44242028 100488d8 3ff91200 
GPR16:  3ff85950 3ff85950 3ffb1254 c0a446e0 c0a44700 c053 c0a44704
GPR24: c0537084 0010  c0539838 80029000 c001444c c053 c0508398
NIP [c006e554] trace_hardirqs_on_caller+0x148/0x18c
LR [c006e53c] trace_hardirqs_on_caller+0x130/0x18c
Call Trace:
[effe9eb0] [c006e50c] trace_hardirqs_on_caller+0x100/0x18c (unreliable)
[effe9ed0] [c001444c] restore+0x10/0x64
[effe9f90] [0010] 0x10
[effe9fb0] [c001c568] mpic_unmask_irq+0x84/0xb8
[effe9fd0] [c00816f4] handle_fasteoi_irq+0xe4/0x138
[effe9ff0] [c0013628] call_handle_irq+0x18/0x28
[c052fdc0] [c0004fe0] handle_one_irq+0x94/0x100
[c052fde0] [c000b120] __ipipe_do_IRQ+0x78/0xa8
[c052fe10] [c0085234] __ipipe_sync_stage+0x1b0/0x33c
[c052fe50] [c000a404] __ipipe_handle_irq+0x20c/0x260
[c052fe90] [c000a614] __ipipe_grab_irq+0x4c/0x188
[c052fec0] [c0014a60] __ipipe_ret_from_except+0x0/0xc
[c052ff80] [c0009224] cpu_idle+0x64/0xe0
[c052ffa0] [c000234c] rest_init+0xd0/0xe4
[c052ffc0] [c04c9ae0] start_kernel+0x2b0/0x348
[c052fff0] [c3c4] skpinv+0x2dc/0x318
Instruction dump:
2f80 409eff5c 800205d8 2f80 419eff50 481b8281 2f83 41beff00
3d20c053 8009732c 2f80 40befef0 0fe0 4bfffee8 7fe3fb78 3881

And after starting my applications, it gets really bad :

Badness at arch/powerpc/mm/mmu_context_nohash.c:209
NIP: c0017958 LR: c039f560 CTR: c00347f8
REGS: ed38bdd0 TRAP: 0700   Tainted: G        W    (2.6.35.11)
MSR: 00021000 ME,CE  CR: 24000828  XER: 
TASK = ecb2c260[405] 'ethctl' THREAD: ed38a000 CPU: 1
GPR00: 0001 ed38be80 ecb2c260 ec70f200 ec89a200 0025 c161ccc0 0003
GPR08: ec7fb000 0001 ec89a3c0 c052 0002a106 101110d0 c04e9cc0 c04e9cc0
GPR16: c04e9cc0 c04e9cc0 c04e92e0 c04e9cc0 ed38a000 c051b084 c051b084 c04e9cc0
GPR24: ecb2c4d0 c0519b2c 0001 c04e92e0 c161ccc0 ecb2c260 ec89a200 ec70f200
NIP [c0017958] switch_mmu_context+0x54/0x3d8
LR [c039f560] schedule+0x780/0x824
Call Trace:
[ed38be80] [24000822] 0x24000822 (unreliable)
[ed38bed0] [c039f560] schedule+0x780/0x824
[ed38bf40] [c00133c0] recheck+0x0/0x24
Instruction dump:
7f49b02e 7ca6 54008ffe 0f00 812401c8 2f83 39290001 912401c8
419e0020 800301c8 7c34 5400d97e 0f00 812301c8 3929 912301c8
[ cut here ]
Badness at arch/powerpc/mm/mmu_context_nohash.c:209
NIP: c0017958 LR: c039f560 CTR: c0034a20
REGS: ed38bdd0 TRAP: 0700   Tainted: G        W    (2.6.35.11)
MSR: 00021000 ME,CE  CR: 24000888  XER: 
TASK = ecb2c260[405] 'ethctl' THREAD: ed38a000 CPU: 1
GPR00: 0001 ed38be80 ecb2c260 

Re: [Xenomai-core] [PowerPC]Badness at mmu_context_nohash

2011-04-28 Thread Richard Cochran
On Wed, Apr 27, 2011 at 11:53:32PM +0200, Philippe Gerum wrote:
 Yes, but that can't be easily summarized here. In short, we have a
 serious problem with the sharing of the MMU context between the Linux
 and Xenomai schedulers in the SMP case on powerpc.

BTW, I have been running xenomai 2.5 on ipipe 2.6.36 on p2020ds and
p2020rdb for several weeks now. Mostly, it seems stable. At least it
runs better than in that report. There are a few rough edges, but I
have not had the time to work on them or report them yet.

In any case, I suggest trying 2.6.36.

HTH,

Richard

(Here is the exact version I have been using...)

commit 0f88f18483390d8c4c9ccf7615120e83193fd3c8
Author: Philippe Gerum r...@xenomai.org
Date:   Tue Mar 8 06:52:33 2011 +0100

ipipe-2.6.36-powerpc-2.12-03

commit ec97ca753f417eb56973111573d367395b676333
Author: Philippe Gerum r...@xenomai.org
Date:   Tue Mar 8 06:51:10 2011 +0100

powerpc/ipipe: sanitize IRQ cascading with uic

commit edb402799e8d65639fddcd03c2b7b78615fd4ef3
Author: Philippe Gerum r...@xenomai.org
Date:   Mon Mar 7 17:46:31 2011 +0100

powerpc/ipipe: fix IRQ cascading with fsl_msi chips

___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core