RT Kernel crash in SMP mode on Marvell board
Hello All,
I'm trying to bring up a Marvell board [db644xx] with dual CPU[7447A]
with kernel 2.6.33.5-rt23[Ingo's RT patch]. For conveniance I have
registered IPI interrupt with IRQF_NODELAY flag, all other interrupts are
threaded. I'm able to boot up the system fine, however after few hours I see
random kernel crash in Marvell's PIC module. I understand that the RT kernel
doesn't like the way spinlock is used, I tried to convert to raw_spin_lock,
but the system would get frozen. I can make the system stable by just not
using any locks, but thats not an ideal solution. Can someone please suggest
a better locking mechanism? Crash dump I have given below:
NIP: LR: 70030258 CTR:
REGS: deeebcf0 TRAP: 0400 Tainted: P(2.6.33.5-rt23)
MSR: 40001032 CR: 24004024 XER: 2004
TASK = df915240[788] 'irq/74-ide0' THREAD: deeea000 CPU: 1
GPR00: 70030390 deeebda0 df915240 72e17ac0 703683f8 0001
645d28e8
GPR08: 0005 46f0b20f 00fe4600
007fff00
GPR16: 0400 003d4ba2 007ffeb0 0080 00ff690c 0340 00f7eba2
deeebf6c
GPR24: 0001 deeea000 72e17ac0 703683f8
703683f8
NIP [] (null)
LR [70030258] enqueue_task+0x3c/0x58
Call Trace:
[deeebda0] [001e] 0x1e (unreliable)
[deeebdb0] [70030390] activate_task+0x40/0x60
[deeebdc0] [70036850] try_to_wake_up+0x248/0x334
[deeebe00] [70069380] wakeup_next_waiter+0x148/0x14c
[deeebe20] [7029b1d4] rt_spin_lock_slowunlock+0x60/0x84
[deeebe30] [7001a168] marvell_disable_IPI_irq+0xfc/0x110
[deeebe50] [70073934] handle_level_irq+0x40/0x15c
[deeebe70] [70006418] do_IRQ+0xc8/0xf4
[deeebe90] [7001276c] ret_from_except+0x0/0x14
--- Exception: 501 at schedule+0x30/0x50
LR = schedule+0x24/0x50
[deeebf60] [700717bc] irq_thread+0x98/0x230
[deeebfa0] [7005686c] kthread+0x78/0x7c
[deeebff0] [70011ef4] kernel_thread+0x4c/0x68
Instruction dump:
Kernel panic - not syncing: Fatal exception in interrupt
Call Trace:
[deeebc20
-
IRQ CONFIGs:
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
# CONFIG_HAVE_SETUP_PER_CPU_AREA is not set
# CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK is not set
CONFIG_IRQ_PER_CPU=y
CONFIG_NR_IRQS=512
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
--
Thanks,
Manik
Think twice about a tree before you take a printout
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Query regarding 2.6.335 RT[Ingo's] and Non-RT performance
Hello All,
I created a very simple program which has higher priority than normal
tasks and runs a tight loop. Under same test environment I ran this
program on both non-rt and rt 2.6.33.5 kernel. To my suprise I see that
performance of non-RT kernel is better than RT. non-RT kernel took 3 sec and
366156 usec while RT kernel took about 3 sec and 418011 usec.Can someone
please explain why the performance of non-rt kernel is better than rt
kernel? From the face of the test result, I feel RT has more overhead,Is
there any configuration that I could do to bring down the overhead?
Processor:
processor : 0
cpu : 7448
clock : 996.00MHz
revision: 2.2 (pvr 8004 0202)
bogomips: 83.10
processor : 1
cpu : 7448
clock : 996.00MHz
revision: 2.2 (pvr 8004 0202)
bogomips: 83.10
CFS optimization:
--
# cat /proc/sys/kernel/sched_rt_runtime_us
100
# cat /proc/sys/kernel/sched_rt_period_us
100
# cat /proc/sys/kernel/sched_compat_yield
1
Test Program:
-
main()
{
int sched_rr_min,sched_rr_max;
struct sched_param scheduling_parameters;
struct timeval tv,late_tv;
suseconds_t usec_diff,avg_usec = 0;
time_t sec_diff, avg_sec = 0;
int i;
long count = 1;
sched_rr_min = sched_get_priority_min(SCHED_RR);
sched_rr_max = sched_get_priority_max(SCHED_RR);
scheduling_parameters.sched_priority = sched_rr_min+4;
sched_setscheduler(0, SCHED_RR, &scheduling_parameters);// Run the
process with the given priority
for(i = 0 ; i < 150 ; i++) {
gettimeofday(&tv, NULL);
while(count > 0){
//printf(".");
count++;
}
gettimeofday(&late_tv, NULL);
count = 1;
sec_diff = (late_tv.tv_sec - tv.tv_sec);
avg_sec += sec_diff;
usec_diff = ( (late_tv.tv_usec > tv.tv_usec) ? (late_tv.tv_usec -
tv.tv_usec) : ( tv.tv_usec - late_tv.tv_usec));
avg_usec += usec_diff;
printf("Iteration #%d sec %x usec %x\n",i,(sec_diff),(usec_diff));
}
printf("Average of #%d sec %x usec %x\n",i,(avg_sec/i),(avg_usec)/i);
}
Partial Result of non-rt kernel:
---
Iteration #140 sec 3 usec 3aef8
Iteration #141 sec 3 usec 3aefe
Iteration #142 sec 3 usec 3aee4
*Iteration #143 sec 4 usec b935b [Why there is this periodic bump ??]
[Scheduler at work??]*
Iteration #144 sec 3 usec 3aef2
Iteration #145 sec 3 usec 3aef0
Iteration #146 sec 3 usec 3aef4
*Iteration #147 sec 4 usec b934b*
Iteration #148 sec 3 usec 3aeed
Iteration #149 sec 3 usec 3aef9
Partial Result of rt kernel:
---
Iteration #135 sec 3 usec 47328
*Iteration #136 sec 4 usec ac4fd
*Iteration #137 sec 3 usec 48b0b
Iteration #138 sec 3 usec 4738c
Iteration #139 sec 4 usec ac4d5
Iteration #140 sec 3 usec 483cb
Iteration #141 sec 3 usec 48500
*Iteration #142 sec 4 usec acc49
*Iteration #143 sec 3 usec 47c1f
Iteration #144 sec 3 usec 478c2
Iteration #145 sec 3 usec 47e48
Iteration #146 sec 4 usec ac9b5
Iteration #147 sec 3 usec 48de4
Iteration #148 sec 3 usec 46fbe
Iteration #149 sec 4 usec ac52e
Average of #150 sec 3 usec 660db
Thanks,
Mani
--
Thanks,
Manik
Think twice about a tree before you take a printout
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: Linuxppc-dev Digest, Vol 72, Issue 63
> Date: Thu, 12 Aug 2010 13:53:51 -0400 > From: Jeff Angielski > To: linuxppc-dev@lists.ozlabs.org > Subject: Re: Query regarding 2.6.335 RT[Ingo's] and Non-RT performance > Message-ID: <4c64352f.4090...@theptrgroup.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 08/11/2010 06:18 PM, Manikandan Ramachandran wrote: > > Hello All, > > I created a very simple program which has higher priority than > > normal tasks and runs a tight loop. Under same test environment I ran > > this program on both non-rt and rt 2.6.33.5 kernel. To my suprise I see > > that performance of non-RT kernel is better than RT. non-RT kernel took > > 3 sec and 366156 usec while RT kernel took about 3 sec and 418011 > > usec.Can someone please explain why the performance of non-rt kernel is > > better than rt kernel? From the face of the test result, I feel RT has > > more overhead,Is there any configuration that I could do to bring down > > the overhead? > > Your "surprise" is due to your definition of "performance". > > The purpose of the -rt kernels is to reduce the kernel latency. This is > important for servicing hardware. Normal users find the -rt useful for > audio/video applications. Engineering and scientific users find the -rt > beneficially for servicing hardware like sensors or control systems. > > If you are just trying to run calculations as fast as you can in user > space, you'd be better off using the non-rt variants. > > > -- > Jeff Angielski > The PTR Group > www.theptrgroup.com > > > Thanks for your response. > > On one hand I hear that RT-kernel is meant for reducing kernel latency on > other hand I see that there is RT-kernel overhead. So what really RT-kernel > brings to system performance? > > Actually I see that latency for higher priority is more or less same for > non-rt system. > > One more thing, since irqs being threaded in RT, and with CFS scheduler in > 2.6.33, wouldn't we bring down system performance as CFS is O(log(n)) in > nature? > -- > Thanks, > Manik > ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: Query regarding 2.6.335 RT[Ingo's] and Non-RT performance
> -- > > Date: Thu, 12 Aug 2010 13:53:51 -0400 > > From: Jeff Angielski > > To: linuxppc-dev@lists.ozlabs.org > > Subject: Re: Query regarding 2.6.335 RT[Ingo's] and Non-RT performance > > Message-ID: <4c64352f.4090...@theptrgroup.com> > > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > > On 08/11/2010 06:18 PM, Manikandan Ramachandran wrote: > > > Hello All, > > > I created a very simple program which has higher priority than > > > normal tasks and runs a tight loop. Under same test environment I ran > > > this program on both non-rt and rt 2.6.33.5 kernel. To my suprise I see > > > that performance of non-RT kernel is better than RT. non-RT kernel took > > > 3 sec and 366156 usec while RT kernel took about 3 sec and 418011 > > > usec.Can someone please explain why the performance of non-rt kernel is > > > better than rt kernel? From the face of the test result, I feel RT has > > > more overhead,Is there any configuration that I could do to bring down > > > the overhead? > > > > Your "surprise" is due to your definition of "performance". > > > > The purpose of the -rt kernels is to reduce the kernel latency. This is > > important for servicing hardware. Normal users find the -rt useful for > > audio/video applications. Engineering and scientific users find the -rt > > beneficially for servicing hardware like sensors or control systems. > > > > If you are just trying to run calculations as fast as you can in user > > space, you'd be better off using the non-rt variants. > > > > > > -- > > Jeff Angielski > > The PTR Group > > www.theptrgroup.com Thanks for your response. On one hand I hear that RT-kernel is meant for reducing kernel latency on other hand I see that there is RT-kernel overhead. So what really RT-kernel brings to system performance? Actually I see that latency for higher priority is more or less same for non-rt system. One more thing, since irqs being threaded in RT, and with CFS scheduler in 2.6.33, wouldn't we bring down system performance as CFS is O(log(n)) in nature? -- Thanks, Manik ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
