I forgot to mention: my board is zcu102, 4 cores A53
-----Original Message-----
From: Devel <devel-bounces@sel4.systems> On Behalf Of Leonid Meyerovich
Sent: Thursday, October 3, 2019 2:40 PM
To: Shen, Yanyan (Data61, Kensington NSW) <yanyan.s...@data61.csiro.au>;
devel@sel4.systems
Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running
Hi Everybody,
I am not sure this is related to my previous question/e-mail, but I have
collected some more data about an interrupts in my system.
I observe a lot of interrupts with ID=0, and later with ID=1 (see below) Both
IDs 0 and 1 are initialized as inactive in while booting all cores
init_irqs(cap_t root_cnode_cap)
{
irq_t i;
for (i = 0; i <= maxIRQ; i++) {
setIRQState(IRQInactive, i);
}
COULD SOMEBODY EXPLAIN WHAT THESE INTERRUPTS MEANS AND WHY I AM GETTING THEM.
Thank you,
Leonid
I have printed interrupt statistic from inside of handleInterrupt(irq_t irq)
function.
It has a following structure:
void
handleInterrupt(irq_t irq)
{
switch (intStateIRQTable[irq]) {
case IRQSignal: {
...
}
case IRQTimer:
timerTick();
resetTimer();
if ((++tickCount % 200) == 0)
{
printInt(0);
printf("- - - \n");
printInt(1);
printf("- - - \n");
printInt(2);
printf("- - - \n");
printInt(3);
printf("- - - - - - - \n");
cleanIntStat();
}
break;
#ifdef ENABLE_SMP_SUPPORT
case IRQIPI:
handleIPI(irq, true);
break;
#endif /* ENABLE_SMP_SUPPORT */
case IRQReserved:
#ifdef CONFIG_IRQ_REPORTING
//printf("Received reserved IRQ: %d", (int)irq); #endif
handleReservedIRQ(irq);
break;
case IRQInactive:
/*
* This case shouldn't happen anyway unless the hardware or
* platform code is broken. Hopefully masking it again should make
* the interrupt go away.
*/
maskInterrupt(true, irq);
#ifdef CONFIG_IRQ_REPORTING
printf("Received disabled IRQ: %d\n", (int)irq); #endif
break;
default:
/* No corresponding haskell error */
fail("Invalid IRQ state");
}
ackInterrupt(irq);
}
printInt function prints all collected interrupts for one core in the following
format:
---irq stat: cpu=0 (2) 26_50_50 0_258_258 the number inside
parentheses is the number of interrupt for particular time interval - 1 sec,
ignore it
Every interrupt statistic: ID_(enter counter)_(exit_counter) each counter
is for 1 sec interval
---irq stat: cpu=0 (2) 26_50_50 0_258_258
- - -
---irq stat: cpu=1 (2) 26_50_50 0_258_258
- - -
---irq stat: cpu=2 (2) 26_50_50 0_258_258
- - -
---irq stat: cpu=3 (3) 26_50_50 65_1_1 59_1_1
- - - - - - -
---irq stat: cpu=0 (2) 26_50_50 0_257_257
- - -
---irq stat: cpu=1 (2) 26_50_50 0_280_280
- - -
---irq stat: cpu=2 (2) 26_50_50 0_280_280
.................................... end of the first excerpt
- - - - - - -
---irq stat: cpu=0 (1) 26_50_50
- - -
---irq stat: cpu=1 (1) 26_50_50
- - -
---irq stat: cpu=2 (1) 26_50_50
- - -
---irq stat: cpu=3 (1) 26_50_50
- - - - - - -
.................................... end of the another excerpt
- - - - - - -
---irq stat: cpu=0 (1) 26_50_51
- - -
---irq stat: cpu=1 (2) 26_50_50 0_4382_4382
- - -
---irq stat: cpu=2 (2) 26_50_49 0_4382_4382
- - -
---irq stat: cpu=3 (2) 26_50_50 0_4382_4382
- - - - - - -
.................................... end of the another excerpt
- - - - - - -
---irq stat: cpu=0 (2) 26_50_51 1_317_317
- - -
---irq stat: cpu=1 (4) 26_50_50 25_34_34 1_132_132 27_16_16
- - -
---irq stat: cpu=2 (4) 26_50_50 27_17_17 25_32_32 1_130_130
- - -
---irq stat: cpu=3 (2) 26_50_49 1_28_28
- - - - - - -
.................................... end of the another excerpt
-----Original Message-----
From: Shen, Yanyan (Data61, Kensington NSW) <yanyan.s...@data61.csiro.au>
Sent: Thursday, October 3, 2019 2:44 AM
To: Leonid Meyerovich <lmeyerov...@i-a-i.com>; devel@sel4.systems
Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running
Hi Leonid,
Could you provide a bit more about your software configuration? For instance,
do you have multiple VMs running on dedicated hardware cores?
How are the VM and processes configured?
Also, you mean there were no interrupts at all on all the four cores?
Regards,
Yanyan
On Wed, 2019-10-02 at 16:01 +0000, Leonid Meyerovich wrote:
> Hello,
>
> We are running seL4 microkernel on 4 cores Zynq UltraScale+ (zcu102
> board). The implementation includes multiple processes, hypervisor and
> virtual machine running on dedicated core. After several hours running
> (it could be 2 or even 8 hours) the whole microkernel locks up. After
> some investigation I have found that no interrupts generated anymore -
> at least there is no interrupts coming to ISR.
> Inside ISR I have monitored PL2 Physical Timer Control register, which
> feeds a scheduler and didn't find any problems - it stays enabled and
> not masked.
>
> I will appreciate any idea/direction for approaching this problem.
>
> Thank you,
>
> Leonid
>
>
>
>
> This message and all attachments are PRIVATE, and contain information
> that is PROPRIETARY to Intelligent Automation, Inc. You are not
> authorized to transmit or otherwise disclose this message or any
> attachments to any third party whatsoever without the express written
> consent of Intelligent Automation, Inc. If you received this message
> in error or you are not willing to view this message or any
> attachments on a confidential basis, please immediately delete this
> email and any attachments and notify Intelligent Automation, Inc.
> _______________________________________________
> Devel mailing list
> Devel@sel4.systems
> https://sel4.systems/lists/listinfo/devel
________________________________
This message and all attachments are PRIVATE, and contain information that is
PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit
or otherwise disclose this message or any attachments to any third party
whatsoever without the express written consent of Intelligent Automation, Inc.
If you received this message in error or you are not willing to view this
message or any attachments on a confidential basis, please immediately delete
this email and any attachments and notify Intelligent Automation, Inc.
_______________________________________________
Devel mailing list
Devel@sel4.systems
https://sel4.systems/lists/listinfo/devel
________________________________
This message and all attachments are PRIVATE, and contain information that is
PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit
or otherwise disclose this message or any attachments to any third party
whatsoever without the express written consent of Intelligent Automation, Inc.
If you received this message in error or you are not willing to view this
message or any attachments on a confidential basis, please immediately delete
this email and any attachments and notify Intelligent Automation, Inc.
_______________________________________________
Devel mailing list
Devel@sel4.systems
https://sel4.systems/lists/listinfo/devel
