Ok, I have not seen this problem, but I got the description below. So
what's your suggestion on fixing the problem? Should we add a stack of
pending interrupts instead of maintaining one single variable?
--
Nilay
On Wed, 26 Nov 2014, Castillo Villar, Emilio via gem5-dev wrote:
Good evening,
I am experiencing a weird issue with the O3CPU, X86 and the interrupt
handling. I am running in FS mode and one simulation just experienced a
weird hang. The simulated machine is doing an spinlock over a value that
an interrupt handler writes.
After some debug I found that when the APIC sends two interruptions to
the cpu in a very short time window, the first interruption is
completely ignored. It can not even complete a commit of the first
instruction in the service routine before all its values get replaced by
the next interrupt. After this interrupt completes, the execution goes
back to the application code and do not execute the code for the first
interrupt.
The problem is that the Lapic has the vector of the first interruption in the
ISR register as it gets restored after the second interruption completes.
Therefore, it thinks that the cpu is currently processing that interruption,
though the cpu went back to execute application code and will never clear this
ISR register.
The Lapic uses this ISR value to filter incoming interruptions and in several
cases, it does not forward those to the cpu, leading to unattended
interruptions and hangs.
I have seen this behavior in the kernels' native_flush_tlb_others function when
a page fault happens. The core in charge of executing it, sends an interrupt to
all the other cores in the system and it does a loop checking that every cores
receive the interruption. When each core receives the interruption, they just
execute the associated handler and perform a write to a variable, notifying
the sender that the interruption was processed and the tlb was flushed.
The problem is that one of the cores is ignoring this interruption, which has a
vector value of 0xf0. I found that this core lapic has a value of 0xf1 in the
ISR, filtering every lower vector.
s
This 0xf1 vector value was set by an interruption that never got to execute
because of the problem explained before, hence the interruption carrying the
0xf0 vector value will never be executed and the native_flush_tlb_others
function will not complete.
I just took a trace of the moment when the 0xf1 interruption gets dropped. The
flags used where Exec, Commit, Faults:
system.cpu00.interrupts: Interrupt 0xf1 sent to core.
7175754213000: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf1: #INTR
7175754213000: system.cpu00.interrupts: NEW IRR 0 NEW ISR f1.
Now the interrupt 0xf3 gets to execute and drops all the first interruption.
7175754217000: system.cpu00.interrupts: Got Trigger Interrupt message with
vector 0xf3.
7175754217000: system.cpu00.interrupts: Interrupt is an Fixed.
7175754220500: system.cpu00.commit: Interrupt detected.
7175754220500: system.cpu00.interrupts: Interrupt 0xf3 sent to core.
7175754220500: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf3: #INTR
7175754220500: system.cpu00.interrupts: NEW IRR 0 NEW ISR f3.
It can be seen how for both interrupts the RIP is the same.
The first committed instruction after all this sequence of events is
7175754228500: system.cpu00 T0 : @handle_mm_fault+992.32768 : Microcode_ROM :
slli t4, t1, 0x4 : IntAlu : D=0x0000000000000f30
which indeed is from the 0xf3 interrupt.
The cpu executes all the handler and then writes to the APIC EOI register
7175754418500: system.cpu00.interrupts: Writing Local APIC register 5 at offset
0xb0 as 0.
7175754418500: system.cpu00.interrupts: WRITING TO EOI NEW ISRV IS 0xf1
Here the APIC believes it is servicing the 0xf1 interrupt. However the cpu goes
back to the code it was executing right before the 0xf1 interrupt, and never
services it.
I was wondering if someone has found this issue before.
Thanks a lot for your time.
Best regards,
---------------------------------------
Emilio Castillo
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
