Hello, The problem is that the first interrupt calls the invoke function of the X86 Fault implementation arch/x86/fault.cc. This function saves the PC and the interrupt vector in the micro arch registers and calls the code at the Microcode rom (isa/insts/romutil.py) "longModeInterrupt" by setting the uPC. This routine saves the pc in the stack, and calculates the address to the OS interrupt service routine using the interrupt vector. Then the simulator will forget about this fault.
If we have a second interrupt when the first interrupt sets this registers but hadn't completed the fetch of the first instruction of the longModeInterrupt routine, the O3 CPU will detect an empty rob and will allow this interrupt to proceed. It will overwrite the registers holding the pc and the interrupt vector, that had not the chance of being saved. Therefore the first interruption data will be lost, and when the longModeInterrupt code first instruction arrives, it sees the status (vector) of the second interruption. I did a hack to fix this situation where I completely disable the interrupts during the time window between the set of this registers and the Microcode rom execution. I added a new register that when set to 0x1, every single interruption is ignored at the x86/interrupt.cc checkInterrupts function. This had to be done because setting the IF at the flags registers do not disable all the interruptions. Then I added a new microop at the x86 arch. that sets this register to 0. I modified the routine that does all the above to call this new instruction at the end. This way I made it work, its a bit hacky solution so there might be some other elegant ways to solve this issue. Hope this can be helpful. Best regards. ________________________________________ De: gem5-dev [[email protected]] en nombre de Nilay Vaish via gem5-dev [[email protected]] Enviado: viernes, 28 de noviembre de 2014 16:03 Para: Castillo Villar, Emilio via gem5-dev Asunto: Re: [gem5-dev] Issue with O3 and interrupts Ok, I have not seen this problem, but I got the description below. So what's your suggestion on fixing the problem? Should we add a stack of pending interrupts instead of maintaining one single variable? -- Nilay On Wed, 26 Nov 2014, Castillo Villar, Emilio via gem5-dev wrote: > Good evening, > > I am experiencing a weird issue with the O3CPU, X86 and the interrupt > handling. I am running in FS mode and one simulation just experienced a > weird hang. The simulated machine is doing an spinlock over a value that > an interrupt handler writes. > > After some debug I found that when the APIC sends two interruptions to > the cpu in a very short time window, the first interruption is > completely ignored. It can not even complete a commit of the first > instruction in the service routine before all its values get replaced by > the next interrupt. After this interrupt completes, the execution goes > back to the application code and do not execute the code for the first > interrupt. > > The problem is that the Lapic has the vector of the first interruption in the > ISR register as it gets restored after the second interruption completes. > Therefore, it thinks that the cpu is currently processing that interruption, > though the cpu went back to execute application code and will never clear > this ISR register. > > The Lapic uses this ISR value to filter incoming interruptions and in several > cases, it does not forward those to the cpu, leading to unattended > interruptions and hangs. > > I have seen this behavior in the kernels' native_flush_tlb_others function > when a page fault happens. The core in charge of executing it, sends an > interrupt to all the other cores in the system and it does a loop checking > that every cores receive the interruption. When each core receives the > interruption, they just execute the associated handler and perform a write > to a variable, notifying the sender that the interruption was processed and > the tlb was flushed. > > The problem is that one of the cores is ignoring this interruption, which has > a vector value of 0xf0. I found that this core lapic has a value of 0xf1 in > the ISR, filtering every lower vector. > s > This 0xf1 vector value was set by an interruption that never got to execute > because of the problem explained before, hence the interruption carrying the > 0xf0 vector value will never be executed and the native_flush_tlb_others > function will not complete. > > I just took a trace of the moment when the 0xf1 interruption gets dropped. > The flags used where Exec, Commit, Faults: > > system.cpu00.interrupts: Interrupt 0xf1 sent to core. > 7175754213000: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf1: #INTR > 7175754213000: system.cpu00.interrupts: NEW IRR 0 NEW ISR f1. > > Now the interrupt 0xf3 gets to execute and drops all the first interruption. > > 7175754217000: system.cpu00.interrupts: Got Trigger Interrupt message with > vector 0xf3. > 7175754217000: system.cpu00.interrupts: Interrupt is an Fixed. > 7175754220500: system.cpu00.commit: Interrupt detected. > 7175754220500: system.cpu00.interrupts: Interrupt 0xf3 sent to core. > 7175754220500: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf3: #INTR > 7175754220500: system.cpu00.interrupts: NEW IRR 0 NEW ISR f3. > > It can be seen how for both interrupts the RIP is the same. > > The first committed instruction after all this sequence of events is > > 7175754228500: system.cpu00 T0 : @handle_mm_fault+992.32768 : Microcode_ROM > : slli t4, t1, 0x4 : IntAlu : D=0x0000000000000f30 > > which indeed is from the 0xf3 interrupt. > The cpu executes all the handler and then writes to the APIC EOI register > > 7175754418500: system.cpu00.interrupts: Writing Local APIC register 5 at > offset 0xb0 as 0. > 7175754418500: system.cpu00.interrupts: WRITING TO EOI NEW ISRV IS 0xf1 > > Here the APIC believes it is servicing the 0xf1 interrupt. However the cpu > goes back to the code it was executing right before the 0xf1 interrupt, and > never services it. > > I was wondering if someone has found this issue before. > > Thanks a lot for your time. > > Best regards, > > --------------------------------------- > > Emilio Castillo > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
