I feel like there should be a simple solution to this problem, but I haven't had the time to really walk through your explanation and understand it yet.
Gabe On Fri, Nov 28, 2014 at 9:30 AM, Castillo Villar, Emilio via gem5-dev < [email protected]> wrote: > Hello, > > The problem is that the first interrupt calls the invoke function of the > X86 Fault implementation arch/x86/fault.cc. > This function saves the PC and the interrupt vector in the micro arch > registers and calls the code at the Microcode rom (isa/insts/romutil.py) > "longModeInterrupt" by setting the uPC. This routine saves the pc in the > stack, and calculates the address to the OS interrupt service routine using > the interrupt vector. Then the simulator will forget about this fault. > > If we have a second interrupt when the first interrupt sets this registers > but hadn't completed the fetch of the first instruction of the > longModeInterrupt routine, the O3 CPU will detect an empty rob and will > allow this interrupt to proceed. It will overwrite the registers holding > the pc and the interrupt vector, that had not the chance of being saved. > Therefore the first interruption data will be lost, and when the > longModeInterrupt code first instruction arrives, it sees the status > (vector) of the second interruption. > > I did a hack to fix this situation where I completely disable the > interrupts during the time window between the set of this registers and the > Microcode rom execution. > > I added a new register that when set to 0x1, every single interruption is > ignored at the x86/interrupt.cc checkInterrupts function. This had to be > done because setting the IF at the flags registers do not disable all the > interruptions. Then I added a new microop at the x86 arch. that sets this > register to 0. I modified the routine that does all the above to call this > new instruction at the end. This way I made it work, its a bit hacky > solution so there might be some other elegant ways to solve this issue. > > Hope this can be helpful. > > Best regards. > ________________________________________ > De: gem5-dev [[email protected]] en nombre de Nilay Vaish via > gem5-dev [[email protected]] > Enviado: viernes, 28 de noviembre de 2014 16:03 > Para: Castillo Villar, Emilio via gem5-dev > Asunto: Re: [gem5-dev] Issue with O3 and interrupts > > Ok, I have not seen this problem, but I got the description below. So > what's your suggestion on fixing the problem? Should we add a stack of > pending interrupts instead of maintaining one single variable? > > -- > Nilay > > On Wed, 26 Nov 2014, Castillo Villar, Emilio via gem5-dev wrote: > > > Good evening, > > > > I am experiencing a weird issue with the O3CPU, X86 and the interrupt > > handling. I am running in FS mode and one simulation just experienced a > > weird hang. The simulated machine is doing an spinlock over a value that > > an interrupt handler writes. > > > > After some debug I found that when the APIC sends two interruptions to > > the cpu in a very short time window, the first interruption is > > completely ignored. It can not even complete a commit of the first > > instruction in the service routine before all its values get replaced by > > the next interrupt. After this interrupt completes, the execution goes > > back to the application code and do not execute the code for the first > > interrupt. > > > > The problem is that the Lapic has the vector of the first interruption > in the ISR register as it gets restored after the second interruption > completes. Therefore, it thinks that the cpu is currently processing that > interruption, though the cpu went back to execute application code and will > never clear this ISR register. > > > > The Lapic uses this ISR value to filter incoming interruptions and in > several cases, it does not forward those to the cpu, leading to unattended > interruptions and hangs. > > > > I have seen this behavior in the kernels' native_flush_tlb_others > function when a page fault happens. The core in charge of executing it, > sends an interrupt to all the other cores in the system and it does a loop > checking that every cores receive the interruption. When each core receives > the interruption, they just execute the associated handler and perform a > write to a variable, notifying the sender that the interruption was > processed and the tlb was flushed. > > > > The problem is that one of the cores is ignoring this interruption, > which has a vector value of 0xf0. I found that this core lapic has a value > of 0xf1 in the ISR, filtering every lower vector. > > s > > This 0xf1 vector value was set by an interruption that never got to > execute because of the problem explained before, hence the interruption > carrying the 0xf0 vector value will never be executed and the > native_flush_tlb_others function will not complete. > > > > I just took a trace of the moment when the 0xf1 interruption gets > dropped. The flags used where Exec, Commit, Faults: > > > > system.cpu00.interrupts: Interrupt 0xf1 sent to core. > > 7175754213000: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf1: > #INTR > > 7175754213000: system.cpu00.interrupts: NEW IRR 0 NEW ISR f1. > > > > Now the interrupt 0xf3 gets to execute and drops all the first > interruption. > > > > 7175754217000: system.cpu00.interrupts: Got Trigger Interrupt message > with vector 0xf3. > > 7175754217000: system.cpu00.interrupts: Interrupt is an Fixed. > > 7175754220500: system.cpu00.commit: Interrupt detected. > > 7175754220500: system.cpu00.interrupts: Interrupt 0xf3 sent to core. > > 7175754220500: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf3: > #INTR > > 7175754220500: system.cpu00.interrupts: NEW IRR 0 NEW ISR f3. > > > > It can be seen how for both interrupts the RIP is the same. > > > > The first committed instruction after all this sequence of events is > > > > 7175754228500: system.cpu00 T0 : @handle_mm_fault+992.32768 : > Microcode_ROM : slli t4, t1, 0x4 : IntAlu : D=0x0000000000000f30 > > > > which indeed is from the 0xf3 interrupt. > > The cpu executes all the handler and then writes to the APIC EOI register > > > > 7175754418500: system.cpu00.interrupts: Writing Local APIC register 5 at > offset 0xb0 as 0. > > 7175754418500: system.cpu00.interrupts: WRITING TO EOI NEW ISRV IS 0xf1 > > > > Here the APIC believes it is servicing the 0xf1 interrupt. However the > cpu goes back to the code it was executing right before the 0xf1 interrupt, > and never services it. > > > > I was wondering if someone has found this issue before. > > > > Thanks a lot for your time. > > > > Best regards, > > > > --------------------------------------- > > > > Emilio Castillo > > _______________________________________________ > > gem5-dev mailing list > > [email protected] > > http://m5sim.org/mailman/listinfo/gem5-dev > > > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
