Dear all, Sorry for my poor explanations, I have tried to give as many details as possible but reading my previous emails they are somehow confusing, and my english skills are not yet very good. I will try to expose the problem in a few lines with references to the actual code. This might be better for the sake of understanding.
When an interruption arrives, The commit stage stops the fetch of new instructions and waits till the rob is empty. Once the rob is empty the commit stage can process the interruption calling the interrupt->invoke method. through the cpu->processInterrupts(cpu->getInterrupts()); call at cpu/o3/commit_impl.hh The invoke call will end up executing the code at X86FaultBase::invoke(ThreadContext * tc, StaticInstPtr inst) (arch/x86/faults.cc) This call does 2 things, it stores the interrupt vector and the actual pc in two registers: tc->setIntReg(INTREG_MICRO(1), vector); tc->setIntReg(INTREG_MICRO(7), pc); And calls the code at arch/x86/isa/insts/romutil.py by setting the uPC: entry = extern_label_longModeInterrupt; ... pcState.upc(romMicroPC(entry)); pcState.nupc(romMicroPC(entry) + 1); tc->pcState(pcState); The code at that file just stores some of the processor status in the stack and jumps to the OS interrupt handler using the interrupt vector stored at INTREG_MICRO(1) register. Now the simulator is done with the interrupt and it gets discarded. Being its vector only accesible at INTREG_MICRO(1). Now there is a very narrow time frame where we have set the register there to hold the vector as a parameter to the longModeInterrupt routine, but we hadn't complete a fetch & decode of the first instruction, so the ROB remains empty. If another interruption arrives at that moment, the commit stage will detect an empty rob and will also do the invoke of the interrupt. Overwriting that registers and losing the first interrupt vector forever. Then the first interruption of that routine gets executed (Both interrupts jump to the same code, so the uPC remains unchanged) and finds the vector of the new interruption instead of the value stored when it was called. What I did to solve the issue was to inhibit all the interruptions during that time frame. It was the fastest thing that I could come up with ... However, this is a extremely weird race that is extremely unlikely to happen. I just had bad luck :). Hope this info is useful and now is better explained than before. Thank you all for your hard work on this wonderful tool!! Best regards, Emilio ________________________________________ De: gem5-dev [[email protected]] en nombre de Gabe Black via gem5-dev [[email protected]] Enviado: viernes, 28 de noviembre de 2014 21:33 Para: gem5 Developer List Asunto: Re: [gem5-dev] Issue with O3 and interrupts I feel like there should be a simple solution to this problem, but I haven't had the time to really walk through your explanation and understand it yet. Gabe On Fri, Nov 28, 2014 at 9:30 AM, Castillo Villar, Emilio via gem5-dev < [email protected]> wrote: > Hello, > > The problem is that the first interrupt calls the invoke function of the > X86 Fault implementation arch/x86/fault.cc. > This function saves the PC and the interrupt vector in the micro arch > registers and calls the code at the Microcode rom (isa/insts/romutil.py) > "longModeInterrupt" by setting the uPC. This routine saves the pc in the > stack, and calculates the address to the OS interrupt service routine using > the interrupt vector. Then the simulator will forget about this fault. > > If we have a second interrupt when the first interrupt sets this registers > but hadn't completed the fetch of the first instruction of the > longModeInterrupt routine, the O3 CPU will detect an empty rob and will > allow this interrupt to proceed. It will overwrite the registers holding > the pc and the interrupt vector, that had not the chance of being saved. > Therefore the first interruption data will be lost, and when the > longModeInterrupt code first instruction arrives, it sees the status > (vector) of the second interruption. > > I did a hack to fix this situation where I completely disable the > interrupts during the time window between the set of this registers and the > Microcode rom execution. > > I added a new register that when set to 0x1, every single interruption is > ignored at the x86/interrupt.cc checkInterrupts function. This had to be > done because setting the IF at the flags registers do not disable all the > interruptions. Then I added a new microop at the x86 arch. that sets this > register to 0. I modified the routine that does all the above to call this > new instruction at the end. This way I made it work, its a bit hacky > solution so there might be some other elegant ways to solve this issue. > > Hope this can be helpful. > > Best regards. > ________________________________________ > De: gem5-dev [[email protected]] en nombre de Nilay Vaish via > gem5-dev [[email protected]] > Enviado: viernes, 28 de noviembre de 2014 16:03 > Para: Castillo Villar, Emilio via gem5-dev > Asunto: Re: [gem5-dev] Issue with O3 and interrupts > > Ok, I have not seen this problem, but I got the description below. So > what's your suggestion on fixing the problem? Should we add a stack of > pending interrupts instead of maintaining one single variable? > > -- > Nilay > > On Wed, 26 Nov 2014, Castillo Villar, Emilio via gem5-dev wrote: > > > Good evening, > > > > I am experiencing a weird issue with the O3CPU, X86 and the interrupt > > handling. I am running in FS mode and one simulation just experienced a > > weird hang. The simulated machine is doing an spinlock over a value that > > an interrupt handler writes. > > > > After some debug I found that when the APIC sends two interruptions to > > the cpu in a very short time window, the first interruption is > > completely ignored. It can not even complete a commit of the first > > instruction in the service routine before all its values get replaced by > > the next interrupt. After this interrupt completes, the execution goes > > back to the application code and do not execute the code for the first > > interrupt. > > > > The problem is that the Lapic has the vector of the first interruption > in the ISR register as it gets restored after the second interruption > completes. Therefore, it thinks that the cpu is currently processing that > interruption, though the cpu went back to execute application code and will > never clear this ISR register. > > > > The Lapic uses this ISR value to filter incoming interruptions and in > several cases, it does not forward those to the cpu, leading to unattended > interruptions and hangs. > > > > I have seen this behavior in the kernels' native_flush_tlb_others > function when a page fault happens. The core in charge of executing it, > sends an interrupt to all the other cores in the system and it does a loop > checking that every cores receive the interruption. When each core receives > the interruption, they just execute the associated handler and perform a > write to a variable, notifying the sender that the interruption was > processed and the tlb was flushed. > > > > The problem is that one of the cores is ignoring this interruption, > which has a vector value of 0xf0. I found that this core lapic has a value > of 0xf1 in the ISR, filtering every lower vector. > > s > > This 0xf1 vector value was set by an interruption that never got to > execute because of the problem explained before, hence the interruption > carrying the 0xf0 vector value will never be executed and the > native_flush_tlb_others function will not complete. > > > > I just took a trace of the moment when the 0xf1 interruption gets > dropped. The flags used where Exec, Commit, Faults: > > > > system.cpu00.interrupts: Interrupt 0xf1 sent to core. > > 7175754213000: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf1: > #INTR > > 7175754213000: system.cpu00.interrupts: NEW IRR 0 NEW ISR f1. > > > > Now the interrupt 0xf3 gets to execute and drops all the first > interruption. > > > > 7175754217000: system.cpu00.interrupts: Got Trigger Interrupt message > with vector 0xf3. > > 7175754217000: system.cpu00.interrupts: Interrupt is an Fixed. > > 7175754220500: system.cpu00.commit: Interrupt detected. > > 7175754220500: system.cpu00.interrupts: Interrupt 0xf3 sent to core. > > 7175754220500: External Interrupt: RIP 0xffffffff8027a0a0: vector 0xf3: > #INTR > > 7175754220500: system.cpu00.interrupts: NEW IRR 0 NEW ISR f3. > > > > It can be seen how for both interrupts the RIP is the same. > > > > The first committed instruction after all this sequence of events is > > > > 7175754228500: system.cpu00 T0 : @handle_mm_fault+992.32768 : > Microcode_ROM : slli t4, t1, 0x4 : IntAlu : D=0x0000000000000f30 > > > > which indeed is from the 0xf3 interrupt. > > The cpu executes all the handler and then writes to the APIC EOI register > > > > 7175754418500: system.cpu00.interrupts: Writing Local APIC register 5 at > offset 0xb0 as 0. > > 7175754418500: system.cpu00.interrupts: WRITING TO EOI NEW ISRV IS 0xf1 > > > > Here the APIC believes it is servicing the 0xf1 interrupt. However the > cpu goes back to the code it was executing right before the 0xf1 interrupt, > and never services it. > > > > I was wondering if someone has found this issue before. > > > > Thanks a lot for your time. > > > > Best regards, > > > > --------------------------------------- > > > > Emilio Castillo > > _______________________________________________ > > gem5-dev mailing list > > [email protected] > > http://m5sim.org/mailman/listinfo/gem5-dev > > > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
