I think it is necessary to place AP into one safe state: (hlt-loop, no page table required, > 1MB reserved space in non-SMM), just like we have done in MpInitExitBootServicesCallback() on normal boot path.
From: Yao, Jiewen Sent: Thursday, November 10, 2016 8:51 AM To: Paolo Bonzini; Laszlo Ersek Cc: Kinney, Michael D; Tian, Feng; edk2-de...@ml01.01.org; Zeng, Star; Fan, Jeff Subject: RE: [edk2] [PATCH V2 0/6] Enable SMM page level protection. Fix a typo. From: Yao, Jiewen Sent: Thursday, November 10, 2016 8:49 AM To: 'Paolo Bonzini' <pbonz...@redhat.com<mailto:pbonz...@redhat.com>>; Laszlo Ersek <ler...@redhat.com<mailto:ler...@redhat.com>> Cc: Kinney, Michael D <michael.d.kin...@intel.com<mailto:michael.d.kin...@intel.com>>; Tian, Feng <feng.t...@intel.com<mailto:feng.t...@intel.com>>; edk2-de...@ml01.01.org<mailto:edk2-de...@ml01.01.org>; Zeng, Star <star.z...@intel.com<mailto:star.z...@intel.com>>; Fan, Jeff <jeff....@intel.com<mailto:jeff....@intel.com>> Subject: RE: [edk2] [PATCH V2 0/6] Enable SMM page level protection. > Anyway, I think if the BSP and the APs are properly synchronized around > the SMI injections in S3ResumeExecuteBootScript(), then this bug is > fixed. In that case, the APs' RSMs will restore the full context for the > APs, including their sleep in the HLT instruction, in CpuMpPei's wakeup > buffer. The BSP will proceed, exit PEI (restoring the CpuMpPei wakeup > buffer -- but the APs will sleep on), and then Linux will bring up the > APs, after taking control. Agreed. [Jiewen] I hold different opinion on that. If the AP is in hlt-loop at some below 1M memory with CR3 pointing to SMRAM, it has 2 issues. * The below 1M memory (0x9f0fd) might be consumed by OS with other instruction. * If AP starts running the code, it will get exception because CR3 is obviously wrong. The AP cannot fetch any code. We might have 2 possibles way to trigger this scenario, at least. A) Jeff and I have discovered one possible case – AP may receive NMI/SMI, such as periodic SMI, before OS sends INIT-SIPI-SIPI to wake up AP. B) Paolo has found one real case - AP is in SMRAM when BSP is out and about to close SMRAM. IMHO, letting BSP/AP sync in S3 resume just resolved B). But it does not help on A). If the system has some special SMI, such as periodic SMI. It will definitely trigger case A). So we have to fix the AP state anyway. Now, if the AP state is fixed, I do not think we need worry about the BSP/AP out of sync issue. BSP and AP can be independent. AP can receive NMI/SMI at any time and just work in HLT-loop. Thank you Yao Jiewen From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: Thursday, November 10, 2016 7:00 AM To: Laszlo Ersek <ler...@redhat.com<mailto:ler...@redhat.com>> Cc: Yao, Jiewen <jiewen....@intel.com<mailto:jiewen....@intel.com>>; Kinney, Michael D <michael.d.kin...@intel.com<mailto:michael.d.kin...@intel.com>>; Tian, Feng <feng.t...@intel.com<mailto:feng.t...@intel.com>>; edk2-de...@ml01.01.org<mailto:edk2-de...@ml01.01.org>; Zeng, Star <star.z...@intel.com<mailto:star.z...@intel.com>>; Fan, Jeff <jeff....@intel.com<mailto:jeff....@intel.com>> Subject: Re: [edk2] [PATCH V2 0/6] Enable SMM page level protection. > Another question I have -- and I feel I should really know it, but I > don't... -- is *why* the APs are executing code from the page at > 0x9f000. This I can answer. :) The APs have done their INIT-SIPI-SIPI, and then went into the CLI;HLT;JMP loop. When the AP exits SMM, it is in the JMP instruction. As suggested by Jiewen, edk2 could jump to a 32-bit loop that is _not_ in the 0-640K area (perhaps it could be in what your doc calls the "permanent PEI memory for the S3 resume path"?). After thinking a bit more about it, it seems simplest to me if CpuS3.c just uses SwitchStack or AsmDisablePaging64 at the end of MPRendezvousProcedure, to jump to a small stub like POP EAX ; pop return address POP EAX ; pop Context1 which is &mNumberToFinish DEC [EAX] 1: CLI HLT JMP 1 > I could be utterly and inexcusably wrong, but I think that the > RIP=0x9f0fd symptom is a red herring. I wouldn't call it a red herring. After all, CR3 points to SMM exactly because the CR3 that was set up for the 0x9f000 stub is CpuS3.c's SMRAM page table root. What _is_ a red herring is KVM's trace showing a RSM instruction at RIP=0x9f0fd. That is clearly bogus, RSM was rather the last instruction executed _before_ getting to that RIP. > > vcpu#0 vcpu#1 vcpu#2 vcpu#3 > > ------ ------ ------ ------ > > enter enter > > enter | enter | > > | | | | > > leave | | | > > <--------------------------- BAD > > enter | | | > > | | | | > > leave leave leave leave > > I don't understand why we don't get horrible faults on the APs > *immediately* when the BSP closes/locks down SMRAM. Everything in SMRAM, > page tables, executable code, everything, will read as 0xff on QEMU. How > can the APs continue in SMM long enough to > > (a) time out and pull the BSP back into SMM, > (b) complete the rendezvous and exit SMM? Because the "0xff" only applies when you're out of SMM. The three states (open, closed, closed/locked) only apply when you're not in SMM. While the AP is in SMM they are executing in a separate address space where SMRAM is "not closed". (In QEMU that's a separate AddressSpace struct, smram_address_space in target-i386/kvm.c). > I think I sort of answered question (2). (Apologies if Paolo and Jiewen > explained the exact same thing before; I had to spell it out for > myself.) That leaves question (1) open. Why enter SMM in > S3ResumeExecuteBootScript() at all? > > Anyway, I think if the BSP and the APs are properly synchronized around > the SMI injections in S3ResumeExecuteBootScript(), then this bug is > fixed. In that case, the APs' RSMs will restore the full context for the > APs, including their sleep in the HLT instruction, in CpuMpPei's wakeup > buffer. The BSP will proceed, exit PEI (restoring the CpuMpPei wakeup > buffer -- but the APs will sleep on), and then Linux will bring up the > APs, after taking control. Agreed. Paolo _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel