Re: 回复: [edk2-devel] 回复: [RESEND v2] MdePkg: use CpuPause() in CpuDeadLoop()
On 2021-03-24 8:15 p.m., gaoliming wrote:
Create PR https://github.com/tianocore/edk2/pull/1515
Thanks Liming.
Ankur
Thanks
Liming
-邮件原件-
发件人: devel@edk2.groups.io 代表 gaoliming
发送时间: 2021年3月22日 9:40
收件人: 'Ankur Arora' ; devel@edk2.groups.io
抄送: 'Michael D Kinney' ; 'Laszlo Ersek'
主题: [edk2-devel] 回复: [RESEND v2] MdePkg: use CpuPause() in
CpuDeadLoop()
Reviewed-by: Liming Gao
Thanks
Liming
-邮件原件-
发件人: Ankur Arora
发送时间: 2021年3月20日 2:14
收件人: devel@edk2.groups.io
抄送: Ankur Arora ; Liming Gao
; Michael D Kinney
; Laszlo Ersek
主题: [RESEND v2] MdePkg: use CpuPause() in CpuDeadLoop()
CpuPause() might allow the CPU to go into a lower power state
state while we spin.
On X86, CpuPause() executes a PAUSE instruction which the Intel
and AMD specs describe as follows:
Intel:
"PAUSE: An additional function of the PAUSE instruction is to reduce
the power consumed by a processor while executing a spin loop. A
processor can execute a spin-wait loop extremely quickly, causing the
processor to consume a lot of power while it waits for the resource it
is spinning on to become available. Inserting a pause instruction in a
spin-wait loop greatly reduces the processor’s power consumption."
AMD:
"PAUSE: Improves the performance of spin loops, by providing a hint to
the processor that the current code is in a spin loop. The processor
may use this to optimize power consumption while in the spin loop.
Architecturally, this instruction behaves like a NOP instruction."
On RISC-V and ARM64, CpuPause() executes a NOP, which is no worse than
the tight loop we have.
Cc: Liming Gao
Signed-off-by: Ankur Arora
Reviewed-by: Michael D Kinney
Reviewed-by: Laszlo Ersek
---
MdePkg/Library/BaseLib/CpuDeadLoop.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/MdePkg/Library/BaseLib/CpuDeadLoop.c
b/MdePkg/Library/BaseLib/CpuDeadLoop.c
index 9e110cacbc96..3cd304351a65 100644
--- a/MdePkg/Library/BaseLib/CpuDeadLoop.c
+++ b/MdePkg/Library/BaseLib/CpuDeadLoop.c
@@ -28,5 +28,7 @@ CpuDeadLoop (
{
volatile UINTN Index;
- for (Index = 0; Index == 0;);
+ for (Index = 0; Index == 0;) {
+CpuPause();
+ }
}
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#73286): https://edk2.groups.io/g/devel/message/73286
Mute This Topic: https://groups.io/mt/81594505/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [RESEND v2] MdePkg: use CpuPause() in CpuDeadLoop()
CpuPause() might allow the CPU to go into a lower power state
state while we spin.
On X86, CpuPause() executes a PAUSE instruction which the Intel
and AMD specs describe as follows:
Intel:
"PAUSE: An additional function of the PAUSE instruction is to reduce
the power consumed by a processor while executing a spin loop. A
processor can execute a spin-wait loop extremely quickly, causing the
processor to consume a lot of power while it waits for the resource it
is spinning on to become available. Inserting a pause instruction in a
spin-wait loop greatly reduces the processor’s power consumption."
AMD:
"PAUSE: Improves the performance of spin loops, by providing a hint to
the processor that the current code is in a spin loop. The processor
may use this to optimize power consumption while in the spin loop.
Architecturally, this instruction behaves like a NOP instruction."
On RISC-V and ARM64, CpuPause() executes a NOP, which is no worse than
the tight loop we have.
Cc: Liming Gao
Signed-off-by: Ankur Arora
Reviewed-by: Michael D Kinney
Reviewed-by: Laszlo Ersek
---
MdePkg/Library/BaseLib/CpuDeadLoop.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/MdePkg/Library/BaseLib/CpuDeadLoop.c
b/MdePkg/Library/BaseLib/CpuDeadLoop.c
index 9e110cacbc96..3cd304351a65 100644
--- a/MdePkg/Library/BaseLib/CpuDeadLoop.c
+++ b/MdePkg/Library/BaseLib/CpuDeadLoop.c
@@ -28,5 +28,7 @@ CpuDeadLoop (
{
volatile UINTN Index;
- for (Index = 0; Index == 0;);
+ for (Index = 0; Index == 0;) {
+CpuPause();
+ }
}
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#73051): https://edk2.groups.io/g/devel/message/73051
Mute This Topic: https://groups.io/mt/81462571/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v2] MdePkg: use CpuPause() in CpuDeadLoop()
CpuPause() might allow the CPU to go into a lower power state
state while we spin.
On X86, CpuPause() executes a PAUSE instruction which the Intel
and AMD specs describe as follows:
Intel:
"PAUSE: An additional function of the PAUSE instruction is to reduce
the power consumed by a processor while executing a spin loop. A
processor can execute a spin-wait loop extremely quickly, causing the
processor to consume a lot of power while it waits for the resource it
is spinning on to become available. Inserting a pause instruction in a
spin-wait loop greatly reduces the processorâs power consumption."
AMD:
"PAUSE: Improves the performance of spin loops, by providing a hint to
the processor that the current code is in a spin loop. The processor
may use this to optimize power consumption while in the spin loop.
Architecturally, this instruction behaves like a NOP instruction."
On RISC-V and ARM64, CpuPause() executes a NOP, which is no worse than
the tight loop we have.
Cc: Liming Gao
Signed-off-by: Ankur Arora
Reviewed-by: Michael D Kinney
Reviewed-by: Laszlo Ersek
---
MdePkg/Library/BaseLib/CpuDeadLoop.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/MdePkg/Library/BaseLib/CpuDeadLoop.c
b/MdePkg/Library/BaseLib/CpuDeadLoop.c
index 9e110cacbc96..3cd304351a65 100644
--- a/MdePkg/Library/BaseLib/CpuDeadLoop.c
+++ b/MdePkg/Library/BaseLib/CpuDeadLoop.c
@@ -28,5 +28,7 @@ CpuDeadLoop (
{
volatile UINTN Index;
- for (Index = 0; Index == 0;);
+ for (Index = 0; Index == 0;) {
+CpuPause();
+ }
}
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#73018): https://edk2.groups.io/g/devel/message/73018
Mute This Topic: https://groups.io/mt/81437102/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] 回复: [PATCH] MdePkg: use CpuPause() in CpuDeadLoop()
On 2021-03-16 6:08 p.m., gaoliming wrote:
Ankur:
Can you give the detail usage for the lower power state when enter into
CpuDeadLoop()?
I'm not really familiar with ARM and RiscV, both of which also have
implementations
for CpuPause(), for the details for X86 variants are below.
From Intel's instruction-set reference:
"An additional function of the PAUSE instruction is to reduce the power consumed
by a processor while executing a spin loop. A processor can execute a spin-wait
loop extremely quickly, causing the processor to consume a lot of power while it
waits for the resource it is spinning on to become available. Inserting a pause
instruction in a spin-wait loop greatly reduces the processor’s power
consumption."
And, from AMD's instruction-set reference:
"Improves the performance of spin loops, by providing a hint to the processor
that the current code is in a spin loop. The processor may use this to optimize
power consumption while in the spin loop. Architecturally, this instruction
behaves like a NOP instruction."
Happy to add this to the commit message if you think it'll be useful.
Thanks
Ankur
Thanks
Liming
-邮件原件-----
发件人: Ankur Arora
发送时间: 2021年3月17日 6:59
收件人: devel@edk2.groups.io
抄送: Ankur Arora ; Liming Gao
; Michael D Kinney
主题: [PATCH] MdePkg: use CpuPause() in CpuDeadLoop()
Use CpuPause() to allow the CPU to go into a lower power state
state while we spin wait.
Cc: Liming Gao
Signed-off-by: Ankur Arora
Reviewed-by: Michael D Kinney
---
MdePkg/Library/BaseLib/CpuDeadLoop.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/MdePkg/Library/BaseLib/CpuDeadLoop.c
b/MdePkg/Library/BaseLib/CpuDeadLoop.c
index 9e110cacbc96..3cd304351a65 100644
--- a/MdePkg/Library/BaseLib/CpuDeadLoop.c
+++ b/MdePkg/Library/BaseLib/CpuDeadLoop.c
@@ -28,5 +28,7 @@ CpuDeadLoop (
{
volatile UINTN Index;
- for (Index = 0; Index == 0;);
+ for (Index = 0; Index == 0;) {
+CpuPause();
+ }
}
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72949): https://edk2.groups.io/g/devel/message/72949
Mute This Topic: https://groups.io/mt/81393215/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH] MdePkg: use CpuPause() in CpuDeadLoop()
Use CpuPause() to allow the CPU to go into a lower power state
state while we spin wait.
Cc: Liming Gao
Signed-off-by: Ankur Arora
Reviewed-by: Michael D Kinney
---
MdePkg/Library/BaseLib/CpuDeadLoop.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/MdePkg/Library/BaseLib/CpuDeadLoop.c
b/MdePkg/Library/BaseLib/CpuDeadLoop.c
index 9e110cacbc96..3cd304351a65 100644
--- a/MdePkg/Library/BaseLib/CpuDeadLoop.c
+++ b/MdePkg/Library/BaseLib/CpuDeadLoop.c
@@ -28,5 +28,7 @@ CpuDeadLoop (
{
volatile UINTN Index;
- for (Index = 0; Index == 0;);
+ for (Index = 0; Index == 0;) {
+CpuPause();
+ }
}
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72938): https://edk2.groups.io/g/devel/message/72938
Mute This Topic: https://groups.io/mt/81390478/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v9 00/10] support CPU hot-unplug
On 2021-03-16 7:07 a.m., Laszlo Ersek wrote:
On 03/12/21 07:26, Ankur Arora wrote:
Hi,
This series adds OVMF support for CPU hot-unplug.
QEMU secureboot hot-unplug logic corresponding to this is in upstream.
Also posted here:
https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/
Testing (with QEMU 5.2.50):
- Stable with randomized CPU plug/unplug (guest maxcpus=33,128)
- Synthetic tests with simultaneous multi CPU hot-unplug
Also at:
github.com/terminus/edk2/ hot-unplug-v9
Changelog:
v9:
- Rebased on top of edd46cd407ea
- Clarify comments around memory-barriers in patches 7, 8, 9
- Address other review comments from v8
Merged as commit range 4751a48aeb2a..f3bdfc41866e, via
<https://github.com/tianocore/edk2/pull/1494>, with the following R-b's
/ comments / light tweaks added:
Awesome! Thanks Laszlo.
And, thanks especially for the thorough reviews.
Ankur
1: 4b4466ed7c44 = 1: 0cb242e33693 OvmfPkg/CpuHotplugSmm: refactor hotplug
logic
2: 719828efdebe ! 2: a752dd07466c OvmfPkg/CpuHotplugSmm: collect
hot-unplug events
@@ -20,6 +20,7 @@
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Message-Id: <20210312062656.2477515-3-ankur.a.ar...@oracle.com>
+Reviewed-by: Laszlo Ersek
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
3: 7d732efcb7af = 3: 2d92e052c3af OvmfPkg/CpuHotplugSmm: add Qemu Cpu
Status helper
4: 64649e42 = 4: 15e6ae8ea400 OvmfPkg/CpuHotplugSmm: introduce
UnplugCpus()
5: 7190a5d134a7 ! 5: 8ade9d425a6e OvmfPkg: define CPU_HOT_EJECT_DATA
@@ -15,6 +15,7 @@
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Message-Id: <20210312062656.2477515-6-ankur.a.ar...@oracle.com>
+Reviewed-by: Laszlo Ersek
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
--- a/OvmfPkg/OvmfPkg.dec
6: 1fd019dac476 ! 6: b6d5996706dd OvmfPkg/SmmCpuFeaturesLib: init CPU
ejection state
@@ -22,6 +22,7 @@
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Message-Id: <20210312062656.2477515-7-ankur.a.ar...@oracle.com>
+Reviewed-by: Laszlo Ersek
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
7: c8758d9b9764 ! 7: af9c77e151fa OvmfPkg/SmmCpuFeaturesLib: call CPU
hot-eject handler
@@ -19,6 +19,7 @@
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Message-Id: <20210312062656.2477515-8-ankur.a.ar...@oracle.com>
+Reviewed-by: Laszlo Ersek
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
8: 5ff8e8700678 ! 8: 30c69d2cfa63 OvmfPkg/CpuHotplugSmm: add EjectCpu()
@@ -19,6 +19,7 @@
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Message-Id: <20210312062656.2477515-9-ankur.a.ar...@oracle.com>
+Reviewed-by: Laszlo Ersek
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
9: d6660f8f2f14 ! 9: f05328886302 OvmfPkg/CpuHotplugSmm: do actual CPU
hot-eject
@@ -27,6 +27,9 @@
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Message-Id: <20210312062656.2477515-10-ankur.a.ar...@oracle.com>
+Reviewed-by: Laszlo Ersek
+[ler...@redhat.com: unneeded inner QemuSelector declaration in
EjectCpu()
+ triggers VS warning #4456 (local variable shadowed); remove it]
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -102,8 +105,6 @@
+UINT32 Idx;
+
+for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) {
-+ UINT64 QemuSelector;
-+
+ QemuSelector = mCpuHotEjectData->QemuSelectorMap[Idx];
+
+ if (QemuSelector != CPU_EJECT_QEMU_SELECTOR_INVALID) {
10: 0a9a1cd40b98 ! 10: f3bdfc41866e OvmfPkg/SmmControl2Dxe: negotiate CPU
hot-unplug
@@ -14,6 +14,9 @@
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Message-Id: <20210312062656.2477515-11-ankur.a.ar...@oracle.com>
[edk2-devel] [PATCH v9 09/10] OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject
Add logic in EjectCpu() to do the actual the CPU ejection.
On the BSP, ejection happens by first selecting the CPU via
its QemuSelector and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
the CPU out of the SMI handler.
Meanwhile the CPU being ejected, waits around in its holding
area until it is context-switched out. Note that it is possible
that a slow CPU gets ejected before it reaches the wait loop.
However, this would never happen before it has executed the
"AllCpusInSync" loop in SmiRendezvous().
It can mean that an ejected CPU does not execute code after
that point but given that the CPU state will be destroyed by
QEMU, the missed cleanup is no great loss.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following comments from v8:
(1a,1b) CheckIfBsp(): get rid of ProcessorNum, document retval.
(2) Line up IsBsp and ApicBaseMsr
(3) s/ongoing SMI iteration/ongoing SMI/
(4) Get rid of the allusions to alignment in the comment in EjectCpu().
() Also reduce some of the repetitive detail in this comment.
(5) EjectCpu(): reorder logic to cleanly separate the AP and the BSP
portions.
(6) Get rid of unnecessary MemoryFence() between QemuCpuhpWrite
and clearing of the eject status.
(7) Change type of QemuSelector to %Lu in DEBUG statement
(8) Get rid of the repetitive comment in SmmCpuFeaturesRendezvousExit().
The necessary parts of this got moved to patch-7.
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c| 113 --
2 files changed, 108 insertions(+), 6 deletions(-)
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index 2ec7a107a64d..d0e83102c13f 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_EJECT BIT3
#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 2eeb4567a262..ae3abd525900 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -18,6 +18,7 @@
#include // CPU_HOT_EJECT_DATA
#include // EFI_MM_CPU_IO_PROTOCOL
#include // EFI_SMM_CPU_SERVICE_PROTOCOL
+#include // MSR_IA32_APIC_BASE_REGISTER
#include// EFI_STATUS
#include "ApicId.h" // APIC_ID
@@ -193,12 +194,40 @@ RevokeNewSlot:
}
/**
+ EjectCpu needs to know the BSP at SMI exit at a point when
+ some of the EFI_SMM_CPU_SERVICE_PROTOCOL state has been torn
+ down.
+ Reuse the logic from OvmfPkg::PlatformSmmBspElection() to
+ do that.
+
+ @retval TRUE If the CPU executing this function is the BSP.
+
+ @retval FALSE If the CPU executing this function is an AP.
+**/
+STATIC
+BOOLEAN
+CheckIfBsp (
+ VOID
+ )
+{
+ MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr;
+ BOOLEAN IsBsp;
+
+ ApicBaseMsr.Uint64 = AsmReadMsr64 (MSR_IA32_APIC_BASE);
+ IsBsp = (BOOLEAN)(ApicBaseMsr.Bits.BSP == 1);
+ return IsBsp;
+}
+
+/**
CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
on each CPU at exit from SMM.
- If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is neither the BSP, nor being ejected, nothing
+ to be done.
If, the executing CPU is being ejected, wait in a halted loop
until ejected.
+ If, the executing CPU is the BSP, set QEMU CPU status to eject
+ for CPUs being ejected.
@param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
and will be used as an index into
@@ -214,6 +243,83 @@ EjectCpu (
{
UINT64 QemuSelector;
+ if (CheckIfBsp ()) {
+UINT32 Idx;
+
+for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) {
+ UINT64 QemuSelector;
+
+ QemuSelector = mCpuHotEjectData->QemuSelectorMap[Idx];
+
+ if (QemuSelector != CPU_EJECT_QEMU_SELECTOR_INVALID) {
+//
+// This to-be-ejected-CPU has already received the BSP's SMI exit
+// signal and will execute SmmCpuFeaturesRendezvousExit()
+// followed by this callback or is already penned in the
+// CpuSleep() loop below.
+//
+// Tell QEMU to context-switch it out.
+//
+QemuCpuhpWriteCpuSelector (mMmCpuIo, (UINT32) QemuSelector);
+
[edk2-devel] [PATCH v9 10/10] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
Advertise OVMF support for CPU hot-unplug and negotiate it
if QEMU requests the feature.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following comments from v8:
(1) Remove inconsistent comment style (and stray newline) around the newly
added ICH9_LPC_SMI_F_CPU_HOT_UNPLUG.
(2) Remove spurious empty line.
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 18 --
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
index c9d875543205..3e2e61e4dbd0 100644
--- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
+++ b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
@@ -28,7 +28,12 @@
// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
//
#define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
-
+//
+// The following bit value stands for "enable CPU hot-unplug, and inject an SMI
+// with control value ICH9_APM_CNT_CPU_HOTPLUG upon hot-unplug", in the
+// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
+//
+#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
//
// Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
// for the S3 boot script fragment to write to and read from.
@@ -112,7 +117,8 @@ NegotiateSmiFeatures (
QemuFwCfgReadBytes (sizeof mSmiFeatures, );
//
- // We want broadcast SMI, SMI on CPU hotplug, and nothing else.
+ // We want broadcast SMI, SMI on CPU hotplug, SMI on CPU hot-unplug
+ // and nothing else.
//
RequestedFeaturesMask = ICH9_LPC_SMI_F_BROADCAST;
if (!MemEncryptSevIsEnabled ()) {
@@ -120,6 +126,7 @@ NegotiateSmiFeatures (
// For now, we only support hotplug with SEV disabled.
//
RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOTPLUG;
+RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
}
mSmiFeatures &= RequestedFeaturesMask;
QemuFwCfgSelectItem (mRequestedFeaturesItem);
@@ -166,6 +173,13 @@ NegotiateSmiFeatures (
__FUNCTION__));
}
+ if ((mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG) == 0) {
+DEBUG ((DEBUG_INFO, "%a: CPU hot-unplug not negotiated\n", __FUNCTION__));
+ } else {
+DEBUG ((DEBUG_INFO, "%a: CPU hot-unplug with SMI negotiated\n",
+ __FUNCTION__));
+ }
+
//
// Negotiation successful (although we may not have gotten the optimal
// feature set).
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72712): https://edk2.groups.io/g/devel/message/72712
Mute This Topic: https://groups.io/mt/81273612/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v9 01/10] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
ProcessHotAddedCpus(). This is in preparation for supporting CPU
hot-unplug.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Reviewed-by: Laszlo Ersek
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 210 ++---
1 file changed, 126 insertions(+), 84 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..bf68fcd42914 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,129 @@ STATIC UINT32 mPostSmmPenAddress;
//
STATIC EFI_HANDLE mDispatchHandle;
+/**
+ Process CPUs that have been hot-added, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, relocate the SMBASE, and report the CPU to PiSmmCpuDxeSmm
+ via EFI_SMM_CPU_SERVICE_PROTOCOL. If the supposedly hot-added CPU is already
+ known, skip it silently.
+
+ @param[in] PluggedApicIdsThe APIC IDs of the CPUs that have been
+ hot-plugged.
+
+ @param[in] PluggedCount The number of filled-in APIC IDs in
+ PluggedApicIds.
+
+ @retval EFI_SUCCESS CPUs corresponding to all the APIC IDs are
+ populated.
+
+ @retval EFI_OUT_OF_RESOURCES Out of APIC ID space in "mCpuHotPlugData".
+
+ @return Error codes propagated from SmbaseRelocate()
+ and mMmCpuService->AddProcessor().
+**/
+STATIC
+EFI_STATUS
+ProcessHotAddedCpus (
+ IN APIC_ID *PluggedApicIds,
+ IN UINT32 PluggedCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 PluggedIdx;
+ UINT32 NewSlot;
+
+ //
+ // The Post-SMM Pen need not be reinstalled multiple times within a single
+ // root MMI handling. Even reinstalling once per root MMI is only prudence;
+ // in theory installing the pen in the driver's entry point function should
+ // suffice.
+ //
+ SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+
+ PluggedIdx = 0;
+ NewSlot = 0;
+ while (PluggedIdx < PluggedCount) {
+APIC_ID NewApicId;
+UINT32 CheckSlot;
+UINTN NewProcessorNumberByProtocol;
+
+NewApicId = PluggedApicIds[PluggedIdx];
+
+//
+// Check if the supposedly hot-added CPU is already known to us.
+//
+for (CheckSlot = 0;
+ CheckSlot < mCpuHotPlugData->ArrayLength;
+ CheckSlot++) {
+ if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
+break;
+ }
+}
+if (CheckSlot < mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
+"before; ignoring it\n", __FUNCTION__, NewApicId));
+ PluggedIdx++;
+ continue;
+}
+
+//
+// Find the first empty slot in CPU_HOT_PLUG_DATA.
+//
+while (NewSlot < mCpuHotPlugData->ArrayLength &&
+ mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
+ NewSlot++;
+}
+if (NewSlot == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
+__FUNCTION__, NewApicId));
+ return EFI_OUT_OF_RESOURCES;
+}
+
+//
+// Store the APIC ID of the new processor to the slot.
+//
+mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
+
+//
+// Relocate the SMBASE of the new CPU.
+//
+Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
+ mPostSmmPenAddress);
+if (EFI_ERROR (Status)) {
+ goto RevokeNewSlot;
+}
+
+//
+// Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
+//
+Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
+ );
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, NewApicId, Status));
+ goto RevokeNewSlot;
+}
+
+DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
+ "EFI_SMM_CPU_SERVICE_PROTOCOL assigned number %Lu\n", __FUNCTION__,
+ NewApicId, (UINT64)mCpuHotPlugData->SmBase[NewSlot],
+ (UINT64)NewProcessorNumberByProtocol));
+
+NewSlot++;
+PluggedIdx++;
+ }
+
+ //
+ // We've processed this batch of hot-added CPUs.
+ //
+ return EFI_SUCCESS;
+
+RevokeNewSlot:
+ mCpuHotPlugData->ApicId[NewSlot] = MAX_UINT64;
+
+ return Status;
+}
/**
CPU Hotplug MMI handler function.
@@ -122,8 +245,6 @@ CpuHotplugMmi (
UINT8 ApmControl;
UINT32 PluggedCount;
UINT32 ToUnplugCount;
- UINT32 PluggedIdx;
- UINT32 NewSlot;
//
// Assert that we are entering this fu
[edk2-devel] [PATCH v9 03/10] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Reviewed-by: Laszlo Ersek
---
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 22 ++
2 files changed, 28 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 3e2c2192e1c0..8bb3c66e9b44 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -42,6 +42,12 @@ QemuCpuhpWriteCpuSelector (
);
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ );
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 8434dd446b96..dc86ab96777a 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -114,6 +114,28 @@ QemuCpuhpWriteCpuSelector (
}
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ )
+{
+ EFI_STATUS Status;
+
+ Status = MmCpuIo->Io.Write (
+ MmCpuIo,
+ MM_IO_UINT8,
+ ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+ 1,
+
+ );
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+ASSERT (FALSE);
+CpuDeadLoop ();
+ }
+}
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72705): https://edk2.groups.io/g/devel/message/72705
Mute This Topic: https://groups.io/mt/81273604/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v9 08/10] OvmfPkg/CpuHotplugSmm: add EjectCpu()
Add EjectCpu(), which handles the CPU ejection, and provides a holding
area for said CPUs. It is called via SmmCpuFeaturesRendezvousExit(),
at the tail end of the SMI handling.
Also UnplugCpus() now stashes QEMU Selectors of CPUs which need to be
ejected in CPU_HOT_EJECT_DATA.QemuSelectorMap. This is used by
EjectCpu() to identify CPUs marked for ejection.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following comments from v8:
(1) Fixup the coment about UnplugCpus() to reference stashing QEMU
Cpu Selectors instead of APIC IDs.
(2) s/ToUnplugSelector/ToUnplugSelectors/
(3) Use plural for APIC ID in comment describing retval EFI_ALREADY_STARTED.
(4) Fixup indentation in check against CPU_EJECT_QEMU_SELECTOR_INVALID.
(5) Clarify comment:
- // never match more than one APIC ID and by transitivity, more than one
- // QemuSelector in a single invocation of UnplugCpus().
+ // never match more than one APIC ID -- nor, by transitivity, designate
+ // more than one QemuSelector -- in a single invocation of UnplugCpus().
(6a) Remove unnecessary UINT64 cast for
mCpuHotEjectData->QemuSelectorMap[ProcessorNum].
(6b) Switch from 0x%Lx -> %Lu for QemuSelectorMap[ProcessorNum].
(6c) Switch from 0x%Lx -> %u for QemuSelector
(7) Switch to "return EFI_ALREADY_STARTED".
(8a) Replace "QemuSelector 0x%Lx" with "QemuSelector %u".
(8b) Replace the mCpuHotEjectData->QemuSelectorMap[ProcessorNum] argument
with just QemuSelector in DEBUG call.
(9) Clarify comment and make the language complementary to that in patch-7
Explicitly mention release memory fence.
OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf | 2 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 154 ++--
2 files changed, 148 insertions(+), 8 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
index 04322b0d7855..ebcc7e2ac63a 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
@@ -40,6 +40,7 @@ [Packages]
[LibraryClasses]
BaseLib
BaseMemoryLib
+ CpuLib
DebugLib
LocalApicLib
MmServicesTableLib
@@ -54,6 +55,7 @@ [Protocols]
[Pcd]
gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress## CONSUMES
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress ## CONSUMES
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase ## CONSUMES
[FeaturePcd]
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 59f000eb7886..2eeb4567a262 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -10,10 +10,12 @@
#include // ICH9_APM_CNT
#include // QEMU_CPUHP_CMD_GET_PENDING
#include // CpuDeadLoop()
+#include // CpuSleep()
#include // ASSERT()
#include // gMmst
#include // PcdGetBool()
#include // SafeUintnSub()
+#include // CPU_HOT_EJECT_DATA
#include // EFI_MM_CPU_IO_PROTOCOL
#include // EFI_SMM_CPU_SERVICE_PROTOCOL
#include// EFI_STATUS
@@ -32,11 +34,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo;
//
STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService;
//
-// This structure is a communication side-channel between the
+// These structures serve as communication side-channels between the
// EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider
// (i.e., PiSmmCpuDxeSmm).
//
STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData;
//
// SMRAM arrays for fetching the APIC IDs of processors with pending events (of
// known event types), for the time of just one MMI.
@@ -190,18 +193,71 @@ RevokeNewSlot:
}
/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
+ on each CPU at exit from SMM.
+
+ If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is being ejected, wait in a halted loop
+ until ejected.
+
+ @param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
+ and will be used as an index into
+ CPU_HOT_EJECT_DATA->QemuSelectorMap. It is
+ identical to the processor handle number in
+ EFI_SMM_CPU_SERVICE_PROTOCOL.
+**/
+VOID
+EFIAPI
+EjectCpu (
+ IN UINTN ProcessorNum
+ )
+{
+ UINT64 QemuSelector;
+
+ QemuSelector = mCpuHotEjectData->QemuSelectorMap[ProcessorNum];
+ if (QemuSelector == CPU_EJECT_QEMU_SELECTOR_INVALID) {
+return;
+ }
+
+ //
+ // APs being unplugged get here from SmmCpuFeatures
[edk2-devel] [PATCH v9 06/10] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection
state between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
The init happens via SmmCpuFeaturesSmmRelocationComplete(), and so it
will run as part of the PiSmmCpuDxeSmm entry point function,
PiCpuSmmEntry(). Once inited, CPU_HOT_EJECT_DATA is exposed via
PcdCpuHotEjectDataAddress.
The CPU hot-eject handler (CPU_HOT_EJECT_DATA->Handler) is setup when
there is an ejection request via CpuHotplugSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following comments from v8:
(1) Remove line before the "if (MaxNumberofCpus == 1)" check.
(3) Fixup the space around "||".
(2,6) Simplify the three SafeInt multiplication into the ones suggested
by Laszlo.
(4) Get rid of the mixed sizeof(mCpuHotEjectData->QemuSelectorMap[0]) and
sizeof(UINT64) in favour of UINT64 everywhere. I was planning to use
the first, but describing the alignment needed is easier in terms of the
second.
Also, as Laszlo's comments on v8-patch-9 mention, we don't really need
this alignment for correctness reasons. This patch retains it, so we
don't pay access penalty for unaligned access.
(5) Change alignment from UINT64 to UINT64-1.
(7) Use the more idiomatic ALIGN_POINTER instead of ALIGN_VALUE.
(8) RETURN_ERROR -> ASSERT_RETURN_ERROR.
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 ++
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 77 ++
2 files changed, 81 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..8a426a4c10fb 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -30,9 +30,13 @@ [LibraryClasses]
BaseMemoryLib
DebugLib
MemEncryptSevLib
+ MemoryAllocationLib
PcdLib
+ SafeIntLib
SmmServicesTableLib
UefiBootServicesTableLib
[Pcd]
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..5c025bc717c3 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -11,10 +11,13 @@
#include
#include
#include
+#include
#include
+#include
#include
#include
#include
+#include
#include
#include
#include
@@ -171,6 +174,77 @@ SmmCpuFeaturesHookReturnFromSmm (
return OriginalInstructionPointer;
}
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+/**
+ Initialize mCpuHotEjectData if PcdCpuMaxLogicalProcessorNumber > 1.
+
+ Also setup the corresponding PcdCpuHotEjectDataAddress.
+**/
+STATIC
+VOID
+InitCpuHotEjectData (
+ VOID
+ )
+{
+ UINTN Size;
+ UINT32 Idx;
+ UINT32 MaxNumberOfCpus;
+ RETURN_STATUS PcdStatus;
+
+ MaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+ if (MaxNumberOfCpus == 1) {
+return;
+ }
+
+ //
+ // We allocate CPU_HOT_EJECT_DATA and CPU_HOT_EJECT_DATA->QemuSelectorMap[]
+ // in a single allocation, and explicitly align the QemuSelectorMap[] (which
+ // is a UINT64 array) at its natural boundary.
+ // Accordingly, allocate:
+ // sizeof(*mCpuHotEjectData) + (MaxNumberOfCpus * sizeof(UINT64))
+ // and, add sizeof(UINT64) - 1 to use as padding if needed.
+ //
+
+ if (RETURN_ERROR (SafeUintnMult (MaxNumberOfCpus, sizeof (UINT64), )) ||
+ RETURN_ERROR (SafeUintnAdd (Size, sizeof (*mCpuHotEjectData), )) ||
+ RETURN_ERROR (SafeUintnAdd (Size, sizeof (UINT64) - 1, ))) {
+DEBUG ((DEBUG_ERROR, "%a: invalid CPU_HOT_EJECT_DATA\n", __FUNCTION__));
+goto Fatal;
+ }
+
+ mCpuHotEjectData = AllocatePool (Size);
+ if (mCpuHotEjectData == NULL) {
+ASSERT (mCpuHotEjectData != NULL);
+goto Fatal;
+ }
+
+ mCpuHotEjectData->Handler = NULL;
+ mCpuHotEjectData->ArrayLength = MaxNumberOfCpus;
+
+ mCpuHotEjectData->QemuSelectorMap = ALIGN_POINTER (mCpuHotEjectData + 1,
+sizeof (UINT64));
+ //
+ // We use mCpuHotEjectData->QemuSelectorMap to map
+ // ProcessorNum -> QemuSelector. Initialize to invalid values.
+ //
+ for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) {
+mCpuHotEjectData->QemuSelectorMap[Idx] = CPU_EJECT_QEMU_SELECTOR_INVALID;
+ }
+
+ //
+ // Expose address of CPU Hot eject Data structure
+ //
+ PcdStatus = PcdSet64S (PcdCpuHotEjectDataAddress,
+
[edk2-devel] [PATCH v9 07/10] OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
Call the CPU hot-eject handler if one is installed. The condition for
installation is (PcdCpuMaxLogicalProcessorNumber > 1), and there's
a hot-unplug request.
The handler is called from SmmCpuFeaturesRendezvousExit(), which is
in-turn called at the tail-end of SmiRendezvous() after the BSP has
signalled an SMI exit via the "AllCpusInSync" loop.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following comments from v8:
(1) Add a MemoryFence() before accessing mCpuHotEjctData->Handler
(and comment to that effect.)
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 34 ++
1 file changed, 34 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 5c025bc717c3..fdf2380974fa 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -452,6 +452,40 @@ SmmCpuFeaturesRendezvousExit (
IN UINTN CpuIndex
)
{
+ //
+ // We only call the Handler if CPU hot-eject is enabled
+ // (PcdCpuMaxLogicalProcessorNumber > 1), and hot-eject is needed
+ // in this SMI exit (otherwise mCpuHotEjectData->Handler is not armed.)
+ //
+
+ if (mCpuHotEjectData != NULL) {
+CPU_HOT_EJECT_HANDLER Handler;
+
+//
+// As the comment above mentions, mCpuHotEjectData->Handler might be
+// written to on the BSP as part of handling of the CPU-ejection.
+//
+// We know that any initial assignment to mCpuHotEjectData->Handler
+// (on the BSP, in the CpuHotplugMmi() context) is ordered-before the
+// load below, since it is guaranteed to happen before the
+// control-dependency of the BSP's SMI exit signal -- by way of a store
+// to AllCpusInSync (on the BSP, in BspHandler()) and the corresponding
+// AllCpusInSync loop (on the APs, in SmiRendezvous()) which depends on
+// that store.
+//
+// This guarantees that these pieces of code can never execute
+// simultaneously. In addition, we ensure that the following load is
+// ordered-after the AllCpusInSync loop by using a MemoryFence() with
+// acquire semantics.
+//
+MemoryFence();
+
+Handler = mCpuHotEjectData->Handler;
+
+if (Handler != NULL) {
+ Handler (CpuIndex);
+}
+ }
}
/**
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72709): https://edk2.groups.io/g/devel/message/72709
Mute This Topic: https://groups.io/mt/81273609/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v9 05/10] OvmfPkg: define CPU_HOT_EJECT_DATA
Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following comments from v8:
(1) Get rid of the unnecessary commit specifier from the subject.
(2) s/MaxNumberOfCpus/PcdCpuMaxLogicalProcessorNumber/
(3) Shifted the comments to be above each structure field.
OvmfPkg/OvmfPkg.dec | 4 +++
OvmfPkg/Include/Pcd/CpuHotEjectData.h | 60 +++
2 files changed, 64 insertions(+)
create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 4348bb45c64a..9629707020ba 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -352,6 +352,10 @@ [PcdsDynamic, PcdsDynamicEx]
# This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+ ## This PCD adds a communication channel between OVMF's SmmCpuFeaturesLib
+ # instance in PiSmmCpuDxeSmm, and CpuHotplugSmm.
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
+
[PcdsFeatureFlag]
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/Include/Pcd/CpuHotEjectData.h
b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
new file mode 100644
index ..06714375526c
--- /dev/null
+++ b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
@@ -0,0 +1,60 @@
+/** @file
+ Definition for the CPU_HOT_EJECT_DATA structure, which shares
+ CPU hot-eject state between OVMF's SmmCpuFeaturesLib instance in
+ PiSmmCpuDxeSmm, and CpuHotplugSmm.
+
+ CPU_HOT_EJECT_DATA is allocated in SMRAM, and pointed-to by
+ PcdCpuHotEjectDataAddress.
+
+ PcdCpuHotEjectDataAddress is valid when SMM_REQUIRE is TRUE
+ and PcdCpuMaxLogicalProcessorNumber > 1.
+
+ Copyright (C) 2021, Oracle Corporation.
+
+ SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#ifndef CPU_HOT_EJECT_DATA_H_
+#define CPU_HOT_EJECT_DATA_H_
+
+/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
+ on each CPU at exit from SMM.
+
+ @param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
+ and will be used as an index into
+ CPU_HOT_EJECT_DATA->QemuSelectorMap. It is
+ identical to the processor handle in
+ EFI_SMM_CPU_SERVICE_PROTOCOL.
+**/
+typedef
+VOID
+(EFIAPI *CPU_HOT_EJECT_HANDLER) (
+ IN UINTN ProcessorNum
+ );
+
+//
+// CPU_EJECT_QEMU_SELECTOR_INVALID marks CPUs not being ejected in
+// CPU_HOT_EJECT_DATA->QemuSelectorMap.
+//
+// QEMU CPU Selector is UINT32, so we choose an invalid value larger
+// than that type.
+//
+#define CPU_EJECT_QEMU_SELECTOR_INVALID (MAX_UINT64)
+
+typedef struct {
+ //
+ // Maps ProcessorNum -> QemuSelector for pending hot-ejects
+ //
+ volatile UINT64 *QemuSelectorMap;
+ //
+ // Handler to do the CPU ejection
+ //
+ volatile CPU_HOT_EJECT_HANDLER Handler;
+ //
+ // Entries in the QemuSelectorMap
+ //
+ UINT32 ArrayLength;
+} CPU_HOT_EJECT_DATA;
+
+#endif // CPU_HOT_EJECT_DATA_H_
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72708): https://edk2.groups.io/g/devel/message/72708
Mute This Topic: https://groups.io/mt/81273608/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v9 04/10] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().
With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Reviewed-by: Laszlo Ersek
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 84 ++
1 file changed, 84 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index ee1497b93140..59f000eb7886 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -190,6 +190,83 @@ RevokeNewSlot:
}
/**
+ Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, report the CPU to PiSmmCpuDxeSmm via
+ EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is
+ unknown, skip it silently.
+
+ @param[in] ToUnplugApicIdsThe APIC IDs of the CPUs that are about to be
+hot-unplugged.
+
+ @param[in] ToUnplugCount The number of filled-in APIC IDs in
+ToUnplugApicIds.
+
+ @retval EFI_SUCCESS Known APIC IDs have been removed from SMM data
+structures.
+
+ @return Error codes propagated from
+mMmCpuService->RemoveProcessor().
+**/
+STATIC
+EFI_STATUS
+UnplugCpus (
+ IN APIC_ID *ToUnplugApicIds,
+ IN UINT32 ToUnplugCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 ToUnplugIdx;
+ UINTN ProcessorNum;
+
+ ToUnplugIdx = 0;
+ while (ToUnplugIdx < ToUnplugCount) {
+APIC_IDRemoveApicId;
+
+RemoveApicId = ToUnplugApicIds[ToUnplugIdx];
+
+//
+// mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+// the ProcessorNum for the APIC ID to be removed.
+//
+for (ProcessorNum = 0;
+ ProcessorNum < mCpuHotPlugData->ArrayLength;
+ ProcessorNum++) {
+ if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+break;
+ }
+}
+
+//
+// Ignore the unplug if APIC ID not found
+//
+if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: did not find APIC ID " FMT_APIC_ID
+" to unplug\n", __FUNCTION__, RemoveApicId));
+ ToUnplugIdx++;
+ continue;
+}
+
+//
+// Mark ProcessorNum for removal from SMM data structures
+//
+Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, RemoveApicId, Status));
+ return Status;
+}
+
+ToUnplugIdx++;
+ }
+
+ //
+ // We've removed this set of APIC IDs from SMM data structures.
+ //
+ return EFI_SUCCESS;
+}
+
+/**
CPU Hotplug MMI handler function.
This is a root MMI handler.
@@ -311,6 +388,13 @@ CpuHotplugMmi (
}
}
+ if (ToUnplugCount > 0) {
+Status = UnplugCpus (mToUnplugApicIds, ToUnplugCount);
+if (EFI_ERROR (Status)) {
+ goto Fatal;
+}
+ }
+
//
// We've handled this MMI.
//
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72706): https://edk2.groups.io/g/devel/message/72706
Mute This Topic: https://groups.io/mt/81273605/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v9 02/10] OvmfPkg/CpuHotplugSmm: collect hot-unplug events
Process fw_remove events in QemuCpuhpCollectApicIds(), and collect APIC IDs
and QEMU CPU Selectors for CPUs being hot-unplugged.
In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.
This is based on the QEMU hot-unplug protocol documented here:
https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imamm...@redhat.com/
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following comments from v8:
(1) Fix commit message to mention that we collect cpu-selectors as well.
(2,3,6) s/UnplugSelector/UnplugSelectors/ in CpuHotplug.c, QemuCpuhp.c
(4) Fix comment above the declaration of the now renamed
mToUnplugSelector.
(5) Fix spacing around "||".
(7) Fix QemuCpuCollectApicIds() comments to line up descriptions for
ToUnplugSelectors and other params.
(8) s/ExtendSel/ExtendSels/.
(9) Add the (ExtendSels => ExtendIds) assert.
(10) Fix the missing CurrentSelector++ bug.
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 1 +
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c| 29 +--
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 101 +++---
4 files changed, 93 insertions(+), 39 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..3e2c2192e1c0 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -55,6 +55,7 @@ QemuCpuhpCollectApicIds (
OUT APIC_ID *PluggedApicIds,
OUT UINT32 *PluggedCount,
OUT APIC_ID *ToUnplugApicIds,
+ OUT UINT32 *ToUnplugSelectors,
OUT UINT32 *ToUnplugCount
);
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..2ec7a107a64d 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index bf68fcd42914..ee1497b93140 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -45,13 +45,16 @@ STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
// don't want to allocate SMRAM at OS runtime, and potentially fail (or
// fragment the SMRAM map).
//
-// These arrays provide room for ("possible CPU count" minus one) APIC IDs
-// each, as we don't expect every possible CPU to appear, or disappear, in a
-// single MMI. The numbers of used (populated) elements in the arrays are
+// The first array stores APIC IDs for hot-plug events, the second and the
+// third store APIC IDs and QEMU CPU Selectors (both indexed similarly) for
+// hot-unplug events. All of these provide room for "possible CPU count" minus
+// one elements as we don't expect every possible CPU to appear, or disappear,
+// in a single MMI. The numbers of used (populated) elements in the arrays are
// determined on every MMI separately.
//
STATIC APIC_ID *mPluggedApicIds;
STATIC APIC_ID *mToUnplugApicIds;
+STATIC UINT32 *mToUnplugSelectors;
//
// Address of the non-SMRAM reserved memory page that contains the Post-SMM Pen
// for hot-added CPUs.
@@ -289,6 +292,7 @@ CpuHotplugMmi (
mPluggedApicIds,
,
mToUnplugApicIds,
+ mToUnplugSelectors,
);
if (EFI_ERROR (Status)) {
@@ -333,7 +337,9 @@ CpuHotplugEntry (
)
{
EFI_STATUS Status;
+ UINTN Len;
UINTN Size;
+ UINTN SizeSel;
//
// This module should only be included when SMM support is required.
@@ -387,8 +393,9 @@ CpuHotplugEntry (
//
// Allocate the data structures that depend on the possible CPU count.
//
- if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
- RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Size, ))) {
+ if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
+ RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Len, )) ||
+ RETURN_ERROR (SafeUintnMult (sizeof (UINT32), Len, ))) {
Status = EFI_ABORTED;
DEBUG ((DEBUG_ERROR, "%a: invalid CPU_HOT_PLUG_DATA\n", __FUNCTION__));
goto Fatal;
@@ -405,6 +412,12 @@ CpuHotplugEntry (
DEBUG ((DEBUG_ERROR, "%a: MmAllocatePool(): %r\n", __FUNCTION__
[edk2-devel] [PATCH v9 00/10] support CPU hot-unplug
Hi, This series adds OVMF support for CPU hot-unplug. QEMU secureboot hot-unplug logic corresponding to this is in upstream. Also posted here: https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/ Testing (with QEMU 5.2.50): - Stable with randomized CPU plug/unplug (guest maxcpus=33,128) - Synthetic tests with simultaneous multi CPU hot-unplug Also at: github.com/terminus/edk2/ hot-unplug-v9 Changelog: v9: - Rebased on top of edd46cd407ea - Clarify comments around memory-barriers in patches 7, 8, 9 - Address other review comments from v8 v8: - Fixes a couple of ECC issues in the code (in patches 7, 9) URL: https://patchew.org/EDK2/20210222071928.1401820-1-ankur.a.ar...@oracle.com/ v7: - Address review comments from v6. - Fix ejection bug where we were using APIC ID to do the ejection rather than the Qemu Selector. - Describes safety properties and ordering needed for concurrent accesses to CPU_HOT_EJECT_DATA->QemuSelectorMap, and CPU_HOT_EJECT_DATA->Handler. URL: https://patchew.org/EDK2/20210219090444.1332380-1-ankur.a.ar...@oracle.com/ v6: - addresses v5 review comments. URL: https://patchew.org/EDK2/20210129005950.467638-1-ankur.a.ar...@oracle.com/ v5: - fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add add Qemu Cpu Status helper"). URL: https://patchew.org/EDK2/20210126064440.299596-1-ankur.a.ar...@oracle.com/ v4: - Gets rid of unnecessary UefiCpuPkg changes URL: https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.ar...@oracle.com/ v3: - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm and OvmfPkg/CpuHotplugSmm - Cleaner split of the hot-unplug code URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/ v2: - Do the ejection via SmmCpuFeaturesRendezvousExit() URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/ RFC: URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/ Please review. Thanks Ankur Ankur Arora (10): OvmfPkg/CpuHotplugSmm: refactor hotplug logic OvmfPkg/CpuHotplugSmm: collect hot-unplug events OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper OvmfPkg/CpuHotplugSmm: introduce UnplugCpus() OvmfPkg: define CPU_HOT_EJECT_DATA OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler OvmfPkg/CpuHotplugSmm: add EjectCpu() OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug OvmfPkg/OvmfPkg.dec| 4 + OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf| 2 + .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 7 + OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 + OvmfPkg/Include/Pcd/CpuHotEjectData.h | 60 +++ OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 576 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 123 +++-- .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 111 OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 18 +- 10 files changed, 775 insertions(+), 132 deletions(-) create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h -- 2.9.3 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#72702): https://edk2.groups.io/g/devel/message/72702 Mute This Topic: https://groups.io/mt/81273600/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v8 09/10] OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject
On 2021-02-23 1:39 p.m., Laszlo Ersek wrote:
On 02/22/21 08:19, Ankur Arora wrote:
Add logic in EjectCpu() to do the actual the CPU ejection.
On the BSP, ejection happens by first selecting the CPU via
its QemuSelector and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
the CPU out of the SMI handler.
Meanwhile the CPU being ejected, waits around in its holding
area until it is context-switched out. Note that it is possible
that a slow CPU gets ejected before it reaches the wait loop.
However, this would never happen before it has executed the
"AllCpusInSync" loop in SmiRendezvous().
It can mean that an ejected CPU does not execute code after
that point but given that the CPU state will be destroyed by
QEMU, the missed cleanup is no great loss.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following reviewing comments from v6:
(1) s/CpuEject/EjectCpu/g
(2,2a,2c) Get rid of eject-worker and related.
(2b,2d) Use the PlatformSmmBspElection() logic to find out IsBSP.
(3,3b) Use CPU_HOT_EJECT_DATA->QemuSelector instead of ApicIdMap to
do the actual ejection.
(4,5a,5b) Fix the format etc in the final unplugged log message
() Also as discussed elsewhere document the ordering requirements for
mCpuHotEjectData->QemuSelector[] and mCpuHotEjectData->Handler.
() [from patch 2] Move definition of QEMU_CPUHP_STAT_EJECTED to this
patch.
() s/QEMU_CPUHP_STAT_EJECTED/QEMU_CPUHP_STAT_EJECT/
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 127 +++--
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 31 +
3 files changed, 152 insertions(+), 7 deletions(-)
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index 2ec7a107a64d..d0e83102c13f 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_EJECT BIT3
#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 851e2b28aad9..0484be8fe43c 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -18,6 +18,7 @@
#include // CPU_HOT_EJECT_DATA
#include // EFI_MM_CPU_IO_PROTOCOL
#include // EFI_SMM_CPU_SERVICE_PROTOCOL
+#include // MSR_IA32_APIC_BASE_REGISTER
#include// EFI_STATUS
#include "ApicId.h" // APIC_ID
@@ -191,12 +192,39 @@ RevokeNewSlot:
}
/**
+ EjectCpu needs to know the BSP at SMI exit at a point when
+ some of the EFI_SMM_CPU_SERVICE_PROTOCOL state has been torn
+ down.
+ Reuse the logic from OvmfPkg::PlatformSmmBspElection() to
+ do that.
+
+ @param[in] ProcessorNum ProcessorNum denotes the processor handle number
+ in EFI_SMM_CPU_SERVICE_PROTOCOL.
+**/
+STATIC
+BOOLEAN
+CheckIfBsp (
+ IN UINTN ProcessorNum
+ )
(1a) Please remove the ProcessorNum parameter -- comment and parameter
list alike --; it's useless. In the parameter list, just replace the
line's contents with "VOID".
Heh. Yeah, I'm not certain why I added that.
(1b) Additionally, please state:
@retval TRUE If the CPU executing this function is the BSP.
@retval FALSE If the CPU executing this function is an AP.
Will add.
+{
+ MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr;
+ BOOLEAN IsBsp;
(2) "IsBsp" should line up with "ApicBaseMsr".
+
+ ApicBaseMsr.Uint64 = AsmReadMsr64 (MSR_IA32_APIC_BASE);
+ IsBsp = (BOOLEAN)(ApicBaseMsr.Bits.BSP == 1);
+ return IsBsp;
+}
+
+/**
CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
on each CPU at exit from SMM.
- If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is neither the BSP, nor being ejected, nothing
+ to be done.
If, the executing CPU is being ejected, wait in a halted loop
until ejected.
+ If, the executing CPU is the BSP, set QEMU CPU status to eject
+ for CPUs being ejected.
@param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
and will be used as an index into
@@ -211,9 +239,99 @@ EjectCpu (
)
{
UINT64 QemuSelector;
+ BOOLEAN IsBsp;
+ IsBsp = CheckI
Re: [edk2-devel] [PATCH v8 08/10] OvmfPkg/CpuHotplugSmm: add EjectCpu()
On 2021-02-23 12:36 p.m., Laszlo Ersek wrote: superficial comments only; the patch is nearly ready: On 02/22/21 08:19, Ankur Arora wrote: Add EjectCpu(), which handles the CPU ejection, and provides a holding area for said CPUs. It is called via SmmCpuFeaturesRendezvousExit(), at the tail end of the SMI handling. Also UnplugCpus() now stashes QEMU Selectors of CPUs which need to be ejected in CPU_HOT_EJECT_DATA.QemuSelectorMap. This is used by EjectCpu() to identify CPUs marked for ejection. Cc: Laszlo Ersek Cc: Jordan Justen Cc: Ard Biesheuvel Cc: Igor Mammedov Cc: Boris Ostrovsky Cc: Aaron Young Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132 Signed-off-by: Ankur Arora --- Notes: Address these review comments from v6: (1) s/CpuEject/EjectCpu/g (2) Ensure that the added include is in sorted order. (3) Switch to a cheaper CpuSleep() based loop instead of CpuDeadLoop(). Also add the CpuLib LibraryClass. (4) Remove the nested else clause (5) Use Laszlo's much clearer comment when we try to map multiple QemuSelector to the same ProcessorNum. (6a) Fix indentation of the debug print in the block in (5). (6b,6c,6d) Fix printf types for ProcessorNum, use FMT_APIC_ID for APIC_ID and 0x%Lx for QemuSelector[]. () As discussed elsewhere add an DEBUG_INFO print logging the correspondence between ProcessorNum, APIC_ID, QemuSelector. (7a,7b) Use EFI_ALREADY_STARTED instead of EFI_INVALID_PARAMETER and document it in the UnplugCpus() comment block. () As discussed elsewhere, add the import statement for PcdCpuHotEjectDataAddress. (9) Use Laszlo's comment in the PcdGet64(PcdCpuHotEjectDataAddress) description block. (10) Change mCpuHotEjectData init state checks from ASSERT to ones consistent with similar checks for mCpuHotPlugData. (11-14) Get rid of mCpuHotEjectData init loop: moved to a prior patch so it can be done at allocation time. (15) s/SmmCpuFeaturesSmiRendezvousExit/SmmCpuFeaturesRendezvousExit/ (16,17) Document the ordering requirements of mCpuHotEjectData->Handler, and mCpuHotEjectData->QemuSelectorMap. Not addressed: (8) Not removing the EjectCount variable as I'd like to minimize stores/loads to CPU_HOT_EJECT_DATA->Handler and so would like to do this a single time at the end of the iteration. (It is safe to write multiple times to the handler in UnplugCpus() but given the ordering concerns around it, it seems cleaner to not access it unnecessarily.) OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf | 2 + OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 157 ++-- 2 files changed, 151 insertions(+), 8 deletions(-) diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf index 04322b0d7855..ebcc7e2ac63a 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf @@ -40,6 +40,7 @@ [Packages] [LibraryClasses] BaseLib BaseMemoryLib + CpuLib DebugLib LocalApicLib MmServicesTableLib @@ -54,6 +55,7 @@ [Protocols] [Pcd] gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress## CONSUMES + gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress ## CONSUMES gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase ## CONSUMES [FeaturePcd] diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c index f07b5072749a..851e2b28aad9 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c @@ -10,10 +10,12 @@ #include // ICH9_APM_CNT #include // QEMU_CPUHP_CMD_GET_PENDING #include // CpuDeadLoop() +#include // CpuSleep() #include // ASSERT() #include // gMmst #include // PcdGetBool() #include // SafeUintnSub() +#include // CPU_HOT_EJECT_DATA #include // EFI_MM_CPU_IO_PROTOCOL #include // EFI_SMM_CPU_SERVICE_PROTOCOL #include// EFI_STATUS @@ -32,11 +34,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo; // STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService; // -// This structure is a communication side-channel between the +// These structures serve as communication side-channels between the // EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider // (i.e., PiSmmCpuDxeSmm). // STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData; +STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData; // // SMRAM arrays for fetching the APIC IDs of processors with pending events (of // known event types), for the time of just one MMI. @@ -188,18 +191,72 @@ RevokeNewSlot: } /** + CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit() + on each CPU at exit from SMM. + + If, the exec
Re: [edk2-devel] [PATCH v8 07/10] OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
On 2021-02-23 9:18 a.m., Paolo Bonzini wrote:
On 23/02/21 18:06, Laszlo Ersek wrote:
On 02/23/21 08:45, Paolo Bonzini wrote:
On 22/02/21 15:53, Laszlo Ersek wrote:
+
+ if (mCpuHotEjectData != NULL) {
+ CPU_HOT_EJECT_HANDLER Handler;
+
+ Handler = mCpuHotEjectData->Handler;
This patch looks otherwise OK to me, but:
In patch v8 08/10, we have a ReleaseMemoryFence(). (For now, it is only
expressed as a MemoryFence() call; we'll make that more precise later.)
(1) I think that should be paired with an AcquireMemoryFence() call,
just before loading "mCpuHotEjectData->Handler" above -- for now, also
expressed as a MemoryFence() call only.
In Linux terms, there is a control dependency here. However, it should
at least be a separate statement to load mCpuHotEjectData (which from my
EDK2 reminiscences should be a global) into a local variable. So
EjectData = mCPUHotEjectData;
// Optional AcquireMemoryFence here
if (EjectData != NULL) {
CPU_HOT_EJECT_HANDLER Handler;
Handler = EjectData->Handler;
if (Handler != NULL) {
Handler (CpuIndex);
}
}
Yes, "mCPUHotEjectData" is a global.
"mCpuHotEjectData" itself is set up on the BSP (from the entry point
function of the PiSmmCpuSmmDxe driver), before any other APs have a
chance to execute any SMM-related code at all. Furthermore, once set up,
mCpuHotEjectData never changes -- it remains set to a particular
non-NULL value forever, or it remains NULL forever. (The latter case
applies when the possible CPU count is 1; IOW, then there is no AP at all.)
Ok, that's what I was missing. However, your code below has *two* loads of
mCpuHotEjectData and the fence would have to go after the second (between the load of
mCpuHotEjectData and the load of the Handler field). Therefore I would still use a local
variable even if you decide to put the fence inside the "if", which I agree is
the clearest.
Sorry, I'm missing something here. As Laszlo said given that mCpuHotEjectData
does not change after being set, so why would it be a problem in referencing it
twice?
The generated code looks like this (load for mCpuHotEjectData at 0xf54b and
then the dependent mCpuHotEjectData->Handler load on 0xf645):
# 17d60
f54b: 48 8b 05 0e 88 00 00mov0x880e(%rip),%rax
f54e: R_X86_64_PC32 .data+0x1d5c
f552: 48 85 c0test %rax,%rax
f555: 0f 85 ea 00 00 00 jnef645
# Handler = mCpuHotEjectData->Handler
f645: 48 8b 40 08 mov0x8(%rax),%rax
f649: 48 85 c0test %rax,%rax
f64c: 74 05 je f653
f64e: 4c 89 e1mov%r12,%rcx
f651: ff d0 callq *%rax
In the worst case, however, maybe it looks like this (two loads for
mCpuHotEjectData and then the dependent load):
# 17d60
f54b: 48 8b 05 0e 88 00 00mov0x880e(%rip),%rax
f54e: R_X86_64_PC32 .data+0x1d5c
f552: 48 85 c0test %rax,%rax
f555: 0f 85 ea 00 00 00 jnef645
# 17d60
f645: 48 8b 05 0e 88 00 00mov0x880e(%rip),%rax
+3: R_X86_64_PC32 .data+0x1d5c
# Handler = mCpuHotEjectData->Handler
+7: 48 8b 40 08 mov0x8(%rax),%rax
+11: 48 85 c0test %rax,%rax
+14: 74 05 je f653
+16: 4c 89 e1mov%r12,%rcx
+19: ff d0 callq *%rax
As you and Laszlo say -- we do need an acquire fence before this line
(which corresponds to the release fence in UnplugCpus(), patch 8
and the release fence in EjectCpu() in patch 9).
# Handler = mCpuHotEjectData->Handler
48 8b 40 08 mov0x8(%rax),%rax
A local variable for mCpuHotEjectData, would be nice to have but I'm
not sure it is needed for correctness.
Ankur
Paolo
Because of that, I thought that the first comparison (mCpuHotEjectData
!= NULL) would not need any fence -- I thought it was similar to a
userspace program that (a) set a global variable in the "main" thread,
before calling pthread_create(), (b) treated the global variable as a
constant, ever after (meaning all threads).
However, mCpuHotEjectData->Handler is changed regularly (modified by the
BSP, and read "later" by all processors). That's why I thought the
acquire fence was needed in the following location:
if (mCpuHotEjectData != NULL) {
CPU_HOT_EJECT_HANDLER Handler;
//
// HERE -- AcquireMemoryFence()
//
Handler = mCpuHotEjectData->Handler;
if (Handler != NULL) {
Handler (CpuIndex);
}
}
Thanks!
Laszlo
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72114): https://edk2.groups.io/g/devel/message/72114
Mute This Topic: https://groups.io/mt/80819862/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub
Re: [edk2-devel] [PATCH v8 07/10] OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
On 2021-02-22 6:53 a.m., Laszlo Ersek wrote:
Adding Paolo, one comment below:
On 02/22/21 08:19, Ankur Arora wrote:
Call the CPU hot-eject handler if one is installed. The condition for
installation is (PcdCpuMaxLogicalProcessorNumber > 1), and there's
a hot-unplug request.
The handler executes in context of SmmCpuFeaturesRendezvousExit(),
which is called at the tail end of SmiRendezvous() after the BSP has
given the signal to exit via the "AllCpusInSync" loop.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Address the following review comments from v6, patch-6:
(19a) Move the call to the ejection handler to a separate patch.
(19b) Describe the calling context of SmmCpuFeaturesRendezvousExit().
(20) Add comment describing the state when the Handler is not armed.
OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 15 +++
1 file changed, 15 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index adbfc90ad46e..99988285b6a2 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -467,6 +467,21 @@ SmmCpuFeaturesRendezvousExit (
IN UINTN CpuIndex
)
{
+ //
+ // We only call the Handler if CPU hot-eject is enabled
+ // (PcdCpuMaxLogicalProcessorNumber > 1), and hot-eject is needed
+ // in this SMI exit (otherwise mCpuHotEjectData->Handler is not armed.)
+ //
+
+ if (mCpuHotEjectData != NULL) {
+CPU_HOT_EJECT_HANDLER Handler;
+
+Handler = mCpuHotEjectData->Handler;
This patch looks otherwise OK to me, but:
In patch v8 08/10, we have a ReleaseMemoryFence(). (For now, it is only
expressed as a MemoryFence() call; we'll make that more precise later.)
(1) I think that should be paired with an AcquireMemoryFence() call,
just before loading "mCpuHotEjectData->Handler" above -- for now, also
expressed as a MemoryFence() call only.
BTW the first article in Paolo's series has been published:
https://lwn.net/Articles/844224/
so in terms of that, we have something similar to this diagram:
thread 1 thread 2
a.x = 1;
smp_wmb();
WRITE_ONCE(message, ); datum = READ_ONCE(message);
smp_rmb();
if (datum != NULL)
printk("%x\n", datum->x);
Thanks for the link (and Paolo for writing it.) This is great.
In patch 8, UnplugCpus() does the first two lines of the "thread 1"
(BSP) actions, and the third line is covered by the final "AllCpusInSync
= FALSE" assignment in BSPHandler() [UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c].
Regarding the thread#2 (AP) actions, line#1 is covered by the
"AllCpusInSync loop" near the end of SmiRendezvous(). Lines 3+ are
covered by our SmmCpuFeaturesRendezvousExit() implementation here. But
line#2 (the AcquireMemoryFence()) is missing.
Yeah you are right. Just think out aloud here... without this it is possible
that on the the AP, the CPU could reorder loads on line-1 and line-3.
This patch does need an AcquireMemoryFence() (or a MemoryFence() and a
comment stating that it needs acquire semantics.
This also makes me realize that although I have somewhat detailed comments
in patches 8 and 9, but I do need to specify which fence needs to have
acquire semantics and which release.
... I'll suspend the review at this point for today; let's see whether
we agree on the comments I've made so far. I hope to continue the review
tomorrow.
Agreed so far! And, thanks.
Ankur
Thanks!
Laszlo
+
+if (Handler != NULL) {
+ Handler (CpuIndex);
+}
+ }
}
/**
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72035): https://edk2.groups.io/g/devel/message/72035
Mute This Topic: https://groups.io/mt/80819862/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v8 06/10] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
On 2021-02-22 6:19 a.m., Laszlo Ersek wrote:
On 02/22/21 08:19, Ankur Arora wrote:
Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection
state between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
The init happens via SmmCpuFeaturesSmmRelocationComplete(), and so it
will run as part of the PiSmmCpuDxeSmm entry point function,
PiCpuSmmEntry(). Once inited, CPU_HOT_EJECT_DATA is exposed via
PcdCpuHotEjectDataAddress.
The CPU hot-eject handler (CPU_HOT_EJECT_DATA->Handler) is setup when
there is an ejection request via CpuHotplugSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments:
(1) Detail in commit message about context in which CPU_HOT_EJECT_DATA
is inited.
(2) Add in sorted order MemoryAllocationLib in LibraryClasses
(3) Sort added includes in SmmCpuFeaturesLib.c
(4a-4b) Fixup linkage directives for mCpuHotEjectData.
(5) s/CpuHotEjectData/mCpuHotEjectData/
(6,10a,10b) Remove dependence on PcdCpuHotPlugSupport
(7) Make the tense structure consistent in block comment for
InitCpuHotEject().
(8) s/SmmCpuFeaturesSmmInitHotEject/InitCpuHotEject/
(9) s/mMaxNumberOfCpus/MaxNumberOfCpus/
(11) Remove a bunch of obvious comments.
(14a,14b,14c) Use SafeUint functions and rework the allocation logic
so we can just use a single allocation.
(12) Remove the AllocatePool() cast.
(13) Use a CpuDeadLoop() in case of failure; albeit via a goto, not
inline.
(15) Initialize the mCpuHotEjectData->QemuSelectorMap locally.
(16) Fix indentation in PcdSet64S.
(17) Change the cast in PcdSet64S() to UINTN.
(18) Use RETURN_STATUS instead of EFI_STATUS.
(19,20) Move the Handler logic in SmmCpuFeaturesRendezvousExit() into
into a separate patch.
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 +
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 92 ++
2 files changed, 96 insertions(+)
Nice, I've only superficial comments:
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..8a426a4c10fb 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -30,9 +30,13 @@ [LibraryClasses]
BaseMemoryLib
DebugLib
MemEncryptSevLib
+ MemoryAllocationLib
PcdLib
+ SafeIntLib
SmmServicesTableLib
UefiBootServicesTableLib
[Pcd]
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..adbfc90ad46e 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -11,10 +11,13 @@
#include
#include
#include
+#include
#include
+#include
#include
#include
#include
+#include
#include
#include
#include
@@ -171,6 +174,92 @@ SmmCpuFeaturesHookReturnFromSmm (
return OriginalInstructionPointer;
}
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+/**
+ Initialize mCpuHotEjectData if PcdCpuMaxLogicalProcessorNumber > 1.
+
+ Also setup the corresponding PcdCpuHotEjectDataAddress.
+**/
+STATIC
+VOID
+InitCpuHotEjectData (
+ VOID
+ )
+{
+ UINTN ArrayLen;
+ UINTN BaseLen;
+ UINTN TotalLen;
+ UINT32 Idx;
+ UINT32 MaxNumberOfCpus;
+ RETURN_STATUS PcdStatus;
+
+ MaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+
(1) This doesn't deserve an update in itself, but since I'll ask for
some more below, I might as well name it: please remove the empty line
here.
+ if (MaxNumberOfCpus == 1) {
+return;
+ }
+
+ //
+ // We want the following lay out for CPU_HOT_EJECT_DATA:
+ // UINTN alignment: CPU_HOT_EJECT_DATA
+ // --- padding if needed ---
+ // UINT64 alignment: CPU_HOT_EJECT_DATA->QemuSelectorMap[]
+ //
+ // Accordingly, we allocate:
+ // sizeof(*mCpuHotEjectData) + (MaxNumberOfCpus *
+ // sizeof(mCpuHotEjectData->QemuSelectorMap[0])).
+ // Add sizeof(UINT64) to use as padding if needed.
+ //
+
+ if (RETURN_ERROR (SafeUintnMult (sizeof (*mCpuHotEjectData), 1, )) ||
(2) This "multiplication" is somewhat awkward.
First, BaseLen is a UINTN, which always matches the return type of the
sizeof operator, so a direct assignment would be OK.
But even that would be overkill -- I suggest removing the BaseLen
variable, and just using "sizeof (*mCpuHotEjectData)" i
Re: [edk2-devel] [PATCH v8 05/10] OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
On 2021-02-22 5:06 a.m., Laszlo Ersek wrote:
On 02/22/21 08:19, Ankur Arora wrote:
Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments in v6:
(1) Dropped modifications to LibraryClasses in OvmfPkg.dec
(2,3) Cleanup comments around PCD PcdCpuHotEjectDataAddress.
(4) Move PCD PcdCpuHotEjectDataAddress declaration in CpuHotplugSmm.inf
to a patch-7 where it actually gets used.
(5a,5b) Change the comment in the top block to use Laszlo's language.
Also detail when the PCD would contain a valid value.
(6) Move Library/CpuHotEjectData.h to Pcd/CpuHotEjectData.h
(7,15,16) Fixup guard macro to be C namespace compliant. Also fixup the
comment style near the endif guard.
(8-10) Rename CPU_HOT_EJECT_FN to a more EDK2 compliant style. Also add
a comment block and fix spacing.
() Rename ApicIdMap -> QemuSelectorMap while keeping the type as UINT64.
Related to a comment in patch-8 ("... add worker to do CPU ejection".)
(11a,11b) Rename CPU_EJECT_INVALID to CPU_EJECT_QEMU_SELECTOR_INVALID
and add a comment about it.
() Remove CPU_EJECT_WORKER based on review comment on a patch 8.
(12,14) Remove CPU_HOT_EJECT_DATA fields Revision and Reserved.
Reorder CPU_HOT_EJECT_DATA to minimize internal padding
and ensure elements are properly aligned.
(13a,13b) Change CpuIndex->ApicId map to ProcessorNum -> QemuSelector
() Make CPU_HOT_EJECT_HANDLER->Handler,
CPU_HOT_EJECT_HANDLER->QemuSelectorMap volatile.
OvmfPkg/OvmfPkg.dec | 4 +++
OvmfPkg/Include/Pcd/CpuHotEjectData.h | 52 +++
2 files changed, 56 insertions(+)
create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h
Very nice updates; I only have superficial comments this time:
(1) The patch -- correctly -- no longer touches CpuHotplugSmm, so please
remove "/CpuHotplugSmm" from the subject line.
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 4348bb45c64a..9629707020ba 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -352,6 +352,10 @@ [PcdsDynamic, PcdsDynamicEx]
# This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+ ## This PCD adds a communication channel between OVMF's SmmCpuFeaturesLib
+ # instance in PiSmmCpuDxeSmm, and CpuHotplugSmm.
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
+
[PcdsFeatureFlag]
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/Include/Pcd/CpuHotEjectData.h
b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
new file mode 100644
index ..024a92726869
--- /dev/null
+++ b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
@@ -0,0 +1,52 @@
+/** @file
+ Definition for the CPU_HOT_EJECT_DATA structure, which shares
+ CPU hot-eject state between OVMF's SmmCpuFeaturesLib instance in
+ PiSmmCpuDxeSmm, and CpuHotplugSmm.
+
+ CPU_HOT_EJECT_DATA is allocated in SMRAM, and pointed-to by
+ PcdCpuHotEjectDataAddress.
+
+ PcdCpuHotEjectDataAddress is valid when SMM_REQUIRE is TRUE
+ and MaxNumberOfCpus > 1.
(2) Looks good, thanks; please clarify it as follows though:
s/MaxNumberOfCpus/PcdCpuMaxLogicalProcessorNumber/
+
+ Copyright (C) 2021, Oracle Corporation.
+
+ SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#ifndef CPU_HOT_EJECT_DATA_H_
+#define CPU_HOT_EJECT_DATA_H_
+
+/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
+ on each CPU at exit from SMM.
+
+ @param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
+ and will be used as an index into
+ CPU_HOT_EJECT_DATA->QemuSelectorMap. It is
+ identical to the processor handle in
+ EFI_SMM_CPU_SERVICE_PROTOCOL.
+**/
+typedef
+VOID
+(EFIAPI *CPU_HOT_EJECT_HANDLER) (
+ IN UINTN ProcessorNum
+ );
+
+//
+// CPU_EJECT_QEMU_SELECTOR_INVALID marks CPUs not being ejected in
+// CPU_HOT_EJECT_DATA->QemuSelectorMap.
+//
+// QEMU CPU Selector is UINT32, so we choose an invalid value larger
+// than that type.
+//
+#define CPU_EJECT_QEMU_SELECTOR_INVALID (MAX_UINT64)
+
+typedef struct {
+ volatile UINT64 *QemuSelectorMap; // Maps ProcessorNum -> QemuSelector
+ // for pending hot-ejects
+ volatile CPU_HOT_
Re: [edk2-devel] [PATCH v8 04/10] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
On 2021-02-22 4:39 a.m., Laszlo Ersek wrote:
On 02/22/21 08:19, Ankur Arora wrote:
Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().
With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses these review comments from v6:
(1) Drop the empty line in the comment block around UnplugCpus().
(2) Make the "did not find APIC ID" DEBUG_VERBOSE instead of DEBUG_INFO.
(3) Un-Indented ("Outdented") the line following the comment "Ignore the
unplug if APIC ID.
(4) Remove the empty line between Status assignment and check.
(5) Drop the "goto Fatal" logic and just return Status directly.
(6) Handle both Plugging and Unplugging of CPUs in one go.
(7) Also nest the EFI_STATUS check.
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 84 ++
1 file changed, 84 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 3192bfea1f15..f07b5072749a 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -188,6 +188,83 @@ RevokeNewSlot:
}
/**
+ Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, report the CPU to PiSmmCpuDxeSmm via
+ EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is
+ unknown, skip it silently.
+
+ @param[in] ToUnplugApicIdsThe APIC IDs of the CPUs that are about to be
+hot-unplugged.
+
+ @param[in] ToUnplugCount The number of filled-in APIC IDs in
+ToUnplugApicIds.
+
+ @retval EFI_SUCCESS Known APIC IDs have been removed from SMM data
+structures.
+
+ @return Error codes propagated from
+mMmCpuService->RemoveProcessor().
+**/
+STATIC
+EFI_STATUS
+UnplugCpus (
+ IN APIC_ID *ToUnplugApicIds,
+ IN UINT32 ToUnplugCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 ToUnplugIdx;
+ UINTN ProcessorNum;
+
+ ToUnplugIdx = 0;
+ while (ToUnplugIdx < ToUnplugCount) {
+APIC_IDRemoveApicId;
+
+RemoveApicId = ToUnplugApicIds[ToUnplugIdx];
+
+//
+// mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+// the ProcessorNum for the APIC ID to be removed.
+//
+for (ProcessorNum = 0;
+ ProcessorNum < mCpuHotPlugData->ArrayLength;
+ ProcessorNum++) {
+ if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+break;
+ }
+}
+
+//
+// Ignore the unplug if APIC ID not found
+//
+if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: did not find APIC ID " FMT_APIC_ID
+" to unplug\n", __FUNCTION__, RemoveApicId));
+ ToUnplugIdx++;
+ continue;
+}
+
+//
+// Mark ProcessorNum for removal from SMM data structures
+//
+Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, RemoveApicId, Status));
+ return Status;
+}
+
+ToUnplugIdx++;
+ }
+
+ //
+ // We've removed this set of APIC IDs from SMM data structures.
+ //
+ return EFI_SUCCESS;
+}
+
+/**
CPU Hotplug MMI handler function.
This is a root MMI handler.
@@ -309,6 +386,13 @@ CpuHotplugMmi (
}
}
+ if (ToUnplugCount > 0) {
+Status = UnplugCpus (mToUnplugApicIds, ToUnplugCount);
+if (EFI_ERROR (Status)) {
+ goto Fatal;
+}
+ }
+
//
// We've handled this MMI.
//
Reviewed-by: Laszlo Ersek
Thanks for the review!
Ankur
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72003): https://edk2.groups.io/g/devel/message/72003
Mute This Topic: https://groups.io/mt/80819859/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v8 03/10] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
On 2021-02-22 4:31 a.m., Laszlo Ersek wrote:
On 02/22/21 08:19, Ankur Arora wrote:
Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Address this review comment:
() Move QemuCpuhpWriteCpuStatus() (declaration and definition) between
QemuCpuhpWriteCpuSelector() and QemuCpuhpWriteCommand() to match
the order of the register descriptions in QEMU.
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 22 ++
2 files changed, 28 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 1e23b150910e..859412c1a173 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -42,6 +42,12 @@ QemuCpuhpWriteCpuSelector (
);
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ );
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 36372a5e6193..9434bb14dd4e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -114,6 +114,28 @@ QemuCpuhpWriteCpuSelector (
}
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ )
+{
+ EFI_STATUS Status;
+
+ Status = MmCpuIo->Io.Write (
+ MmCpuIo,
+ MM_IO_UINT8,
+ ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+ 1,
+
+ );
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+ASSERT (FALSE);
+CpuDeadLoop ();
+ }
+}
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
Reviewed-by: Laszlo Ersek
Thanks!
Ankur
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72002): https://edk2.groups.io/g/devel/message/72002
Mute This Topic: https://groups.io/mt/80819857/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v8 02/10] OvmfPkg/CpuHotplugSmm: collect hot-unplug events
On 2021-02-22 4:27 a.m., Laszlo Ersek wrote:
On 02/22/21 08:19, Ankur Arora wrote:
Process fw_remove events in QemuCpuhpCollectApicIds() and collect
corresponding APIC IDs for CPUs that are being hot-unplugged.
(1) We also collect selectors for those; please mention the fact here.
In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.
This is based on the QEMU hot-unplug protocol documented here:
https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imamm...@redhat.com/
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments from v6:
(1,4) Move (and also rename) QEMU_CPUHP_STAT_EJECTED to patch 8,
where we actually use it.
(2) Downgrade debug mask from DEBUG_INFO to DEBUG_VERBOSE.
(3a,3b,3c) Keep the CurrentSelector increment operation at
the tail of the loop.
() As discussed elsewhere we also need to get the CpuSelector while
collecting ApicIds in QemuCpuhpCollectApicIds(). This patch adds a
separate parameter for the CpuSelector values, because that works
better alongside the hotplug ExtendIds logic.
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 1 +
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c| 21 +-
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 84 ---
4 files changed, 79 insertions(+), 28 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..1e23b150910e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -55,6 +55,7 @@ QemuCpuhpCollectApicIds (
OUT APIC_ID *PluggedApicIds,
OUT UINT32 *PluggedCount,
OUT APIC_ID *ToUnplugApicIds,
+ OUT UINT32 *ToUnplugSelector,
OUT UINT32 *ToUnplugCount
);
(2) For consistency with the parameter names "PluggedApicIds" (plural)
and "ToUnplugApicIds" (plural), please rename this parameter to
"ToUnplugSelectors".
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..2ec7a107a64d 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index bf68fcd42914..3192bfea1f15 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -52,6 +52,7 @@ STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
//
STATIC APIC_ID *mPluggedApicIds;
STATIC APIC_ID *mToUnplugApicIds;
+STATIC UINT32 *mToUnplugSelector;
(3) For consistency with the other two global variable identifiers,
please call this "mToUnplugSelector" (plural), too.
(4) The comment above these global variables only talks about APIC IDs;
please update the comment with a reference / explanation of the selectors.
//
// Address of the non-SMRAM reserved memory page that contains the Post-SMM
Pen
// for hot-added CPUs.
@@ -289,6 +290,7 @@ CpuHotplugMmi (
mPluggedApicIds,
,
mToUnplugApicIds,
+ mToUnplugSelector,
);
if (EFI_ERROR (Status)) {
@@ -333,7 +335,9 @@ CpuHotplugEntry (
)
{
EFI_STATUS Status;
+ UINTN Len;
UINTN Size;
+ UINTN SizeSel;
//
// This module should only be included when SMM support is required.
@@ -387,8 +391,9 @@ CpuHotplugEntry (
//
// Allocate the data structures that depend on the possible CPU count.
//
- if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
- RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Size, ))) {
+ if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
+ RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Len, ))||
(5) Missing space character before "||".
+ RETURN_ERROR (SafeUintnMult (sizeof (UINT32), Len, ))) {
Status = EFI_ABORTED;
DEBUG ((DEBUG_ERROR, "%a: invalid CPU_HOT_PLUG_DATA\n", __FUNCTION__));
goto Fatal;
@@ -405,6 +410,12 @@ CpuHotplugEntry (
DEBUG ((DEBUG_ERROR, "%a: MmAllocatePool(): %r\n", __FUNCTION__, Status));
goto ReleasePluggedApicI
[edk2-devel] [PATCH v8 09/10] OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject
Add logic in EjectCpu() to do the actual the CPU ejection.
On the BSP, ejection happens by first selecting the CPU via
its QemuSelector and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
the CPU out of the SMI handler.
Meanwhile the CPU being ejected, waits around in its holding
area until it is context-switched out. Note that it is possible
that a slow CPU gets ejected before it reaches the wait loop.
However, this would never happen before it has executed the
"AllCpusInSync" loop in SmiRendezvous().
It can mean that an ejected CPU does not execute code after
that point but given that the CPU state will be destroyed by
QEMU, the missed cleanup is no great loss.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following reviewing comments from v6:
(1) s/CpuEject/EjectCpu/g
(2,2a,2c) Get rid of eject-worker and related.
(2b,2d) Use the PlatformSmmBspElection() logic to find out IsBSP.
(3,3b) Use CPU_HOT_EJECT_DATA->QemuSelector instead of ApicIdMap to
do the actual ejection.
(4,5a,5b) Fix the format etc in the final unplugged log message
() Also as discussed elsewhere document the ordering requirements for
mCpuHotEjectData->QemuSelector[] and mCpuHotEjectData->Handler.
() [from patch 2] Move definition of QEMU_CPUHP_STAT_EJECTED to this
patch.
() s/QEMU_CPUHP_STAT_EJECTED/QEMU_CPUHP_STAT_EJECT/
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 127 +++--
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 31 +
3 files changed, 152 insertions(+), 7 deletions(-)
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index 2ec7a107a64d..d0e83102c13f 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_EJECT BIT3
#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 851e2b28aad9..0484be8fe43c 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -18,6 +18,7 @@
#include // CPU_HOT_EJECT_DATA
#include // EFI_MM_CPU_IO_PROTOCOL
#include // EFI_SMM_CPU_SERVICE_PROTOCOL
+#include // MSR_IA32_APIC_BASE_REGISTER
#include// EFI_STATUS
#include "ApicId.h" // APIC_ID
@@ -191,12 +192,39 @@ RevokeNewSlot:
}
/**
+ EjectCpu needs to know the BSP at SMI exit at a point when
+ some of the EFI_SMM_CPU_SERVICE_PROTOCOL state has been torn
+ down.
+ Reuse the logic from OvmfPkg::PlatformSmmBspElection() to
+ do that.
+
+ @param[in] ProcessorNum ProcessorNum denotes the processor handle number
+ in EFI_SMM_CPU_SERVICE_PROTOCOL.
+**/
+STATIC
+BOOLEAN
+CheckIfBsp (
+ IN UINTN ProcessorNum
+ )
+{
+ MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr;
+ BOOLEAN IsBsp;
+
+ ApicBaseMsr.Uint64 = AsmReadMsr64 (MSR_IA32_APIC_BASE);
+ IsBsp = (BOOLEAN)(ApicBaseMsr.Bits.BSP == 1);
+ return IsBsp;
+}
+
+/**
CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
on each CPU at exit from SMM.
- If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is neither the BSP, nor being ejected, nothing
+ to be done.
If, the executing CPU is being ejected, wait in a halted loop
until ejected.
+ If, the executing CPU is the BSP, set QEMU CPU status to eject
+ for CPUs being ejected.
@param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
and will be used as an index into
@@ -211,9 +239,99 @@ EjectCpu (
)
{
UINT64 QemuSelector;
+ BOOLEAN IsBsp;
+ IsBsp = CheckIfBsp (ProcessorNum);
+
+ //
+ // mCpuHotEjectData->QemuSelectorMap[ProcessorNum] is updated
+ // on the BSP in the ongoing SMI iteration at two places:
+ //
+ // - UnplugCpus() where the BSP determines if a CPU is under ejection
+ // or not. As the comment where mCpuHotEjectData->Handler is set-up
+ // describes any such updates are guaranteed to be ordered-before the
+ // dereference below.
+ //
+ // - EjectCpu() on the BSP updates QemuSelectorMap[ProcessorNum] for
+ // CPUs after they have been hot-ejected.
+ //
+ // The CPU under ejection: might be executing anywhere between the
+ //
[edk2-devel] [PATCH v8 06/10] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection
state between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
The init happens via SmmCpuFeaturesSmmRelocationComplete(), and so it
will run as part of the PiSmmCpuDxeSmm entry point function,
PiCpuSmmEntry(). Once inited, CPU_HOT_EJECT_DATA is exposed via
PcdCpuHotEjectDataAddress.
The CPU hot-eject handler (CPU_HOT_EJECT_DATA->Handler) is setup when
there is an ejection request via CpuHotplugSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments:
(1) Detail in commit message about context in which CPU_HOT_EJECT_DATA
is inited.
(2) Add in sorted order MemoryAllocationLib in LibraryClasses
(3) Sort added includes in SmmCpuFeaturesLib.c
(4a-4b) Fixup linkage directives for mCpuHotEjectData.
(5) s/CpuHotEjectData/mCpuHotEjectData/
(6,10a,10b) Remove dependence on PcdCpuHotPlugSupport
(7) Make the tense structure consistent in block comment for
InitCpuHotEject().
(8) s/SmmCpuFeaturesSmmInitHotEject/InitCpuHotEject/
(9) s/mMaxNumberOfCpus/MaxNumberOfCpus/
(11) Remove a bunch of obvious comments.
(14a,14b,14c) Use SafeUint functions and rework the allocation logic
so we can just use a single allocation.
(12) Remove the AllocatePool() cast.
(13) Use a CpuDeadLoop() in case of failure; albeit via a goto, not
inline.
(15) Initialize the mCpuHotEjectData->QemuSelectorMap locally.
(16) Fix indentation in PcdSet64S.
(17) Change the cast in PcdSet64S() to UINTN.
(18) Use RETURN_STATUS instead of EFI_STATUS.
(19,20) Move the Handler logic in SmmCpuFeaturesRendezvousExit() into
into a separate patch.
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 +
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 92 ++
2 files changed, 96 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..8a426a4c10fb 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -30,9 +30,13 @@ [LibraryClasses]
BaseMemoryLib
DebugLib
MemEncryptSevLib
+ MemoryAllocationLib
PcdLib
+ SafeIntLib
SmmServicesTableLib
UefiBootServicesTableLib
[Pcd]
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..adbfc90ad46e 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -11,10 +11,13 @@
#include
#include
#include
+#include
#include
+#include
#include
#include
#include
+#include
#include
#include
#include
@@ -171,6 +174,92 @@ SmmCpuFeaturesHookReturnFromSmm (
return OriginalInstructionPointer;
}
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+/**
+ Initialize mCpuHotEjectData if PcdCpuMaxLogicalProcessorNumber > 1.
+
+ Also setup the corresponding PcdCpuHotEjectDataAddress.
+**/
+STATIC
+VOID
+InitCpuHotEjectData (
+ VOID
+ )
+{
+ UINTN ArrayLen;
+ UINTN BaseLen;
+ UINTN TotalLen;
+ UINT32 Idx;
+ UINT32 MaxNumberOfCpus;
+ RETURN_STATUS PcdStatus;
+
+ MaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+
+ if (MaxNumberOfCpus == 1) {
+return;
+ }
+
+ //
+ // We want the following lay out for CPU_HOT_EJECT_DATA:
+ // UINTN alignment: CPU_HOT_EJECT_DATA
+ // --- padding if needed ---
+ // UINT64 alignment: CPU_HOT_EJECT_DATA->QemuSelectorMap[]
+ //
+ // Accordingly, we allocate:
+ // sizeof(*mCpuHotEjectData) + (MaxNumberOfCpus *
+ // sizeof(mCpuHotEjectData->QemuSelectorMap[0])).
+ // Add sizeof(UINT64) to use as padding if needed.
+ //
+
+ if (RETURN_ERROR (SafeUintnMult (sizeof (*mCpuHotEjectData), 1, )) ||
+ RETURN_ERROR (SafeUintnMult (
+ sizeof (mCpuHotEjectData->QemuSelectorMap[0]),
+ MaxNumberOfCpus, )) ||
+ RETURN_ERROR (SafeUintnAdd (BaseLen, ArrayLen, ))||
+ RETURN_ERROR (SafeUintnAdd (TotalLen, sizeof (UINT64), ))) {
+DEBUG ((DEBUG_ERROR, "%a: invalid CPU_HOT_EJECT_DATA\n", __FUNCTION__));
+goto Fatal;
+ }
+
+ mCpuHotEjectData = AllocatePool (TotalLen);
+ if (mCpuHotEjectData == NULL) {
+ASSERT (mCpuHotEjectData != NULL);
+goto Fatal;
+ }
+
+ mCpuHotEjectData->Handler = NULL;
+ mCpuHotEjectData->ArrayLength = MaxNumberOfCpus;
+
+
[edk2-devel] [PATCH v8 10/10] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
Advertise OVMF support for CPU hot-unplug and negotiate it
if QEMU requests the feature.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments:
(1,3) s/hot unplug/hot-unplug/
(2) Get rid of the reference to the made up ICH9_APM_CNT_CPU_HOT_UNPLUG
(4,6) Remove the artificial tie in between
ICH9_LPC_SMI_F_CPU_HOTPLUG, ICH9_LPC_SMI_F_CPU_HOT_UNPLUG.
(5) Fully spell out "SMI on CPU hot-unplug".
(7) Emit separate messages on negotiation (or not) of
ICH9_LPC_SMI_F_CPU_HOT_UNPLUG.
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 18 +-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
index c9d875543205..b1d59a559dae 100644
--- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
+++ b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
@@ -29,6 +29,12 @@
//
#define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
+// The following bit value stands for "enable CPU hot-unplug, and inject an SMI
+// with control value ICH9_APM_CNT_CPU_HOTPLUG upon hot-unplug", in the
+// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
+//
+#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
+
//
// Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
// for the S3 boot script fragment to write to and read from.
@@ -112,7 +118,8 @@ NegotiateSmiFeatures (
QemuFwCfgReadBytes (sizeof mSmiFeatures, );
//
- // We want broadcast SMI, SMI on CPU hotplug, and nothing else.
+ // We want broadcast SMI, SMI on CPU hotplug, SMI on CPU hot-unplug
+ // and nothing else.
//
RequestedFeaturesMask = ICH9_LPC_SMI_F_BROADCAST;
if (!MemEncryptSevIsEnabled ()) {
@@ -120,8 +127,10 @@ NegotiateSmiFeatures (
// For now, we only support hotplug with SEV disabled.
//
RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOTPLUG;
+RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
}
mSmiFeatures &= RequestedFeaturesMask;
+
QemuFwCfgSelectItem (mRequestedFeaturesItem);
QemuFwCfgWriteBytes (sizeof mSmiFeatures, );
@@ -166,6 +175,13 @@ NegotiateSmiFeatures (
__FUNCTION__));
}
+ if ((mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG) == 0) {
+DEBUG ((DEBUG_INFO, "%a: CPU hot-unplug not negotiated\n", __FUNCTION__));
+ } else {
+DEBUG ((DEBUG_INFO, "%a: CPU hot-unplug with SMI negotiated\n",
+ __FUNCTION__));
+ }
+
//
// Negotiation successful (although we may not have gotten the optimal
// feature set).
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71921): https://edk2.groups.io/g/devel/message/71921
Mute This Topic: https://groups.io/mt/80819865/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v8 07/10] OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
Call the CPU hot-eject handler if one is installed. The condition for
installation is (PcdCpuMaxLogicalProcessorNumber > 1), and there's
a hot-unplug request.
The handler executes in context of SmmCpuFeaturesRendezvousExit(),
which is called at the tail end of SmiRendezvous() after the BSP has
given the signal to exit via the "AllCpusInSync" loop.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Address the following review comments from v6, patch-6:
(19a) Move the call to the ejection handler to a separate patch.
(19b) Describe the calling context of SmmCpuFeaturesRendezvousExit().
(20) Add comment describing the state when the Handler is not armed.
OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 15 +++
1 file changed, 15 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index adbfc90ad46e..99988285b6a2 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -467,6 +467,21 @@ SmmCpuFeaturesRendezvousExit (
IN UINTN CpuIndex
)
{
+ //
+ // We only call the Handler if CPU hot-eject is enabled
+ // (PcdCpuMaxLogicalProcessorNumber > 1), and hot-eject is needed
+ // in this SMI exit (otherwise mCpuHotEjectData->Handler is not armed.)
+ //
+
+ if (mCpuHotEjectData != NULL) {
+CPU_HOT_EJECT_HANDLER Handler;
+
+Handler = mCpuHotEjectData->Handler;
+
+if (Handler != NULL) {
+ Handler (CpuIndex);
+}
+ }
}
/**
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71918): https://edk2.groups.io/g/devel/message/71918
Mute This Topic: https://groups.io/mt/80819862/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v8 08/10] OvmfPkg/CpuHotplugSmm: add EjectCpu()
Add EjectCpu(), which handles the CPU ejection, and provides a holding area for said CPUs. It is called via SmmCpuFeaturesRendezvousExit(), at the tail end of the SMI handling. Also UnplugCpus() now stashes QEMU Selectors of CPUs which need to be ejected in CPU_HOT_EJECT_DATA.QemuSelectorMap. This is used by EjectCpu() to identify CPUs marked for ejection. Cc: Laszlo Ersek Cc: Jordan Justen Cc: Ard Biesheuvel Cc: Igor Mammedov Cc: Boris Ostrovsky Cc: Aaron Young Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132 Signed-off-by: Ankur Arora --- Notes: Address these review comments from v6: (1) s/CpuEject/EjectCpu/g (2) Ensure that the added include is in sorted order. (3) Switch to a cheaper CpuSleep() based loop instead of CpuDeadLoop(). Also add the CpuLib LibraryClass. (4) Remove the nested else clause (5) Use Laszlo's much clearer comment when we try to map multiple QemuSelector to the same ProcessorNum. (6a) Fix indentation of the debug print in the block in (5). (6b,6c,6d) Fix printf types for ProcessorNum, use FMT_APIC_ID for APIC_ID and 0x%Lx for QemuSelector[]. () As discussed elsewhere add an DEBUG_INFO print logging the correspondence between ProcessorNum, APIC_ID, QemuSelector. (7a,7b) Use EFI_ALREADY_STARTED instead of EFI_INVALID_PARAMETER and document it in the UnplugCpus() comment block. () As discussed elsewhere, add the import statement for PcdCpuHotEjectDataAddress. (9) Use Laszlo's comment in the PcdGet64(PcdCpuHotEjectDataAddress) description block. (10) Change mCpuHotEjectData init state checks from ASSERT to ones consistent with similar checks for mCpuHotPlugData. (11-14) Get rid of mCpuHotEjectData init loop: moved to a prior patch so it can be done at allocation time. (15) s/SmmCpuFeaturesSmiRendezvousExit/SmmCpuFeaturesRendezvousExit/ (16,17) Document the ordering requirements of mCpuHotEjectData->Handler, and mCpuHotEjectData->QemuSelectorMap. Not addressed: (8) Not removing the EjectCount variable as I'd like to minimize stores/loads to CPU_HOT_EJECT_DATA->Handler and so would like to do this a single time at the end of the iteration. (It is safe to write multiple times to the handler in UnplugCpus() but given the ordering concerns around it, it seems cleaner to not access it unnecessarily.) OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf | 2 + OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 157 ++-- 2 files changed, 151 insertions(+), 8 deletions(-) diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf index 04322b0d7855..ebcc7e2ac63a 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf @@ -40,6 +40,7 @@ [Packages] [LibraryClasses] BaseLib BaseMemoryLib + CpuLib DebugLib LocalApicLib MmServicesTableLib @@ -54,6 +55,7 @@ [Protocols] [Pcd] gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress## CONSUMES + gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress ## CONSUMES gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase ## CONSUMES [FeaturePcd] diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c index f07b5072749a..851e2b28aad9 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c @@ -10,10 +10,12 @@ #include // ICH9_APM_CNT #include // QEMU_CPUHP_CMD_GET_PENDING #include // CpuDeadLoop() +#include // CpuSleep() #include // ASSERT() #include // gMmst #include // PcdGetBool() #include // SafeUintnSub() +#include // CPU_HOT_EJECT_DATA #include // EFI_MM_CPU_IO_PROTOCOL #include // EFI_SMM_CPU_SERVICE_PROTOCOL #include// EFI_STATUS @@ -32,11 +34,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo; // STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService; // -// This structure is a communication side-channel between the +// These structures serve as communication side-channels between the // EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider // (i.e., PiSmmCpuDxeSmm). // STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData; +STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData; // // SMRAM arrays for fetching the APIC IDs of processors with pending events (of // known event types), for the time of just one MMI. @@ -188,18 +191,72 @@ RevokeNewSlot: } /** + CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit() + on each CPU at exit from SMM. + + If, the executing CPU is not being ejected, nothing to be done. + If, the executing CPU is being ejected, wait in a halted loop + until ejected. + + @param[in] ProcessorNum ProcessorNum denotes the CPU ex
[edk2-devel] [PATCH v8 04/10] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().
With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses these review comments from v6:
(1) Drop the empty line in the comment block around UnplugCpus().
(2) Make the "did not find APIC ID" DEBUG_VERBOSE instead of DEBUG_INFO.
(3) Un-Indented ("Outdented") the line following the comment "Ignore the
unplug if APIC ID.
(4) Remove the empty line between Status assignment and check.
(5) Drop the "goto Fatal" logic and just return Status directly.
(6) Handle both Plugging and Unplugging of CPUs in one go.
(7) Also nest the EFI_STATUS check.
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 84 ++
1 file changed, 84 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 3192bfea1f15..f07b5072749a 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -188,6 +188,83 @@ RevokeNewSlot:
}
/**
+ Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, report the CPU to PiSmmCpuDxeSmm via
+ EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is
+ unknown, skip it silently.
+
+ @param[in] ToUnplugApicIdsThe APIC IDs of the CPUs that are about to be
+hot-unplugged.
+
+ @param[in] ToUnplugCount The number of filled-in APIC IDs in
+ToUnplugApicIds.
+
+ @retval EFI_SUCCESS Known APIC IDs have been removed from SMM data
+structures.
+
+ @return Error codes propagated from
+mMmCpuService->RemoveProcessor().
+**/
+STATIC
+EFI_STATUS
+UnplugCpus (
+ IN APIC_ID *ToUnplugApicIds,
+ IN UINT32 ToUnplugCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 ToUnplugIdx;
+ UINTN ProcessorNum;
+
+ ToUnplugIdx = 0;
+ while (ToUnplugIdx < ToUnplugCount) {
+APIC_IDRemoveApicId;
+
+RemoveApicId = ToUnplugApicIds[ToUnplugIdx];
+
+//
+// mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+// the ProcessorNum for the APIC ID to be removed.
+//
+for (ProcessorNum = 0;
+ ProcessorNum < mCpuHotPlugData->ArrayLength;
+ ProcessorNum++) {
+ if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+break;
+ }
+}
+
+//
+// Ignore the unplug if APIC ID not found
+//
+if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: did not find APIC ID " FMT_APIC_ID
+" to unplug\n", __FUNCTION__, RemoveApicId));
+ ToUnplugIdx++;
+ continue;
+}
+
+//
+// Mark ProcessorNum for removal from SMM data structures
+//
+Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, RemoveApicId, Status));
+ return Status;
+}
+
+ToUnplugIdx++;
+ }
+
+ //
+ // We've removed this set of APIC IDs from SMM data structures.
+ //
+ return EFI_SUCCESS;
+}
+
+/**
CPU Hotplug MMI handler function.
This is a root MMI handler.
@@ -309,6 +386,13 @@ CpuHotplugMmi (
}
}
+ if (ToUnplugCount > 0) {
+Status = UnplugCpus (mToUnplugApicIds, ToUnplugCount);
+if (EFI_ERROR (Status)) {
+ goto Fatal;
+}
+ }
+
//
// We've handled this MMI.
//
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71915): https://edk2.groups.io/g/devel/message/71915
Mute This Topic: https://groups.io/mt/80819859/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v8 05/10] OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments in v6:
(1) Dropped modifications to LibraryClasses in OvmfPkg.dec
(2,3) Cleanup comments around PCD PcdCpuHotEjectDataAddress.
(4) Move PCD PcdCpuHotEjectDataAddress declaration in CpuHotplugSmm.inf
to a patch-7 where it actually gets used.
(5a,5b) Change the comment in the top block to use Laszlo's language.
Also detail when the PCD would contain a valid value.
(6) Move Library/CpuHotEjectData.h to Pcd/CpuHotEjectData.h
(7,15,16) Fixup guard macro to be C namespace compliant. Also fixup the
comment style near the endif guard.
(8-10) Rename CPU_HOT_EJECT_FN to a more EDK2 compliant style. Also add
a comment block and fix spacing.
() Rename ApicIdMap -> QemuSelectorMap while keeping the type as UINT64.
Related to a comment in patch-8 ("... add worker to do CPU ejection".)
(11a,11b) Rename CPU_EJECT_INVALID to CPU_EJECT_QEMU_SELECTOR_INVALID
and add a comment about it.
() Remove CPU_EJECT_WORKER based on review comment on a patch 8.
(12,14) Remove CPU_HOT_EJECT_DATA fields Revision and Reserved.
Reorder CPU_HOT_EJECT_DATA to minimize internal padding
and ensure elements are properly aligned.
(13a,13b) Change CpuIndex->ApicId map to ProcessorNum -> QemuSelector
() Make CPU_HOT_EJECT_HANDLER->Handler,
CPU_HOT_EJECT_HANDLER->QemuSelectorMap volatile.
OvmfPkg/OvmfPkg.dec | 4 +++
OvmfPkg/Include/Pcd/CpuHotEjectData.h | 52 +++
2 files changed, 56 insertions(+)
create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 4348bb45c64a..9629707020ba 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -352,6 +352,10 @@ [PcdsDynamic, PcdsDynamicEx]
# This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+ ## This PCD adds a communication channel between OVMF's SmmCpuFeaturesLib
+ # instance in PiSmmCpuDxeSmm, and CpuHotplugSmm.
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
+
[PcdsFeatureFlag]
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/Include/Pcd/CpuHotEjectData.h
b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
new file mode 100644
index ..024a92726869
--- /dev/null
+++ b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
@@ -0,0 +1,52 @@
+/** @file
+ Definition for the CPU_HOT_EJECT_DATA structure, which shares
+ CPU hot-eject state between OVMF's SmmCpuFeaturesLib instance in
+ PiSmmCpuDxeSmm, and CpuHotplugSmm.
+
+ CPU_HOT_EJECT_DATA is allocated in SMRAM, and pointed-to by
+ PcdCpuHotEjectDataAddress.
+
+ PcdCpuHotEjectDataAddress is valid when SMM_REQUIRE is TRUE
+ and MaxNumberOfCpus > 1.
+
+ Copyright (C) 2021, Oracle Corporation.
+
+ SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#ifndef CPU_HOT_EJECT_DATA_H_
+#define CPU_HOT_EJECT_DATA_H_
+
+/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
+ on each CPU at exit from SMM.
+
+ @param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
+ and will be used as an index into
+ CPU_HOT_EJECT_DATA->QemuSelectorMap. It is
+ identical to the processor handle in
+ EFI_SMM_CPU_SERVICE_PROTOCOL.
+**/
+typedef
+VOID
+(EFIAPI *CPU_HOT_EJECT_HANDLER) (
+ IN UINTN ProcessorNum
+ );
+
+//
+// CPU_EJECT_QEMU_SELECTOR_INVALID marks CPUs not being ejected in
+// CPU_HOT_EJECT_DATA->QemuSelectorMap.
+//
+// QEMU CPU Selector is UINT32, so we choose an invalid value larger
+// than that type.
+//
+#define CPU_EJECT_QEMU_SELECTOR_INVALID (MAX_UINT64)
+
+typedef struct {
+ volatile UINT64 *QemuSelectorMap; // Maps ProcessorNum -> QemuSelector
+ // for pending hot-ejects
+ volatile CPU_HOT_EJECT_HANDLER Handler; // Handler to do the CPU ejection
+ UINT32ArrayLength; // Entries in the QemuSelectorMap
+} CPU_HOT_EJECT_DATA;
+
+#endif // CPU_HOT_EJECT_DATA_H_
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71916): https://edk2.groups.io/g/devel/message/71916
Mute This Topic: https://groups.io/mt/80819860/21656
Group Owner: devel+ow
[edk2-devel] [PATCH v8 02/10] OvmfPkg/CpuHotplugSmm: collect hot-unplug events
Process fw_remove events in QemuCpuhpCollectApicIds() and collect
corresponding APIC IDs for CPUs that are being hot-unplugged.
In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.
This is based on the QEMU hot-unplug protocol documented here:
https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imamm...@redhat.com/
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments from v6:
(1,4) Move (and also rename) QEMU_CPUHP_STAT_EJECTED to patch 8,
where we actually use it.
(2) Downgrade debug mask from DEBUG_INFO to DEBUG_VERBOSE.
(3a,3b,3c) Keep the CurrentSelector increment operation at
the tail of the loop.
() As discussed elsewhere we also need to get the CpuSelector while
collecting ApicIds in QemuCpuhpCollectApicIds(). This patch adds a
separate parameter for the CpuSelector values, because that works
better alongside the hotplug ExtendIds logic.
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 1 +
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c| 21 +-
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 84 ---
4 files changed, 79 insertions(+), 28 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..1e23b150910e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -55,6 +55,7 @@ QemuCpuhpCollectApicIds (
OUT APIC_ID *PluggedApicIds,
OUT UINT32 *PluggedCount,
OUT APIC_ID *ToUnplugApicIds,
+ OUT UINT32 *ToUnplugSelector,
OUT UINT32 *ToUnplugCount
);
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..2ec7a107a64d 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index bf68fcd42914..3192bfea1f15 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -52,6 +52,7 @@ STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
//
STATIC APIC_ID *mPluggedApicIds;
STATIC APIC_ID *mToUnplugApicIds;
+STATIC UINT32 *mToUnplugSelector;
//
// Address of the non-SMRAM reserved memory page that contains the Post-SMM Pen
// for hot-added CPUs.
@@ -289,6 +290,7 @@ CpuHotplugMmi (
mPluggedApicIds,
,
mToUnplugApicIds,
+ mToUnplugSelector,
);
if (EFI_ERROR (Status)) {
@@ -333,7 +335,9 @@ CpuHotplugEntry (
)
{
EFI_STATUS Status;
+ UINTN Len;
UINTN Size;
+ UINTN SizeSel;
//
// This module should only be included when SMM support is required.
@@ -387,8 +391,9 @@ CpuHotplugEntry (
//
// Allocate the data structures that depend on the possible CPU count.
//
- if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
- RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Size, ))) {
+ if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
+ RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Len, ))||
+ RETURN_ERROR (SafeUintnMult (sizeof (UINT32), Len, ))) {
Status = EFI_ABORTED;
DEBUG ((DEBUG_ERROR, "%a: invalid CPU_HOT_PLUG_DATA\n", __FUNCTION__));
goto Fatal;
@@ -405,6 +410,12 @@ CpuHotplugEntry (
DEBUG ((DEBUG_ERROR, "%a: MmAllocatePool(): %r\n", __FUNCTION__, Status));
goto ReleasePluggedApicIds;
}
+ Status = gMmst->MmAllocatePool (EfiRuntimeServicesData, SizeSel,
+(VOID **));
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: MmAllocatePool(): %r\n", __FUNCTION__, Status));
+goto ReleaseToUnplugApicIds;
+ }
//
// Allocate the Post-SMM Pen for hot-added CPUs.
@@ -412,7 +423,7 @@ CpuHotplugEntry (
Status = SmbaseAllocatePostSmmPen (,
SystemTable->BootServices);
if (EFI_ERROR (Status)) {
-goto ReleaseToUnplugApicIds;
+goto ReleaseToUnplugSelector;
}
//
@@ -472,6 +483,10 @@ ReleasePostSmmPen:
SmbaseReleasePostSmmPen (mPostSmmPenAddress, SystemTable->BootServices);
mPostSmmPenAddress = 0;
+ReleaseToUnplugSelector:
+ gMmst->MmFreePool
[edk2-devel] [PATCH v8 03/10] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Address this review comment:
() Move QemuCpuhpWriteCpuStatus() (declaration and definition) between
QemuCpuhpWriteCpuSelector() and QemuCpuhpWriteCommand() to match
the order of the register descriptions in QEMU.
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 22 ++
2 files changed, 28 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 1e23b150910e..859412c1a173 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -42,6 +42,12 @@ QemuCpuhpWriteCpuSelector (
);
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ );
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 36372a5e6193..9434bb14dd4e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -114,6 +114,28 @@ QemuCpuhpWriteCpuSelector (
}
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ )
+{
+ EFI_STATUS Status;
+
+ Status = MmCpuIo->Io.Write (
+ MmCpuIo,
+ MM_IO_UINT8,
+ ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+ 1,
+
+ );
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+ASSERT (FALSE);
+CpuDeadLoop ();
+ }
+}
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71913): https://edk2.groups.io/g/devel/message/71913
Mute This Topic: https://groups.io/mt/80819857/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v8 01/10] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
ProcessHotAddedCpus(). This is in preparation for supporting CPU
hot-unplug.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Reviewed-by: Laszlo Ersek
---
Notes:
Addresses these review comments from v6:
(1) s/EFI_ERROR(/EFI_ERROR (/
(2) Remove the empty line in the comment block above
ProcessHotAddedCpus().
() Nest the EFI_ERROR handling inside the (PluggedCount > 0) clause.
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 210 ++---
1 file changed, 126 insertions(+), 84 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..bf68fcd42914 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,129 @@ STATIC UINT32 mPostSmmPenAddress;
//
STATIC EFI_HANDLE mDispatchHandle;
+/**
+ Process CPUs that have been hot-added, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, relocate the SMBASE, and report the CPU to PiSmmCpuDxeSmm
+ via EFI_SMM_CPU_SERVICE_PROTOCOL. If the supposedly hot-added CPU is already
+ known, skip it silently.
+
+ @param[in] PluggedApicIdsThe APIC IDs of the CPUs that have been
+ hot-plugged.
+
+ @param[in] PluggedCount The number of filled-in APIC IDs in
+ PluggedApicIds.
+
+ @retval EFI_SUCCESS CPUs corresponding to all the APIC IDs are
+ populated.
+
+ @retval EFI_OUT_OF_RESOURCES Out of APIC ID space in "mCpuHotPlugData".
+
+ @return Error codes propagated from SmbaseRelocate()
+ and mMmCpuService->AddProcessor().
+**/
+STATIC
+EFI_STATUS
+ProcessHotAddedCpus (
+ IN APIC_ID *PluggedApicIds,
+ IN UINT32 PluggedCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 PluggedIdx;
+ UINT32 NewSlot;
+
+ //
+ // The Post-SMM Pen need not be reinstalled multiple times within a single
+ // root MMI handling. Even reinstalling once per root MMI is only prudence;
+ // in theory installing the pen in the driver's entry point function should
+ // suffice.
+ //
+ SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+
+ PluggedIdx = 0;
+ NewSlot = 0;
+ while (PluggedIdx < PluggedCount) {
+APIC_ID NewApicId;
+UINT32 CheckSlot;
+UINTN NewProcessorNumberByProtocol;
+
+NewApicId = PluggedApicIds[PluggedIdx];
+
+//
+// Check if the supposedly hot-added CPU is already known to us.
+//
+for (CheckSlot = 0;
+ CheckSlot < mCpuHotPlugData->ArrayLength;
+ CheckSlot++) {
+ if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
+break;
+ }
+}
+if (CheckSlot < mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
+"before; ignoring it\n", __FUNCTION__, NewApicId));
+ PluggedIdx++;
+ continue;
+}
+
+//
+// Find the first empty slot in CPU_HOT_PLUG_DATA.
+//
+while (NewSlot < mCpuHotPlugData->ArrayLength &&
+ mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
+ NewSlot++;
+}
+if (NewSlot == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
+__FUNCTION__, NewApicId));
+ return EFI_OUT_OF_RESOURCES;
+}
+
+//
+// Store the APIC ID of the new processor to the slot.
+//
+mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
+
+//
+// Relocate the SMBASE of the new CPU.
+//
+Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
+ mPostSmmPenAddress);
+if (EFI_ERROR (Status)) {
+ goto RevokeNewSlot;
+}
+
+//
+// Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
+//
+Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
+ );
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, NewApicId, Status));
+ goto RevokeNewSlot;
+}
+
+DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
+ "EFI_SMM_CPU_SERVICE_PROTOCOL assigned number %Lu\n", __FUNCTION__,
+ NewApicId, (UINT64)mCpuHotPlugData->SmBase[NewSlot],
+ (UINT64)NewProcessorNumberByProtocol));
+
+NewSlot++;
+PluggedIdx++;
+ }
+
+ //
+ // We've processed this batch of hot-added CPUs.
+ //
+ return EFI_SUCCESS;
+
+RevokeNewSlot:
+ mCpuHotPlugData->ApicId[NewSlot] = MAX_UINT64;
+
+ return Status;
+}
[edk2-devel] [PATCH v8 00/10] support CPU hot-unplug
Hi, [ Note, v8 is substantially same as v7, but fixes a couple of CI errors. ] This series adds OVMF support for CPU hot-unplug. QEMU secureboot hot-unplug logic corresponding to this is in upstream. Also posted here: https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/ Testing (with QEMU 5.2.50): - Stable with randomized CPU plug/unplug (guest maxcpus=33,128) - Synthetic tests with simultaneous multi CPU hot-unplug Also at: github.com/terminus/edk2/ hot-unplug-v8 Changelog: v8: - Fixes a couple of ECC issues in the code (in patches 7, 9) v7: - Address review comments from v6. - Fix ejection bug where we were using APIC ID to do the ejection rather than the Qemu Selector. - Describes safety properties and ordering needed for concurrent accesses to CPU_HOT_EJECT_DATA->QemuSelectorMap, and CPU_HOT_EJECT_DATA->Handler. URL: https://patchew.org/EDK2/20210219090444.1332380-1-ankur.a.ar...@oracle.com/ v6: - addresses v5 review comments. URL: https://patchew.org/EDK2/20210129005950.467638-1-ankur.a.ar...@oracle.com/ v5: - fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add add Qemu Cpu Status helper"). URL: https://patchew.org/EDK2/20210126064440.299596-1-ankur.a.ar...@oracle.com/ v4: - Gets rid of unnecessary UefiCpuPkg changes URL: https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.ar...@oracle.com/ v3: - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm and OvmfPkg/CpuHotplugSmm - Cleaner split of the hot-unplug code URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/ v2: - Do the ejection via SmmCpuFeaturesRendezvousExit() URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/ RFC: URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/ Please review. Thanks Ankur Ankur Arora (10): OvmfPkg/CpuHotplugSmm: refactor hotplug logic OvmfPkg/CpuHotplugSmm: collect hot-unplug events OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper OvmfPkg/CpuHotplugSmm: introduce UnplugCpus() OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler OvmfPkg/CpuHotplugSmm: add EjectCpu() OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug OvmfPkg/OvmfPkg.dec| 4 + OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf| 2 + .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 7 + OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 + OvmfPkg/Include/Pcd/CpuHotEjectData.h | 52 ++ OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 583 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 106 +++- .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 138 + OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 18 +- 10 files changed, 796 insertions(+), 120 deletions(-) create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h -- 2.9.3 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#71911): https://edk2.groups.io/g/devel/message/71911 Mute This Topic: https://groups.io/mt/80819855/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v7 00/10] support CPU hot-unplug
On 2021-02-19 2:12 p.m., Laszlo Ersek wrote: Hi Ankur, On 02/19/21 10:04, Ankur Arora wrote: Hi, This series adds OVMF support for CPU hot-unplug. QEMU secureboot hot-unplug logic corresponding to this is in upstream. Also posted here: https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/ Testing (with QEMU 5.2.50): - Stable with randomized CPU plug/unplug (guest maxcpus=33,128) - Synthetic tests with simultaneous multi CPU hot-unplug Also at: github.com/terminus/edk2/ hot-unplug-v7 Changelog: v7: - Address review comments from v6. - Fix ejection bug where we were using APIC ID to do the ejection rather than the Qemu Selector. - Describes safety properties and ordering needed for concurrent accesses to CPU_HOT_EJECT_DATA->QemuSelectorMap, and CPU_HOT_EJECT_DATA->Handler, I plan to review this series early next week. Thanks. This work is now going to miss the upcoming stable tag (soft feature freeze is 2021-02-22 00:00:00 UTC-8), but we can hopefully merge it early in the next development cycle. Yeah, I did get a little late with v7. Meanwhile, based on my github notifications folder, I believe you may not have put this version of the series through CI -- if that's correct, then please consider doing it. I hadn't. Will do. Thanks Ankur Thanks! Laszlo v6: - addresses v5 review comments. URL: https://patchew.org/EDK2/20210129005950.467638-1-ankur.a.ar...@oracle.com/ v5: - fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add add Qemu Cpu Status helper"). URL: https://patchew.org/EDK2/20210126064440.299596-1-ankur.a.ar...@oracle.com/ v4: - Gets rid of unnecessary UefiCpuPkg changes URL: https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.ar...@oracle.com/ v3: - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm and OvmfPkg/CpuHotplugSmm - Cleaner split of the hot-unplug code URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/ v2: - Do the ejection via SmmCpuFeaturesRendezvousExit() URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/ RFC: URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/ Please review. Thanks Ankur Ankur Arora (10): OvmfPkg/CpuHotplugSmm: refactor hotplug logic OvmfPkg/CpuHotplugSmm: collect hot-unplug events OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper OvmfPkg/CpuHotplugSmm: introduce UnplugCpus() OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler OvmfPkg/CpuHotplugSmm: add EjectCpu() OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug OvmfPkg/OvmfPkg.dec| 4 + OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf| 2 + .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 7 + OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 + OvmfPkg/Include/Pcd/CpuHotEjectData.h | 52 ++ OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 578 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 106 +++- .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 136 + OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 18 +- 10 files changed, 789 insertions(+), 120 deletions(-) create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#71861): https://edk2.groups.io/g/devel/message/71861 Mute This Topic: https://groups.io/mt/80751391/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v7 10/10] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
Advertise OVMF support for CPU hot-unplug and negotiate it
if QEMU requests the feature.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments:
(1,3) s/hot unplug/hot-unplug/
(2) Get rid of the reference to the made up ICH9_APM_CNT_CPU_HOT_UNPLUG
(4,6) Remove the artificial tie in between
ICH9_LPC_SMI_F_CPU_HOTPLUG, ICH9_LPC_SMI_F_CPU_HOT_UNPLUG.
(5) Fully spell out "SMI on CPU hot-unplug".
(7) Emit separate messages on negotiation (or not) of
ICH9_LPC_SMI_F_CPU_HOT_UNPLUG.
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 18 +-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
index c9d875543205..b1d59a559dae 100644
--- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
+++ b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
@@ -29,6 +29,12 @@
//
#define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
+// The following bit value stands for "enable CPU hot-unplug, and inject an SMI
+// with control value ICH9_APM_CNT_CPU_HOTPLUG upon hot-unplug", in the
+// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
+//
+#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
+
//
// Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
// for the S3 boot script fragment to write to and read from.
@@ -112,7 +118,8 @@ NegotiateSmiFeatures (
QemuFwCfgReadBytes (sizeof mSmiFeatures, );
//
- // We want broadcast SMI, SMI on CPU hotplug, and nothing else.
+ // We want broadcast SMI, SMI on CPU hotplug, SMI on CPU hot-unplug
+ // and nothing else.
//
RequestedFeaturesMask = ICH9_LPC_SMI_F_BROADCAST;
if (!MemEncryptSevIsEnabled ()) {
@@ -120,8 +127,10 @@ NegotiateSmiFeatures (
// For now, we only support hotplug with SEV disabled.
//
RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOTPLUG;
+RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
}
mSmiFeatures &= RequestedFeaturesMask;
+
QemuFwCfgSelectItem (mRequestedFeaturesItem);
QemuFwCfgWriteBytes (sizeof mSmiFeatures, );
@@ -166,6 +175,13 @@ NegotiateSmiFeatures (
__FUNCTION__));
}
+ if ((mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG) == 0) {
+DEBUG ((DEBUG_INFO, "%a: CPU hot-unplug not negotiated\n", __FUNCTION__));
+ } else {
+DEBUG ((DEBUG_INFO, "%a: CPU hot-unplug with SMI negotiated\n",
+ __FUNCTION__));
+ }
+
//
// Negotiation successful (although we may not have gotten the optimal
// feature set).
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71838): https://edk2.groups.io/g/devel/message/71838
Mute This Topic: https://groups.io/mt/80751403/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v7 00/10] support CPU hot-unplug
Hi, This series adds OVMF support for CPU hot-unplug. QEMU secureboot hot-unplug logic corresponding to this is in upstream. Also posted here: https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/ Testing (with QEMU 5.2.50): - Stable with randomized CPU plug/unplug (guest maxcpus=33,128) - Synthetic tests with simultaneous multi CPU hot-unplug Also at: github.com/terminus/edk2/ hot-unplug-v7 Changelog: v7: - Address review comments from v6. - Fix ejection bug where we were using APIC ID to do the ejection rather than the Qemu Selector. - Describes safety properties and ordering needed for concurrent accesses to CPU_HOT_EJECT_DATA->QemuSelectorMap, and CPU_HOT_EJECT_DATA->Handler, v6: - addresses v5 review comments. URL: https://patchew.org/EDK2/20210129005950.467638-1-ankur.a.ar...@oracle.com/ v5: - fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add add Qemu Cpu Status helper"). URL: https://patchew.org/EDK2/20210126064440.299596-1-ankur.a.ar...@oracle.com/ v4: - Gets rid of unnecessary UefiCpuPkg changes URL: https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.ar...@oracle.com/ v3: - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm and OvmfPkg/CpuHotplugSmm - Cleaner split of the hot-unplug code URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/ v2: - Do the ejection via SmmCpuFeaturesRendezvousExit() URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/ RFC: URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/ Please review. Thanks Ankur Ankur Arora (10): OvmfPkg/CpuHotplugSmm: refactor hotplug logic OvmfPkg/CpuHotplugSmm: collect hot-unplug events OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper OvmfPkg/CpuHotplugSmm: introduce UnplugCpus() OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler OvmfPkg/CpuHotplugSmm: add EjectCpu() OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug OvmfPkg/OvmfPkg.dec| 4 + OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf| 2 + .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 7 + OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 + OvmfPkg/Include/Pcd/CpuHotEjectData.h | 52 ++ OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 578 + OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 106 +++- .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 136 + OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 18 +- 10 files changed, 789 insertions(+), 120 deletions(-) create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h -- 2.9.3 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#71828): https://edk2.groups.io/g/devel/message/71828 Mute This Topic: https://groups.io/mt/80751391/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v7 02/10] OvmfPkg/CpuHotplugSmm: collect hot-unplug events
Process fw_remove events in QemuCpuhpCollectApicIds() and collect
corresponding APIC IDs for CPUs that are being hot-unplugged.
In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.
This is based on the QEMU hot-unplug protocol documented here:
https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imamm...@redhat.com/
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments from v6:
(1,4) Move (and also rename) QEMU_CPUHP_STAT_EJECTED to patch 8,
where we actually use it.
(2) Downgrade debug mask from DEBUG_INFO to DEBUG_VERBOSE.
(3a,3b,3c) Keep the CurrentSelector increment operation at
the tail of the loop.
() As discussed elsewhere we also need to get the CpuSelector while
collecting ApicIds in QemuCpuhpCollectApicIds(). This patch adds a
separate parameter for the CpuSelector values, because that works
better alongside the hotplug ExtendIds logic.
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 1 +
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c| 21 +-
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 84 ---
4 files changed, 79 insertions(+), 28 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..1e23b150910e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -55,6 +55,7 @@ QemuCpuhpCollectApicIds (
OUT APIC_ID *PluggedApicIds,
OUT UINT32 *PluggedCount,
OUT APIC_ID *ToUnplugApicIds,
+ OUT UINT32 *ToUnplugSelector,
OUT UINT32 *ToUnplugCount
);
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..2ec7a107a64d 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index bf68fcd42914..3192bfea1f15 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -52,6 +52,7 @@ STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
//
STATIC APIC_ID *mPluggedApicIds;
STATIC APIC_ID *mToUnplugApicIds;
+STATIC UINT32 *mToUnplugSelector;
//
// Address of the non-SMRAM reserved memory page that contains the Post-SMM Pen
// for hot-added CPUs.
@@ -289,6 +290,7 @@ CpuHotplugMmi (
mPluggedApicIds,
,
mToUnplugApicIds,
+ mToUnplugSelector,
);
if (EFI_ERROR (Status)) {
@@ -333,7 +335,9 @@ CpuHotplugEntry (
)
{
EFI_STATUS Status;
+ UINTN Len;
UINTN Size;
+ UINTN SizeSel;
//
// This module should only be included when SMM support is required.
@@ -387,8 +391,9 @@ CpuHotplugEntry (
//
// Allocate the data structures that depend on the possible CPU count.
//
- if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
- RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Size, ))) {
+ if (RETURN_ERROR (SafeUintnSub (mCpuHotPlugData->ArrayLength, 1, )) ||
+ RETURN_ERROR (SafeUintnMult (sizeof (APIC_ID), Len, ))||
+ RETURN_ERROR (SafeUintnMult (sizeof (UINT32), Len, ))) {
Status = EFI_ABORTED;
DEBUG ((DEBUG_ERROR, "%a: invalid CPU_HOT_PLUG_DATA\n", __FUNCTION__));
goto Fatal;
@@ -405,6 +410,12 @@ CpuHotplugEntry (
DEBUG ((DEBUG_ERROR, "%a: MmAllocatePool(): %r\n", __FUNCTION__, Status));
goto ReleasePluggedApicIds;
}
+ Status = gMmst->MmAllocatePool (EfiRuntimeServicesData, SizeSel,
+(VOID **));
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: MmAllocatePool(): %r\n", __FUNCTION__, Status));
+goto ReleaseToUnplugApicIds;
+ }
//
// Allocate the Post-SMM Pen for hot-added CPUs.
@@ -412,7 +423,7 @@ CpuHotplugEntry (
Status = SmbaseAllocatePostSmmPen (,
SystemTable->BootServices);
if (EFI_ERROR (Status)) {
-goto ReleaseToUnplugApicIds;
+goto ReleaseToUnplugSelector;
}
//
@@ -472,6 +483,10 @@ ReleasePostSmmPen:
SmbaseReleasePostSmmPen (mPostSmmPenAddress, SystemTable->BootServices);
mPostSmmPenAddress = 0;
+ReleaseToUnplugSelector:
+ gMmst->MmFreePool
[edk2-devel] [PATCH v7 09/10] OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject
Add logic in EjectCpu() to do the actual the CPU ejection.
On the BSP, ejection happens by first selecting the CPU via
its QemuSelector and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
the CPU out of the SMI handler.
Meanwhile the CPU being ejected, waits around in its holding
area until it is context-switched out. Note that it is possible
that a slow CPU gets ejected before it reaches the wait loop.
However, this would never happen before it has executed the
"AllCpusInSync" loop in SmiRendezvous().
It can mean that an ejected CPU does not execute code after
that point but given that the CPU state will be destroyed by
QEMU, the missed cleanup is no great loss.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following reviewing comments from v6:
(1) s/CpuEject/EjectCpu/g
(2,2a,2c) Get rid of eject-worker and related.
(2b,2d) Use the PlatformSmmBspElection() logic to find out IsBSP.
(3,3b) Use CPU_HOT_EJECT_DATA->QemuSelector instead of ApicIdMap to
do the actual ejection.
(4,5a,5b) Fix the format etc in the final unplugged log message
() Also as discussed elsewhere document the ordering requirements for
mCpuHotEjectData->QemuSelector[] and mCpuHotEjectData->Handler.
() [from patch 2] Move definition of QEMU_CPUHP_STAT_EJECTED to this
patch.
() s/QEMU_CPUHP_STAT_EJECTED/QEMU_CPUHP_STAT_EJECT/
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 +
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 122 +++--
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 31 ++
3 files changed, 147 insertions(+), 7 deletions(-)
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index 2ec7a107a64d..d0e83102c13f 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,7 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_EJECT BIT3
#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 5dcaca53d42f..b80d4a2283a4 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -18,6 +18,7 @@
#include // CPU_HOT_EJECT_DATA
#include // EFI_MM_CPU_IO_PROTOCOL
#include // EFI_SMM_CPU_SERVICE_PROTOCOL
+#include // MSR_IA32_APIC_BASE_REGISTER
#include// EFI_STATUS
#include "ApicId.h" // APIC_ID
@@ -191,12 +192,36 @@ RevokeNewSlot:
}
/**
+ EjectCpu needs to know the BSP at SMI exit at a point when
+ some of the EFI_SMM_CPU_SERVICE_PROTOCOL state has been torn
+ down.
+ Reuse the logic from OvmfPkg::PlatformSmmBspElection() to
+ do that.
+**/
+STATIC
+BOOLEAN
+CheckIfBsp (
+ IN UINTN ProcessorNum
+ )
+{
+ MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr;
+ BOOLEAN IsBsp;
+
+ ApicBaseMsr.Uint64 = AsmReadMsr64 (MSR_IA32_APIC_BASE);
+ IsBsp = (BOOLEAN)(ApicBaseMsr.Bits.BSP == 1);
+ return IsBsp;
+}
+
+/**
CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
on each CPU at exit from SMM.
- If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is neither the BSP, nor being ejected, nothing
+ to be done.
If, the executing CPU is being ejected, wait in a halted loop
until ejected.
+ If, the executing CPU is the BSP, set QEMU CPU status to eject
+ for CPUs being ejected.
@param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
and will be used as an index into
@@ -211,9 +236,97 @@ EjectCpu (
)
{
UINT64 QemuSelector;
+ BOOLEAN IsBsp = CheckIfBsp (ProcessorNum);
+ //
+ // mCpuHotEjectData->QemuSelectorMap[ProcessorNum] is updated
+ // on the BSP in the ongoing SMI iteration at two places:
+ //
+ // - UnplugCpus() where the BSP determines if a CPU is under ejection
+ // or not. As the comment where mCpuHotEjectData->Handler is set-up
+ // describes any such updates are guaranteed to be ordered-before the
+ // dereference below.
+ //
+ // - EjectCpu() on the BSP updates QemuSelectorMap[ProcessorNum] for
+ // CPUs after they have been hot-ejected.
+ //
+ // The CPU under ejection: might be executing anywhere between the
+ // "AllCpusInSync" exit loop in SmiRendezvous() to about to
+ // dereference QemuSelectorMap[ProcessorNum].
+ // Given that the BSP ensures that
[edk2-devel] [PATCH v7 08/10] OvmfPkg/CpuHotplugSmm: add EjectCpu()
Add EjectCpu(), which handles the CPU ejection, and provides a holding area for said CPUs. It is called via SmmCpuFeaturesRendezvousExit(), at the tail end of the SMI handling. Also UnplugCpus() now stashes QEMU Selectors of CPUs which need to be ejected in CPU_HOT_EJECT_DATA.QemuSelectorMap. This is used by EjectCpu() to identify CPUs marked for ejection. Cc: Laszlo Ersek Cc: Jordan Justen Cc: Ard Biesheuvel Cc: Igor Mammedov Cc: Boris Ostrovsky Cc: Aaron Young Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132 Signed-off-by: Ankur Arora --- Notes: Address these review comments from v6: (1) s/CpuEject/EjectCpu/g (2) Ensure that the added include is in sorted order. (3) Switch to a cheaper CpuSleep() based loop instead of CpuDeadLoop(). Also add the CpuLib LibraryClass. (4) Remove the nested else clause (5) Use Laszlo's much clearer comment when we try to map multiple QemuSelector to the same ProcessorNum. (6a) Fix indentation of the debug print in the block in (5). (6b,6c,6d) Fix printf types for ProcessorNum, use FMT_APIC_ID for APIC_ID and 0x%Lx for QemuSelector[]. () As discussed elsewhere add an DEBUG_INFO print logging the correspondence between ProcessorNum, APIC_ID, QemuSelector. (7a,7b) Use EFI_ALREADY_STARTED instead of EFI_INVALID_PARAMETER and document it in the UnplugCpus() comment block. () As discussed elsewhere, add the import statement for PcdCpuHotEjectDataAddress. (9) Use Laszlo's comment in the PcdGet64(PcdCpuHotEjectDataAddress) description block. (10) Change mCpuHotEjectData init state checks from ASSERT to ones consistent with similar checks for mCpuHotPlugData. (11-14) Get rid of mCpuHotEjectData init loop: moved to a prior patch so it can be done at allocation time. (15) s/SmmCpuFeaturesSmiRendezvousExit/SmmCpuFeaturesRendezvousExit/ (16,17) Document the ordering requirements of mCpuHotEjectData->Handler, and mCpuHotEjectData->QemuSelectorMap. Not addressed: (8) Not removing the EjectCount variable as I'd like to minimize stores/loads to CPU_HOT_EJECT_DATA->Handler and so would like to do this a single time at the end of the iteration. (It is safe to write multiple times to the handler in UnplugCpus() but given the ordering concerns around it, it seems cleaner to not access it unnecessarily.) OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf | 2 + OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 157 ++-- 2 files changed, 151 insertions(+), 8 deletions(-) diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf index 04322b0d7855..ebcc7e2ac63a 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf @@ -40,6 +40,7 @@ [Packages] [LibraryClasses] BaseLib BaseMemoryLib + CpuLib DebugLib LocalApicLib MmServicesTableLib @@ -54,6 +55,7 @@ [Protocols] [Pcd] gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress## CONSUMES + gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress ## CONSUMES gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase ## CONSUMES [FeaturePcd] diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c index f07b5072749a..5dcaca53d42f 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c @@ -10,10 +10,12 @@ #include // ICH9_APM_CNT #include // QEMU_CPUHP_CMD_GET_PENDING #include // CpuDeadLoop() +#include // CpuSleep() #include // ASSERT() #include // gMmst #include // PcdGetBool() #include // SafeUintnSub() +#include // CPU_HOT_EJECT_DATA #include // EFI_MM_CPU_IO_PROTOCOL #include // EFI_SMM_CPU_SERVICE_PROTOCOL #include// EFI_STATUS @@ -32,11 +34,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo; // STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService; // -// This structure is a communication side-channel between the +// These structures serve as communication side-channels between the // EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider // (i.e., PiSmmCpuDxeSmm). // STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData; +STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData; // // SMRAM arrays for fetching the APIC IDs of processors with pending events (of // known event types), for the time of just one MMI. @@ -188,18 +191,72 @@ RevokeNewSlot: } /** + CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit() + on each CPU at exit from SMM. + + If, the executing CPU is not being ejected, nothing to be done. + If, the executing CPU is being ejected, wait in a halted loop + until ejected. + + @param[in] ProcessorNum ProcessorNum denotes the CPU ex
[edk2-devel] [PATCH v7 07/10] OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
Call the CPU hot-eject handler if one is installed. The condition for
installation is (PcdCpuMaxLogicalProcessorNumber > 1), and there's
a hot-unplug request.
The handler executes in context of SmmCpuFeaturesRendezvousExit(),
which is called at the tail end of SmiRendezvous() after the BSP has
given the signal to exit via the "AllCpusInSync" loop.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Address the following review comments from v6, patch-6:
(19a) Move the call to the ejection handler to a separate patch.
(19b) Describe the calling context of SmmCpuFeaturesRendezvousExit().
(20) Add comment describing the state when the Handler is not armed.
OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 13 +
1 file changed, 13 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index adbfc90ad46e..c5cfe2610c23 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -467,6 +467,19 @@ SmmCpuFeaturesRendezvousExit (
IN UINTN CpuIndex
)
{
+ //
+ // We only call the Handler if CPU hot-eject is enabled
+ // (PcdCpuMaxLogicalProcessorNumber > 1), and hot-eject is needed
+ // in this SMI exit (otherwise mCpuHotEjectData->Handler is not armed.)
+ //
+
+ if (mCpuHotEjectData != NULL) {
+CPU_HOT_EJECT_HANDLER Handler = mCpuHotEjectData->Handler;
+
+if (Handler != NULL) {
+ Handler (CpuIndex);
+}
+ }
}
/**
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71835): https://edk2.groups.io/g/devel/message/71835
Mute This Topic: https://groups.io/mt/80751399/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v7 06/10] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection
state between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
The init happens via SmmCpuFeaturesSmmRelocationComplete(), and so it
will run as part of the PiSmmCpuDxeSmm entry point function,
PiCpuSmmEntry(). Once inited, CPU_HOT_EJECT_DATA is exposed via
PcdCpuHotEjectDataAddress.
The CPU hot-eject handler (CPU_HOT_EJECT_DATA->Handler) is setup when
there is an ejection request via CpuHotplugSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments:
(1) Detail in commit message about context in which CPU_HOT_EJECT_DATA
is inited.
(2) Add in sorted order MemoryAllocationLib in LibraryClasses
(3) Sort added includes in SmmCpuFeaturesLib.c
(4a-4b) Fixup linkage directives for mCpuHotEjectData.
(5) s/CpuHotEjectData/mCpuHotEjectData/
(6,10a,10b) Remove dependence on PcdCpuHotPlugSupport
(7) Make the tense structure consistent in block comment for
InitCpuHotEject().
(8) s/SmmCpuFeaturesSmmInitHotEject/InitCpuHotEject/
(9) s/mMaxNumberOfCpus/MaxNumberOfCpus/
(11) Remove a bunch of obvious comments.
(14a,14b,14c) Use SafeUint functions and rework the allocation logic
so we can just use a single allocation.
(12) Remove the AllocatePool() cast.
(13) Use a CpuDeadLoop() in case of failure; albeit via a goto, not
inline.
(15) Initialize the mCpuHotEjectData->QemuSelectorMap locally.
(16) Fix indentation in PcdSet64S.
(17) Change the cast in PcdSet64S() to UINTN.
(18) Use RETURN_STATUS instead of EFI_STATUS.
(19,20) Move the Handler logic in SmmCpuFeaturesRendezvousExit() into
into a separate patch.
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 4 +
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 92 ++
2 files changed, 96 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..8a426a4c10fb 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -30,9 +30,13 @@ [LibraryClasses]
BaseMemoryLib
DebugLib
MemEncryptSevLib
+ MemoryAllocationLib
PcdLib
+ SafeIntLib
SmmServicesTableLib
UefiBootServicesTableLib
[Pcd]
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..adbfc90ad46e 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -11,10 +11,13 @@
#include
#include
#include
+#include
#include
+#include
#include
#include
#include
+#include
#include
#include
#include
@@ -171,6 +174,92 @@ SmmCpuFeaturesHookReturnFromSmm (
return OriginalInstructionPointer;
}
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+/**
+ Initialize mCpuHotEjectData if PcdCpuMaxLogicalProcessorNumber > 1.
+
+ Also setup the corresponding PcdCpuHotEjectDataAddress.
+**/
+STATIC
+VOID
+InitCpuHotEjectData (
+ VOID
+ )
+{
+ UINTN ArrayLen;
+ UINTN BaseLen;
+ UINTN TotalLen;
+ UINT32 Idx;
+ UINT32 MaxNumberOfCpus;
+ RETURN_STATUS PcdStatus;
+
+ MaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+
+ if (MaxNumberOfCpus == 1) {
+return;
+ }
+
+ //
+ // We want the following lay out for CPU_HOT_EJECT_DATA:
+ // UINTN alignment: CPU_HOT_EJECT_DATA
+ // --- padding if needed ---
+ // UINT64 alignment: CPU_HOT_EJECT_DATA->QemuSelectorMap[]
+ //
+ // Accordingly, we allocate:
+ // sizeof(*mCpuHotEjectData) + (MaxNumberOfCpus *
+ // sizeof(mCpuHotEjectData->QemuSelectorMap[0])).
+ // Add sizeof(UINT64) to use as padding if needed.
+ //
+
+ if (RETURN_ERROR (SafeUintnMult (sizeof (*mCpuHotEjectData), 1, )) ||
+ RETURN_ERROR (SafeUintnMult (
+ sizeof (mCpuHotEjectData->QemuSelectorMap[0]),
+ MaxNumberOfCpus, )) ||
+ RETURN_ERROR (SafeUintnAdd (BaseLen, ArrayLen, ))||
+ RETURN_ERROR (SafeUintnAdd (TotalLen, sizeof (UINT64), ))) {
+DEBUG ((DEBUG_ERROR, "%a: invalid CPU_HOT_EJECT_DATA\n", __FUNCTION__));
+goto Fatal;
+ }
+
+ mCpuHotEjectData = AllocatePool (TotalLen);
+ if (mCpuHotEjectData == NULL) {
+ASSERT (mCpuHotEjectData != NULL);
+goto Fatal;
+ }
+
+ mCpuHotEjectData->Handler = NULL;
+ mCpuHotEjectData->ArrayLength = MaxNumberOfCpus;
+
+
[edk2-devel] [PATCH v7 04/10] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().
With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses these review comments from v6:
(1) Drop the empty line in the comment block around UnplugCpus().
(2) Make the "did not find APIC ID" DEBUG_VERBOSE instead of DEBUG_INFO.
(3) Un-Indented ("Outdented") the line following the comment "Ignore the
unplug if APIC ID.
(4) Remove the empty line between Status assignment and check.
(5) Drop the "goto Fatal" logic and just return Status directly.
(6) Handle both Plugging and Unplugging of CPUs in one go.
(7) Also nest the EFI_STATUS check.
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 84 ++
1 file changed, 84 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 3192bfea1f15..f07b5072749a 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -188,6 +188,83 @@ RevokeNewSlot:
}
/**
+ Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, report the CPU to PiSmmCpuDxeSmm via
+ EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is
+ unknown, skip it silently.
+
+ @param[in] ToUnplugApicIdsThe APIC IDs of the CPUs that are about to be
+hot-unplugged.
+
+ @param[in] ToUnplugCount The number of filled-in APIC IDs in
+ToUnplugApicIds.
+
+ @retval EFI_SUCCESS Known APIC IDs have been removed from SMM data
+structures.
+
+ @return Error codes propagated from
+mMmCpuService->RemoveProcessor().
+**/
+STATIC
+EFI_STATUS
+UnplugCpus (
+ IN APIC_ID *ToUnplugApicIds,
+ IN UINT32 ToUnplugCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 ToUnplugIdx;
+ UINTN ProcessorNum;
+
+ ToUnplugIdx = 0;
+ while (ToUnplugIdx < ToUnplugCount) {
+APIC_IDRemoveApicId;
+
+RemoveApicId = ToUnplugApicIds[ToUnplugIdx];
+
+//
+// mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+// the ProcessorNum for the APIC ID to be removed.
+//
+for (ProcessorNum = 0;
+ ProcessorNum < mCpuHotPlugData->ArrayLength;
+ ProcessorNum++) {
+ if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+break;
+ }
+}
+
+//
+// Ignore the unplug if APIC ID not found
+//
+if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: did not find APIC ID " FMT_APIC_ID
+" to unplug\n", __FUNCTION__, RemoveApicId));
+ ToUnplugIdx++;
+ continue;
+}
+
+//
+// Mark ProcessorNum for removal from SMM data structures
+//
+Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, RemoveApicId, Status));
+ return Status;
+}
+
+ToUnplugIdx++;
+ }
+
+ //
+ // We've removed this set of APIC IDs from SMM data structures.
+ //
+ return EFI_SUCCESS;
+}
+
+/**
CPU Hotplug MMI handler function.
This is a root MMI handler.
@@ -309,6 +386,13 @@ CpuHotplugMmi (
}
}
+ if (ToUnplugCount > 0) {
+Status = UnplugCpus (mToUnplugApicIds, ToUnplugCount);
+if (EFI_ERROR (Status)) {
+ goto Fatal;
+}
+ }
+
//
// We've handled this MMI.
//
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71833): https://edk2.groups.io/g/devel/message/71833
Mute This Topic: https://groups.io/mt/80751397/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v7 05/10] OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Addresses the following review comments in v6:
(1) Dropped modifications to LibraryClasses in OvmfPkg.dec
(2,3) Cleanup comments around PCD PcdCpuHotEjectDataAddress.
(4) Move PCD PcdCpuHotEjectDataAddress declaration in CpuHotplugSmm.inf
to a patch-7 where it actually gets used.
(5a,5b) Change the comment in the top block to use Laszlo's language.
Also detail when the PCD would contain a valid value.
(6) Move Library/CpuHotEjectData.h to Pcd/CpuHotEjectData.h
(7,15,16) Fixup guard macro to be C namespace compliant. Also fixup the
comment style near the endif guard.
(8-10) Rename CPU_HOT_EJECT_FN to a more EDK2 compliant style. Also add
a comment block and fix spacing.
() Rename ApicIdMap -> QemuSelectorMap while keeping the type as UINT64.
Related to a comment in patch-8 ("... add worker to do CPU ejection".)
(11a,11b) Rename CPU_EJECT_INVALID to CPU_EJECT_QEMU_SELECTOR_INVALID
and add a comment about it.
() Remove CPU_EJECT_WORKER based on review comment on a patch 8.
(12,14) Remove CPU_HOT_EJECT_DATA fields Revision and Reserved.
Reorder CPU_HOT_EJECT_DATA to minimize internal padding
and ensure elements are properly aligned.
(13a,13b) Change CpuIndex->ApicId map to ProcessorNum -> QemuSelector
() Make CPU_HOT_EJECT_HANDLER->Handler,
CPU_HOT_EJECT_HANDLER->QemuSelectorMap volatile.
OvmfPkg/OvmfPkg.dec | 4 +++
OvmfPkg/Include/Pcd/CpuHotEjectData.h | 52 +++
2 files changed, 56 insertions(+)
create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 4348bb45c64a..9629707020ba 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -352,6 +352,10 @@ [PcdsDynamic, PcdsDynamicEx]
# This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+ ## This PCD adds a communication channel between OVMF's SmmCpuFeaturesLib
+ # instance in PiSmmCpuDxeSmm, and CpuHotplugSmm.
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
+
[PcdsFeatureFlag]
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/Include/Pcd/CpuHotEjectData.h
b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
new file mode 100644
index ..024a92726869
--- /dev/null
+++ b/OvmfPkg/Include/Pcd/CpuHotEjectData.h
@@ -0,0 +1,52 @@
+/** @file
+ Definition for the CPU_HOT_EJECT_DATA structure, which shares
+ CPU hot-eject state between OVMF's SmmCpuFeaturesLib instance in
+ PiSmmCpuDxeSmm, and CpuHotplugSmm.
+
+ CPU_HOT_EJECT_DATA is allocated in SMRAM, and pointed-to by
+ PcdCpuHotEjectDataAddress.
+
+ PcdCpuHotEjectDataAddress is valid when SMM_REQUIRE is TRUE
+ and MaxNumberOfCpus > 1.
+
+ Copyright (C) 2021, Oracle Corporation.
+
+ SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#ifndef CPU_HOT_EJECT_DATA_H_
+#define CPU_HOT_EJECT_DATA_H_
+
+/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit()
+ on each CPU at exit from SMM.
+
+ @param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM,
+ and will be used as an index into
+ CPU_HOT_EJECT_DATA->QemuSelectorMap. It is
+ identical to the processor handle in
+ EFI_SMM_CPU_SERVICE_PROTOCOL.
+**/
+typedef
+VOID
+(EFIAPI *CPU_HOT_EJECT_HANDLER) (
+ IN UINTN ProcessorNum
+ );
+
+//
+// CPU_EJECT_QEMU_SELECTOR_INVALID marks CPUs not being ejected in
+// CPU_HOT_EJECT_DATA->QemuSelectorMap.
+//
+// QEMU CPU Selector is UINT32, so we choose an invalid value larger
+// than that type.
+//
+#define CPU_EJECT_QEMU_SELECTOR_INVALID (MAX_UINT64)
+
+typedef struct {
+ volatile UINT64 *QemuSelectorMap; // Maps ProcessorNum -> QemuSelector
+ // for pending hot-ejects
+ volatile CPU_HOT_EJECT_HANDLER Handler; // Handler to do the CPU ejection
+ UINT32ArrayLength; // Entries in the QemuSelectorMap
+} CPU_HOT_EJECT_DATA;
+
+#endif // CPU_HOT_EJECT_DATA_H_
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71832): https://edk2.groups.io/g/devel/message/71832
Mute This Topic: https://groups.io/mt/80751396/21656
Group Owner: devel+ow
[edk2-devel] [PATCH v7 03/10] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
Address this review comment:
() Move QemuCpuhpWriteCpuStatus() (declaration and definition) between
QemuCpuhpWriteCpuSelector() and QemuCpuhpWriteCommand() to match
the order of the register descriptions in QEMU.
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 22 ++
2 files changed, 28 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 1e23b150910e..859412c1a173 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -42,6 +42,12 @@ QemuCpuhpWriteCpuSelector (
);
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ );
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 36372a5e6193..9434bb14dd4e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -114,6 +114,28 @@ QemuCpuhpWriteCpuSelector (
}
VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ )
+{
+ EFI_STATUS Status;
+
+ Status = MmCpuIo->Io.Write (
+ MmCpuIo,
+ MM_IO_UINT8,
+ ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+ 1,
+
+ );
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+ASSERT (FALSE);
+CpuDeadLoop ();
+ }
+}
+
+VOID
QemuCpuhpWriteCommand (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
IN UINT8Command
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71831): https://edk2.groups.io/g/devel/message/71831
Mute This Topic: https://groups.io/mt/80751394/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v7 01/10] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
ProcessHotAddedCpus(). This is in preparation for supporting CPU
hot-unplug.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
Reviewed-by: Laszlo Ersek
---
Notes:
Addresses these review comments from v6:
(1) s/EFI_ERROR(/EFI_ERROR (/
(2) Remove the empty line in the comment block above
ProcessHotAddedCpus().
() Nest the EFI_ERROR handling inside the (PluggedCount > 0) clause.
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 210 ++---
1 file changed, 126 insertions(+), 84 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..bf68fcd42914 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,129 @@ STATIC UINT32 mPostSmmPenAddress;
//
STATIC EFI_HANDLE mDispatchHandle;
+/**
+ Process CPUs that have been hot-added, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, relocate the SMBASE, and report the CPU to PiSmmCpuDxeSmm
+ via EFI_SMM_CPU_SERVICE_PROTOCOL. If the supposedly hot-added CPU is already
+ known, skip it silently.
+
+ @param[in] PluggedApicIdsThe APIC IDs of the CPUs that have been
+ hot-plugged.
+
+ @param[in] PluggedCount The number of filled-in APIC IDs in
+ PluggedApicIds.
+
+ @retval EFI_SUCCESS CPUs corresponding to all the APIC IDs are
+ populated.
+
+ @retval EFI_OUT_OF_RESOURCES Out of APIC ID space in "mCpuHotPlugData".
+
+ @return Error codes propagated from SmbaseRelocate()
+ and mMmCpuService->AddProcessor().
+**/
+STATIC
+EFI_STATUS
+ProcessHotAddedCpus (
+ IN APIC_ID *PluggedApicIds,
+ IN UINT32 PluggedCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 PluggedIdx;
+ UINT32 NewSlot;
+
+ //
+ // The Post-SMM Pen need not be reinstalled multiple times within a single
+ // root MMI handling. Even reinstalling once per root MMI is only prudence;
+ // in theory installing the pen in the driver's entry point function should
+ // suffice.
+ //
+ SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+
+ PluggedIdx = 0;
+ NewSlot = 0;
+ while (PluggedIdx < PluggedCount) {
+APIC_ID NewApicId;
+UINT32 CheckSlot;
+UINTN NewProcessorNumberByProtocol;
+
+NewApicId = PluggedApicIds[PluggedIdx];
+
+//
+// Check if the supposedly hot-added CPU is already known to us.
+//
+for (CheckSlot = 0;
+ CheckSlot < mCpuHotPlugData->ArrayLength;
+ CheckSlot++) {
+ if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
+break;
+ }
+}
+if (CheckSlot < mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
+"before; ignoring it\n", __FUNCTION__, NewApicId));
+ PluggedIdx++;
+ continue;
+}
+
+//
+// Find the first empty slot in CPU_HOT_PLUG_DATA.
+//
+while (NewSlot < mCpuHotPlugData->ArrayLength &&
+ mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
+ NewSlot++;
+}
+if (NewSlot == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
+__FUNCTION__, NewApicId));
+ return EFI_OUT_OF_RESOURCES;
+}
+
+//
+// Store the APIC ID of the new processor to the slot.
+//
+mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
+
+//
+// Relocate the SMBASE of the new CPU.
+//
+Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
+ mPostSmmPenAddress);
+if (EFI_ERROR (Status)) {
+ goto RevokeNewSlot;
+}
+
+//
+// Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
+//
+Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
+ );
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, NewApicId, Status));
+ goto RevokeNewSlot;
+}
+
+DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
+ "EFI_SMM_CPU_SERVICE_PROTOCOL assigned number %Lu\n", __FUNCTION__,
+ NewApicId, (UINT64)mCpuHotPlugData->SmBase[NewSlot],
+ (UINT64)NewProcessorNumberByProtocol));
+
+NewSlot++;
+PluggedIdx++;
+ }
+
+ //
+ // We've processed this batch of hot-added CPUs.
+ //
+ return EFI_SUCCESS;
+
+RevokeNewSlot:
+ mCpuHotPlugData->ApicId[NewSlot] = MAX_UINT64;
+
+ return Status;
+}
Re: [edk2-devel] [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
On 2021-02-05 8:06 a.m., Laszlo Ersek wrote: Hi Ankur, I figure it's prudent for me to follow up here too: On 02/04/21 03:49, Ankur Arora wrote: On 2021-02-03 12:58 p.m., Laszlo Ersek wrote: On 02/03/21 07:45, Ankur Arora wrote: On 2021-02-02 6:15 a.m., Laszlo Ersek wrote: On 02/02/21 15:00, Laszlo Ersek wrote: ... I guess that volatile-qualifying both CPU_HOT_EJECT_DATA, and the array pointed-to by CPU_HOT_EJECT_DATA.ApicIdMap, should suffice. In combination with the sync-up point that you quoted. This seems to match existing practice in PiSmmCpuDxeSmm -- there are no concurrent accesses, so atomicity is not a concern, and serializing the instruction streams coarsely, with the sync-up, in combination with volatile accesses, should presumably guarantee visibility (on x86 anyway). To summarize, this is what I would ask for: - make CPU_HOT_EJECT_DATA volatile - make (*CPU_HOT_EJECT_DATA.ApicIdMap) volatile - after storing something to CPU_HOT_EJECT_DATA or CPU_HOT_EJECT_DATA.ApicIdMap on the BSP, execute a MemoryFence() - before fetching something from CPU_HOT_EJECT_DATA or CPU_HOT_EJECT_DATA.ApicIdMap on an AP, execute a MemoryFence() Except: MemoryFence() isn't a *memory fence* in fact. See "MdePkg/Library/BaseLib/X64/GccInline.c". It's just a compiler barrier, which may not add anything beyond what we'd already have from "volatile". Case in point: PiSmmCpuDxeSmm performs heavy multi-processing, but does not contain a single invocation of MemoryFence(). It uses volatile objects, and a handful of InterlockedCompareExchangeXx() calls, for implementing semaphores. (NB: there is no 8-bit variant of InterlockedCompareExchange(), as "volatile UINT8" is considered atomic in itself, and a suitable basis for a sempahore too.) And given the synchronization from those semaphores, PiSmmCpuDpxeSmm trusts that updates to the *other* volatile objects are both atomic and visible. I'm pretty sure this only works because x86 is in-order. There are instruction stream barriers in place, and compiler barriers too, but no actual memory barriers. Right and just to separate them explicitly, there are two problems: - compiler: where the compiler caches stuff in or looks at stale memory locations. Now, AFAICS, this should not happen because the ApicIdMap would never change once set so the compiler should reasonably be able to cache the address of ApicIdMap and dereference it (thus obviating the need for volatile.) (CPU_HOT_EJECT_DATA.Handler does change though.) Yeah, I did kinda elide over that. Let me think this through in v7 and add more explicit comments and then we can see if it still looks fishy? Thanks Ankur The compiler could, however, cache any assignments to ApicIdMap[Idx] (especially given LTO) and we could use a MemoryFence() (as the compiler barrier that it is) to force the store. - CPU pipeline: as you said, right now we basically depend on x86 store order semantics (and the CpuPause() loop around AllCpusInSync, kinda provides a barrier.) So the BSP writes in this order: ApicIdMap[Idx]=x; ... ->AllCpusInSync = true And whenever the AP sees ->AllCpusInSync == True, it has to now see ApicIdMap[Idx] == x. Yes. Maybe the thing to do is to detail this barrier in a commit note/comment? That would be nice. And add the MemoryFence() but not the volatile. Yes, that should work. Please *do* add the volatile, and also the MemoryFence(). When built with Visual Studio, MemoryFence() does nothing at all (at least when LTO is in effect -- which it almost always is). So we should have the volatile for making things work, and MemoryFence() as a conceptual reminder, so we know where to fix up things, when (if!) we come around fixing this mess with MemoryFence(). Reference: https://edk2.groups.io/g/rfc/message/500 https://edk2.groups.io/g/rfc/message/501 https://edk2.groups.io/g/rfc/message/502 https://edk2.groups.io/g/rfc/message/503 Did see it on the thread. Yeah agreed, Visual Studio does necessitate volatile here. Will add. Thanks Ankur Thanks! Laszlo -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#71427): https://edk2.groups.io/g/devel/message/71427 Mute This Topic: https://groups.io/mt/80199926/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 9/9] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
On 2021-02-03 12:45 p.m., Laszlo Ersek wrote:
On 02/03/21 06:46, Ankur Arora wrote:
On 2021-02-01 9:37 a.m., Laszlo Ersek wrote:
(6) Please drop this hunk. We don't try to be smarter than QEMU, in
general, whenever we perform feature negotiation.
Also, AFAICS, we will do the hotplug (and now hot-unplug) even if it wasn't
negotiated?
Yes, totally. We don't try to "evict" CpuHotplugSmm in case the related
features are not supported/offered by QEMU, we'll just leave
CpuHotplugSmm unused.
Here's why: the SMI feature negotiation interface is locked down at a
certain point; the negotiation of all of the feature bits needs to
happen centrally, in a common spot; and it would require a really quirky
solution in the firmware to let independent drivers negotiate *subsets*
of the features.
Right, I see your point. Firmware doesn't really get to stand on
ceremony when HW asks it to do stuff.
Thanks
Ankur
You have correctly determined that SmmControl2Dxe, the runtime DXE
driver that produces EFI_SMM_CONTROL2_PROTOCOL, has nothing much to do
with CPU hot-(un)plug. It's just that this is the driver that first
used, and therefore now *owns*, the SMI feature negotiation. (See commit
5ba203b54e59 ("OvmfPkg/SmmControl2Dxe: negotiate
ICH9_LPC_SMI_F_CPU_HOTPLUG", 2020-08-24).)
So, to reformulate your question/statement: the firmware will retain the
ability to do hot-(un)plug even if QEMU doesn't contain (or enable)
those particular features.
Thanks
Laszlo
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71133): https://edk2.groups.io/g/devel/message/71133
Mute This Topic: https://groups.io/mt/80199973/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 6/9] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
On 2021-02-03 12:36 p.m., Laszlo Ersek wrote: On 02/03/21 06:20, Ankur Arora wrote: Just as a sidenote, I do see two copies of the mCpuHotEjectData in the PiSmmCpuSmm and CpuHotplugSmm maps (which makes sense, given that both include SmmCpuFeaturesLib): .bss.mCpuHotEjectData 0x00017d60 0x8 /tmp/PiSmmCpuDxeSmm.dll.0k4hl8.ltrans1.ltrans.o .bss.mCpuHotEjectData 0x5110 0x8 /tmp/CpuHotplugSmm.dll.ixiN9a.ltrans0.ltrans.o I imagine they do get unified in the build process later, but that's the point my understanding stops. The PiSmmCpuDxeSmm binary has a (static global) variable called "mCpuHotEjectData" via OVMF's SmmCpuFeaturesLib instance, from this patch (patch#6). The CpuHotplugSmm binary has a (static global) variable called "mCpuHotEjectData" because the "CpuHotplug.c" source file defines that variable, from patch#7. (CpuHotplugSmm does not consume the SmmCpuFeaturesLib class -- it has no reason to.) In other words, there's nothing common between the two variables, beyond the name. If you rename the first to mCpuHotEjectData1, and the second to mCpuHotEjectData2, just for the experiment's sake, nothing will break. Yeah you are right. I completely forgot that I had defined mCpuHotEjectData in CpuHotplug.c and then assumed that it was because we link with SmmCpuFeaturesLib.a Sorry for the confusion. Ankur PiSmmCpuDxeSmm and CpuHotplugSmm never get unified in the build process; they are independent binaries. Thanks Laszlo -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#71129): https://edk2.groups.io/g/devel/message/71129 Mute This Topic: https://groups.io/mt/80199922/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
On 2021-02-03 12:55 p.m., Laszlo Ersek wrote:
On 02/03/21 07:13, Ankur Arora wrote:
On 2021-02-02 6:00 a.m., Laszlo Ersek wrote:
On 02/01/21 21:12, Ankur Arora wrote:
On 2021-02-01 11:08 a.m., Laszlo Ersek wrote:
(16) This function uses a data structure for communication between BSP
and APs -- mCpuHotEjectData->ApicIdMap is modified in UnplugCpus() on
the BSP, and checked above by the APs (too).
What guarantees the visibility of mCpuHotEjectData->ApicIdMap?
I was banking on SmiRendezvous() explicitly signalling that all
processing on the BSP was done before any AP will look at
mCpuHotEjectData in SmmCpuFeaturesRendezvousExit().
1716 //
1717 // Wait for BSP's signal to exit SMI
1718 //
1719 while (*mSmmMpSyncData->AllCpusInSync) {
1720 CpuPause ();
1721 }
1722 }
1723
1724 Exit:
1725 SmmCpuFeaturesRendezvousExit (CpuIndex);
Right; it's a general pattern in edk2: volatile UINT8 (aka BOOLEAN)
objects are considered atomic. (See
SMM_DISPATCHER_MP_SYNC_DATA.AllCpusInSync -- it's a pointer to a
volatile BOOLEAN.)
But our UINT64 values are neither volatile nor UINT8, and I got suddenly
doubtful about "AllCpusInSync" working as a multiprocessor barrier.
(I could be unjustifiedly worried, as a bunch of other fields in
SMM_DISPATCHER_MP_SYNC_DATA are volatile, wider than UINT8, and *not*
accessed with InterlockedCompareExchageXx().)
Thanks for pointing me to this code. There's a curious comment in
about making this structure uncache-able in the declaration here
(though I couldn't figure out how that is done):
418 typedef struct {
419 //
420 // Pointer to an array. The array should be located immediately
after this structure
421 // so that UC cache-ability can be set together.
422 //
This is probably through SMRR manipulation.
The "UefiCpuPkg/Library/SmmCpuFeaturesLib" instance contains SMRR support.
The "OvmfPkg/Library/SmmCpuFeaturesLib" instance contains no SMRR
support. (Just search both source files for "SMRR".)
Oh, now I see what SMRR does. Thanks that helps make sense of
what's going on here.
Ankur
423 SMM_CPU_DATA_BLOCK *CpuData;
424 volatile UINT32 *Counter;
425 volatile UINT32 BspIndex;
426 volatile BOOLEAN *InsideSmm;
427 volatile BOOLEAN *AllCpusInSync;
428 volatile SMM_CPU_SYNC_MODE EffectiveSyncMode;
429 volatile BOOLEAN SwitchBsp;
430 volatile BOOLEAN *CandidateBsp;
431 EFI_AP_PROCEDURE StartupProcedure;
432 VOID *StartupProcArgs;
433 } SMM_DISPATCHER_MP_SYNC_DATA;
Also, is there an expectation that these fields (at least some of
them) switch over when a new leader is chosen?
Yes, see for example the "Elect BSP" section in SmiRendezvous().
Otherwise I'm not sure why for instance, AllCpusInSync would be
a pointer.
TBH I can't explain that; I'm not too familiar with those parts...
CpuEject():
218 ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
For the to-be-ejected-AP, this value can only move from
valid-APIC-ID (=> wait in CpuDeadLoop()) -> CPU_EJECT_INVALID.
Given that, by the time the worker does the write on line 254, this
AP is guaranteed to be dead already, I don't think there's any
scenario where the to-be-ejected-AP can see anything other than
a valid-APIC-ID.
The scenario I had in mind was different: what guarantees that the
effect of
375 mCpuHotEjectData->ApicIdMap[ProcessorNum] =
(UINT64)RemoveApicId;
which is performed by the BSP in UnplugCpus(), is visible by the AP on
line 218 (see your quote above)?
What if the AP gets to line 218 before the BSP's write on line 375
*propagates* sufficiently?
I understand. That does make sense. And, as you said elsewhere, a real
memory fence would come in useful here.
We could use AsmCpuid() as a poor man's mfence, but that seems overkill
given that x86 at least guarantees store-order.
Right -- I don't recall any examples of AsmCpuid() being used like that
in edk2.
Thanks!
Laszlo
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71128): https://edk2.groups.io/g/devel/message/71128
Mute This Topic: https://groups.io/mt/80199926/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
On 2021-02-03 12:58 p.m., Laszlo Ersek wrote: On 02/03/21 07:45, Ankur Arora wrote: On 2021-02-02 6:15 a.m., Laszlo Ersek wrote: On 02/02/21 15:00, Laszlo Ersek wrote: ... I guess that volatile-qualifying both CPU_HOT_EJECT_DATA, and the array pointed-to by CPU_HOT_EJECT_DATA.ApicIdMap, should suffice. In combination with the sync-up point that you quoted. This seems to match existing practice in PiSmmCpuDxeSmm -- there are no concurrent accesses, so atomicity is not a concern, and serializing the instruction streams coarsely, with the sync-up, in combination with volatile accesses, should presumably guarantee visibility (on x86 anyway). To summarize, this is what I would ask for: - make CPU_HOT_EJECT_DATA volatile - make (*CPU_HOT_EJECT_DATA.ApicIdMap) volatile - after storing something to CPU_HOT_EJECT_DATA or CPU_HOT_EJECT_DATA.ApicIdMap on the BSP, execute a MemoryFence() - before fetching something from CPU_HOT_EJECT_DATA or CPU_HOT_EJECT_DATA.ApicIdMap on an AP, execute a MemoryFence() Except: MemoryFence() isn't a *memory fence* in fact. See "MdePkg/Library/BaseLib/X64/GccInline.c". It's just a compiler barrier, which may not add anything beyond what we'd already have from "volatile". Case in point: PiSmmCpuDxeSmm performs heavy multi-processing, but does not contain a single invocation of MemoryFence(). It uses volatile objects, and a handful of InterlockedCompareExchangeXx() calls, for implementing semaphores. (NB: there is no 8-bit variant of InterlockedCompareExchange(), as "volatile UINT8" is considered atomic in itself, and a suitable basis for a sempahore too.) And given the synchronization from those semaphores, PiSmmCpuDpxeSmm trusts that updates to the *other* volatile objects are both atomic and visible. I'm pretty sure this only works because x86 is in-order. There are instruction stream barriers in place, and compiler barriers too, but no actual memory barriers. Right and just to separate them explicitly, there are two problems: - compiler: where the compiler caches stuff in or looks at stale memory locations. Now, AFAICS, this should not happen because the ApicIdMap would never change once set so the compiler should reasonably be able to cache the address of ApicIdMap and dereference it (thus obviating the need for volatile.) (CPU_HOT_EJECT_DATA.Handler does change though.) Yeah, I did kinda elide over that. Let me think this through in v7 and add more explicit comments and then we can see if it still looks fishy? Thanks Ankur The compiler could, however, cache any assignments to ApicIdMap[Idx] (especially given LTO) and we could use a MemoryFence() (as the compiler barrier that it is) to force the store. - CPU pipeline: as you said, right now we basically depend on x86 store order semantics (and the CpuPause() loop around AllCpusInSync, kinda provides a barrier.) So the BSP writes in this order: ApicIdMap[Idx]=x; ... ->AllCpusInSync = true And whenever the AP sees ->AllCpusInSync == True, it has to now see ApicIdMap[Idx] == x. Yes. Maybe the thing to do is to detail this barrier in a commit note/comment? That would be nice. And add the MemoryFence() but not the volatile. Yes, that should work. Thanks, Laszlo -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#71127): https://edk2.groups.io/g/devel/message/71127 Mute This Topic: https://groups.io/mt/80199926/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
On 2021-02-02 6:15 a.m., Laszlo Ersek wrote: On 02/02/21 15:00, Laszlo Ersek wrote: ... I guess that volatile-qualifying both CPU_HOT_EJECT_DATA, and the array pointed-to by CPU_HOT_EJECT_DATA.ApicIdMap, should suffice. In combination with the sync-up point that you quoted. This seems to match existing practice in PiSmmCpuDxeSmm -- there are no concurrent accesses, so atomicity is not a concern, and serializing the instruction streams coarsely, with the sync-up, in combination with volatile accesses, should presumably guarantee visibility (on x86 anyway). To summarize, this is what I would ask for: - make CPU_HOT_EJECT_DATA volatile - make (*CPU_HOT_EJECT_DATA.ApicIdMap) volatile - after storing something to CPU_HOT_EJECT_DATA or CPU_HOT_EJECT_DATA.ApicIdMap on the BSP, execute a MemoryFence() - before fetching something from CPU_HOT_EJECT_DATA or CPU_HOT_EJECT_DATA.ApicIdMap on an AP, execute a MemoryFence() Except: MemoryFence() isn't a *memory fence* in fact. See "MdePkg/Library/BaseLib/X64/GccInline.c". It's just a compiler barrier, which may not add anything beyond what we'd already have from "volatile". Case in point: PiSmmCpuDxeSmm performs heavy multi-processing, but does not contain a single invocation of MemoryFence(). It uses volatile objects, and a handful of InterlockedCompareExchangeXx() calls, for implementing semaphores. (NB: there is no 8-bit variant of InterlockedCompareExchange(), as "volatile UINT8" is considered atomic in itself, and a suitable basis for a sempahore too.) And given the synchronization from those semaphores, PiSmmCpuDpxeSmm trusts that updates to the *other* volatile objects are both atomic and visible. I'm pretty sure this only works because x86 is in-order. There are instruction stream barriers in place, and compiler barriers too, but no actual memory barriers. Right and just to separate them explicitly, there are two problems: - compiler: where the compiler caches stuff in or looks at stale memory locations. Now, AFAICS, this should not happen because the ApicIdMap would never change once set so the compiler should reasonably be able to cache the address of ApicIdMap and dereference it (thus obviating the need for volatile.) The compiler could, however, cache any assignments to ApicIdMap[Idx] (especially given LTO) and we could use a MemoryFence() (as the compiler barrier that it is) to force the store. - CPU pipeline: as you said, right now we basically depend on x86 store order semantics (and the CpuPause() loop around AllCpusInSync, kinda provides a barrier.) So the BSP writes in this order: ApicIdMap[Idx]=x; ... ->AllCpusInSync = true And whenever the AP sees ->AllCpusInSync == True, it has to now see ApicIdMap[Idx] == x. Maybe the thing to do is to detail this barrier in a commit note/comment? And add the MemoryFence() but not the volatile. Thanks Ankur Now the question is whether we have managed to *sufficiently* imitate these patterns from PiSmmCpuDxeSmm, in this patch set. Making stuff volatile, and relying on the existent sync-up point, might suffice. Thanks Laszlo -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#71095): https://edk2.groups.io/g/devel/message/71095 Mute This Topic: https://groups.io/mt/80199926/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
On 2021-02-02 6:00 a.m., Laszlo Ersek wrote:
On 02/01/21 21:12, Ankur Arora wrote:
On 2021-02-01 11:08 a.m., Laszlo Ersek wrote:
apologies, I've got more comments here:
On 01/29/21 01:59, Ankur Arora wrote:
/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(),
+ on each CPU at exit from SMM.
+
+ If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is being ejected, wait in a CpuDeadLoop()
+ until ejected.
+
+ @param[in] ProcessorNum Index of executing CPU.
+
+**/
+VOID
+EFIAPI
+CpuEject (
+ IN UINTN ProcessorNum
+ )
+{
+ //
+ // APIC ID is UINT32, but mCpuHotEjectData->ApicIdMap[] is UINT64
+ // so use UINT64 throughout.
+ //
+ UINT64 ApicId;
+
+ ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
+ if (ApicId == CPU_EJECT_INVALID) {
+return;
+ }
+
+ //
+ // CPU(s) being unplugged get here from
SmmCpuFeaturesSmiRendezvousExit()
+ // after having been cleared to exit the SMI by the monarch and
thus have
+ // no SMM processing remaining.
+ //
+ // Given that we cannot allow them to escape to the guest, we pen
them
+ // here until the SMM monarch tells the HW to unplug them.
+ //
+ CpuDeadLoop ();
+}
(15) There is no such function as SmmCpuFeaturesSmiRendezvousExit() --
it's SmmCpuFeaturesRendezvousExit().
(16) This function uses a data structure for communication between BSP
and APs -- mCpuHotEjectData->ApicIdMap is modified in UnplugCpus() on
the BSP, and checked above by the APs (too).
What guarantees the visibility of mCpuHotEjectData->ApicIdMap?
I was banking on SmiRendezvous() explicitly signalling that all
processing on the BSP was done before any AP will look at
mCpuHotEjectData in SmmCpuFeaturesRendezvousExit().
1716 //
1717 // Wait for BSP's signal to exit SMI
1718 //
1719 while (*mSmmMpSyncData->AllCpusInSync) {
1720 CpuPause ();
1721 }
1722 }
1723
1724 Exit:
1725 SmmCpuFeaturesRendezvousExit (CpuIndex);
Right; it's a general pattern in edk2: volatile UINT8 (aka BOOLEAN)
objects are considered atomic. (See
SMM_DISPATCHER_MP_SYNC_DATA.AllCpusInSync -- it's a pointer to a
volatile BOOLEAN.)
But our UINT64 values are neither volatile nor UINT8, and I got suddenly
doubtful about "AllCpusInSync" working as a multiprocessor barrier.
(I could be unjustifiedly worried, as a bunch of other fields in
SMM_DISPATCHER_MP_SYNC_DATA are volatile, wider than UINT8, and *not*
accessed with InterlockedCompareExchageXx().)
Thanks for pointing me to this code. There's a curious comment in
about making this structure uncache-able in the declaration here
(though I couldn't figure out how that is done):
418 typedef struct {
419 //
420 // Pointer to an array. The array should be located immediately after
this structure
421 // so that UC cache-ability can be set together.
422 //
423 SMM_CPU_DATA_BLOCK*CpuData;
424 volatile UINT32 *Counter;
425 volatile UINT32 BspIndex;
426 volatile BOOLEAN *InsideSmm;
427 volatile BOOLEAN *AllCpusInSync;
428 volatile SMM_CPU_SYNC_MODEEffectiveSyncMode;
429 volatile BOOLEAN SwitchBsp;
430 volatile BOOLEAN *CandidateBsp;
431 EFI_AP_PROCEDURE StartupProcedure;
432 VOID *StartupProcArgs;
433 } SMM_DISPATCHER_MP_SYNC_DATA;
Also, is there an expectation that these fields (at least some of
them) switch over when a new leader is chosen?
Otherwise I'm not sure why for instance, AllCpusInSync would be
a pointer.
I think we might want to use InterlockedCompareExchange64() in both
EjectCpu() and UnplugCpus() (and make "ApicIdMap" volatile, in
addition). InterlockedCompareExchange64() can be used just for
comparison as well, by passing ExchangeValue=CompareValue.
Speaking specifically about the ApicIdMap, I'm not sure I fully
agree (assuming my comment just above is correct.)
The only AP (reader) ApicIdMap deref is here:
CpuEject():
218 ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
For the to-be-ejected-AP, this value can only move from
valid-APIC-ID (=> wait in CpuDeadLoop()) -> CPU_EJECT_INVALID.
Given that, by the time the worker does the write on line 254, this
AP is guaranteed to be dead already, I don't think there's any
scenario where the to-be-ejected-AP can see anything other than
a valid-APIC-ID.
The scenario I had in mind was different: what guarantees that the
effect of
375mCpuHotEjectData->ApicIdMap[ProcessorNum] = (UINT64)RemoveApicId;
which is performed by the BSP in UnplugCpus(), is visible by the AP on
line 218 (see your quote above)?
What if the AP gets to line 218 before the BSP's write on line 375
*propagates* sufficiently?
I understand. That does make sense. And, as you said elsewhere, a real
memory fence would come in useful here.
We could use AsmCpuid() as a poor man's mfence,
Re: [edk2-devel] [PATCH v6 9/9] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
On 2021-02-01 9:37 a.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
As part of the negotiation treat ICH9_LPC_SMI_F_CPU_HOT_UNPLUG as a
subfeature of feature flag ICH9_LPC_SMI_F_CPU_HOTPLUG, so enable it
only if the other is also being negotiated.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 25 ++---
1 file changed, 22 insertions(+), 3 deletions(-)
diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
index c9d875543205..e70f3f8b58cb 100644
--- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
+++ b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
@@ -29,6 +29,13 @@
//
#define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
+// The following bit value stands for "enable CPU hot unplug, and inject an SMI
(1) s/hot unplug/hot-unplug/
+// with control value ICH9_APM_CNT_CPU_HOT_UNPLUG upon hot unplug", in the
(2) There is no such thing as ICH9_APM_CNT_CPU_HOT_UNPLUG; we use the
same SMI command value ICH9_APM_CNT_CPU_HOTPLUG (= 4) for unplug.
In QEMU, the macro is called OVMF_CPUHP_SMI_CMD.
(3) s/hot unplug/hot-unplug/.
+// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
+// Is only negotiated alongside ICH9_LPC_SMI_F_CPU_HOTPLUG.
(4) Please drop the last sentence (see more on it below).
+//
+#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
+
//
// Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
// for the S3 boot script fragment to write to and read from.
@@ -112,7 +119,8 @@ NegotiateSmiFeatures (
QemuFwCfgReadBytes (sizeof mSmiFeatures, );
//
- // We want broadcast SMI, SMI on CPU hotplug, and nothing else.
+ // We want broadcast SMI, SMI on CPU hotplug, on CPU hot-unplug
+ // and nothing else.
//
RequestedFeaturesMask = ICH9_LPC_SMI_F_BROADCAST;
if (!MemEncryptSevIsEnabled ()) {
(5) Please spell out the full expression "SMI on CPU hot-unplug".
@@ -120,8 +128,18 @@ NegotiateSmiFeatures (
// For now, we only support hotplug with SEV disabled.
//
RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOTPLUG;
+RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
}
mSmiFeatures &= RequestedFeaturesMask;
+
+ if (!(mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) &&
+ (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG)) {
+DEBUG ((DEBUG_WARN, "%a CPU host-features %Lx, requested mask %Lx\n",
+ __FUNCTION__, mSmiFeatures, RequestedFeaturesMask));
+
+mSmiFeatures &= ~ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
+ }
+
QemuFwCfgSelectItem (mRequestedFeaturesItem);
QemuFwCfgWriteBytes (sizeof mSmiFeatures, );
(6) Please drop this hunk. We don't try to be smarter than QEMU, in
general, whenever we perform feature negotiation.
Also, AFAICS, we will do the hotplug (and now hot-unplug) even if it wasn't
negotiated?
For example, the pre-patch code doesn't attempt to notice if QEMU
acknowledges ICH9_LPC_SMI_F_CPU_HOTPLUG but not ICH9_LPC_SMI_F_BROADCAST.
@@ -162,8 +180,9 @@ NegotiateSmiFeatures (
if ((mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) == 0) {
DEBUG ((DEBUG_INFO, "%a: CPU hotplug not negotiated\n", __FUNCTION__));
} else {
-DEBUG ((DEBUG_INFO, "%a: CPU hotplug with SMI negotiated\n",
- __FUNCTION__));
+DEBUG ((DEBUG_INFO, "%a: CPU hotplug%s with SMI negotiated\n",
+ __FUNCTION__,
+ (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG) ? ", unplug" : ""));
}
//
(7) Rather than combining these two in a common debug message, please
just add a separate "if" that follows the whole pattern seen with
ICH9_LPC_SMI_F_CPU_HOTPLUG. Thus, for each feature bit we care about,
we'll have a dedicated log message, saying yes or no.
Acking this set (and the ones down-thread.) Will fix.
Thanks
Ankur
Thanks!
Laszlo
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71093): https://edk2.groups.io/g/devel/message/71093
Mute This Topic: https://groups.io/mt/80199973/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 8/9] OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
On 2021-02-02 5:23 a.m., Laszlo Ersek wrote:
On 02/01/21 20:21, Ankur Arora wrote:
On 2021-02-01 9:22 a.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Designate a worker CPU (we use the one executing the root MMI
handler), which will do the actual ejection via QEMU in CpuEject().
CpuEject(), on the worker CPU, ejects each marked CPU by first
selecting its APIC ID and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
it out of the SMI.
CpuEject(), on the CPU being ejected, spins around in its holding
area until this final context-switch. This does mean that there is
some CPU state that would ordinarily be restored (in SmiRendezvous()
and in SmiEntry.nasm::CommonHandler), but will not be anymore.
This unrestored state includes FPU state, CET enable, stuffing of
RSB and the final RSM. Since the CPU state is destroyed by QEMU,
this should be okay.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 73
++
1 file changed, 67 insertions(+), 6 deletions(-)
(1) s/CpuEject/EjectCpu/g, per previous request (affects commit message
and code too).
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 526f51faf070..bf91344eef9c 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -193,9 +193,12 @@ RevokeNewSlot:
CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(),
on each CPU at exit from SMM.
- If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is neither a worker, nor being ejected, nothing
+ to be done.
If, the executing CPU is being ejected, wait in a CpuDeadLoop()
until ejected.
+ If, the executing CPU is a worker CPU, set QEMU CPU status to eject
+ for CPUs being ejected.
@param[in] ProcessorNum Index of executing CPU.
@@ -217,6 +220,56 @@ CpuEject (
return;
}
+ if (ApicId == CPU_EJECT_WORKER) {
(2) The CPU_EJECT_WORKER approach is needlessly complicated (speculative
generality). I wish I understood this idea earlier in the patch set.
(2a) In patch #5 (subject "OvmfPkg/CpuHotplugSmm: define
CPU_HOT_EJECT_DATA"), the CPU_EJECT_WORKER macro definition should be
dropped.
(2b) In this patch, the question whether the executing CPU is the BSP or
not, should be decided with the same logic that is visible in
PlatformSmmBspElection()
[OvmfPkg/Library/SmmCpuPlatformHookLibQemu/SmmCpuPlatformHookLibQemu.c]:
MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr;
BOOLEAN IsBsp;
ApicBaseMsr.Uint64 = AsmReadMsr64 (MSR_IA32_APIC_BASE);
IsBsp = (BOOLEAN)(ApicBaseMsr.Bits.BSP == 1);
(2c) Point (2b) obviates the explicit "mark as worker" logic entirely,
in UnplugCpus() below.
(2d) The "is worker" language (in comments etc) should be replaced with
direct "is BSP" language.
+ UINT32 CpuIndex;
+
+ for (CpuIndex = 0; CpuIndex < mCpuHotEjectData->ArrayLength;
CpuIndex++) {
+ UINT64 RemoveApicId;
+
+ RemoveApicId = mCpuHotEjectData->ApicIdMap[CpuIndex];
+
+ if ((RemoveApicId != CPU_EJECT_INVALID &&
+ RemoveApicId != CPU_EJECT_WORKER)) {
+ //
+ // This to-be-ejected-CPU has already received the BSP's SMI
exit
+ // signal and, will execute SmmCpuFeaturesSmiRendezvousExit()
+ // followed by this callback or is already waiting in the
+ // CpuDeadLoop() below.
+ //
+ // Tell QEMU to context-switch it out.
+ //
+ QemuCpuhpWriteCpuSelector (mMmCpuIo, (APIC_ID) RemoveApicId);
+ QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
(3) While the QEMU CPU selector value *usually* matches the APIC ID,
it's not an invariant. APIC IDs have an internal structure, composed of
bit-fields, where each bit-field accommodates one hierarchy level in the
CPU topology (thread, core, die (maybe), and socket).
However, this mapping need not be surjective. QEMU lets you create
"pathological" CPU topologies, for example one with:
- 3 threads/core,
- 5 cores/socket,
- (say) 2 sockets.
Under that example, the bit-field standing for the "thread number" level
would have 2 bits, theoretically permitting *4* threads/core, and the
bit-field standing for the "core number" level would have 3 bits,
theoretically allowing for *8* cores/socket.
Considering the fully populated topology, you'd see the CPU selector
range from 0 to (3*5*2-1)=29, inclusive (corresponding to 30 logical
processors in total). However, the APIC ID *image* of this CPU selector
*domain* would not be "contiguous" -- the APIC ID space, with the
above-described structu
Re: [edk2-devel] [PATCH v6 6/9] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
On 2021-02-01 5:36 a.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection state
between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
CpuHotplugSmm also sets up the CPU ejection mechanism via
CPU_HOT_EJECT_DATA->Handler.
Additionally, expose CPU_HOT_EJECT_DATA via PcdCpuHotEjectDataAddress.
(1) Please mention that the logic is added to
SmmCpuFeaturesSmmRelocationComplete(), and so it will run as part of the
PiSmmCpuDxeSmm entry point function, PiCpuSmmEntry().
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 3 +
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 78 ++
2 files changed, 81 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..32c63722ee62 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -35,4 +35,7 @@ [LibraryClasses]
UefiBootServicesTableLib
[Pcd]
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugSupport
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..33dd5da92432 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -14,7 +14,9 @@
#include
#include
#include
+#include
(2) The MemoryAllocationLib class is not listed in the [LibraryClasses]
section of "OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf"; so
please list it there as well.
(Please keep the [LibraryClasses] section in the INF file sorted, while
at it.)
#include
+#include
(3) This will change, once you move the header file under
"OvmfPkg/Include/Pcd/"; either way, please keep the #include directives
alphabetically sorted.
(Before this patch, the #include list is well-sorted.)
#include
#include
#include
@@ -171,6 +173,70 @@ SmmCpuFeaturesHookReturnFromSmm (
return OriginalInstructionPointer;
}
+GLOBAL_REMOVE_IF_UNREFERENCED
(4a) This is useless unless building with MSVC; I don't really remember
introducing any instance of this macro myself, ever. I suggest dropping
it.
Will drop. As you said, it's a NOP for !MSVC.
Just as a sidenote, I do see two copies of the mCpuHotEjectData in
the PiSmmCpuSmm and CpuHotplugSmm maps (which makes sense, given
that both include SmmCpuFeaturesLib):
.bss.mCpuHotEjectData
0x00017d600x8 /tmp/PiSmmCpuDxeSmm.dll.0k4hl8.ltrans1.ltrans.o
.bss.mCpuHotEjectData
0x51100x8 /tmp/CpuHotplugSmm.dll.ixiN9a.ltrans0.ltrans.o
I imagine they do get unified in the build process later, but that's the point
my understanding stops.
(4b) On the other hand, STATIC it should be.
Yeah, that it should be.
+CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+/**
+ Initialize CpuHotEjectData if PcdCpuHotPlugSupport is enabled
+ and, if more than 1 CPU is configured.
+
+ Also sets up the corresponding PcdCpuHotEjectDataAddress.
+**/
(5) typo: s/CpuHotEjectData/mCpuHotEjectData/
(6) As requested elsewhere under v6, there's no need to make this
dependent on PcdCpuHotPlugSupport.
(7) "Initialize" is imperative mood, "sets up" is indicative mood.
Either one is fine, just be consistent please.
Will fix, and thanks for the very careful reading.
Thanks
Ankur
+STATIC
+VOID
+SmmCpuFeaturesSmmInitHotEject (
(8) This is a STATIC function (i.e., it has internal linkage); there's
no need to complicate its name with the "SmmCpuFeatures..." prefix.
I suggest "InitCpuHotEjectData".
+ VOID
+ )
+{
+ UINT32 mMaxNumberOfCpus;
(9) This is a variable with automatic storage duration, so the "m"
prefix is invalid.
+ EFI_STATUS Status;
+
+ if (!FeaturePcdGet (PcdCpuHotPlugSupport)) {
+return;
+ }
(10a) Please drop this, per prior discussion.
(10b) Please drop the PCD from the INF file too.
(11) In the rest of this function, the comment style is incorrect in
several spots. The idiomatic style is:
//
// Blah.
//
I.e., normally we'd need leading and trailing empty comment lines.
*However*, most of those comments don't really explain much beyond
what's emergent from the code anyway, to me anyway, thus, I would simply
suggest dropping those comments.
+
+ // PcdCpuHotPlugSupport => PcdCpuMaxLogicalProcessorNumber
+ mMaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProce
Re: [edk2-devel] [PATCH v6 4/9] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
On 2021-01-31 7:13 p.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().
With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 84 ++
1 file changed, 84 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 05b1f8cb63a6..70d69f6ed65b 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -188,6 +188,88 @@ RevokeNewSlot:
}
/**
+ Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, report the CPU to PiSmmCpuDxeSmm via
+ EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is
+ unknown, skip it silently.
+
+ @param[in] ToUnplugApicIdsThe APIC IDs of the CPUs that are about to be
+hot-unplugged.
+
+ @param[in] ToUnplugCount The number of filled-in APIC IDs in
+ToUnplugApicIds.
+
+ @retval EFI_SUCCESS Known APIC IDs have been removed from SMM data
+structures.
+
+ @return Error codes propagated from
+mMmCpuService->RemoveProcessor().
+
(1) Please drop this empty line (just before the '**/').
+**/
+STATIC
+EFI_STATUS
+UnplugCpus (
+ IN APIC_ID *ToUnplugApicIds,
+ IN UINT32 ToUnplugCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 ToUnplugIdx;
+ UINTN ProcessorNum;
+
+ ToUnplugIdx = 0;
+ while (ToUnplugIdx < ToUnplugCount) {
+APIC_IDRemoveApicId;
+
+RemoveApicId = ToUnplugApicIds[ToUnplugIdx];
+
+//
+// mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+// the ProcessorNum for the APIC ID to be removed.
+//
+for (ProcessorNum = 0;
+ ProcessorNum < mCpuHotPlugData->ArrayLength;
+ ProcessorNum++) {
+ if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+break;
+ }
+}
+
+//
+// Ignore the unplug if APIC ID not found
+//
+if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_INFO, "%a: did not find APIC ID " FMT_APIC_ID
+ " to unplug\n", __FUNCTION__, RemoveApicId));
(2) Please use DEBUG_VERBOSE here.
(I agree that we should have *one* DEBUG_INFO message that relates to
the removal of an individual processor; however, I think we should emit
that message when we finally signal QEMU to eject the processor.)
Based on our discussion around establishing the correspondence between
The ProcessorNum, APIC-ID and CPU selector, I'll change this to
DEBUG_VERBOSE and add a new DEBUG_INFO print after successfully
putting it in the APICIdMap.
(3) Please un-indent ("outdent"?) the second line by two spaces.
+ ToUnplugIdx++;
+ continue;
+}
+
+//
+// Mark ProcessorNum for removal from SMM data structures
+//
+Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+
(4) It would be more idiomatic to remove this empty line (between Status
assignment and check).
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, RemoveApicId, Status));
+ goto Fatal;
(5) Please just "return Status" here, and drop the "Fatal" label.
+}
+
+ToUnplugIdx++;
+ }
+
+ //
+ // We've removed this set of APIC IDs from SMM data structures.
+ //
+ return EFI_SUCCESS;
+
+Fatal:
+ return Status;
+}
+
+/**
CPU Hotplug MMI handler function.
This is a root MMI handler.
@@ -303,6 +385,8 @@ CpuHotplugMmi (
if (PluggedCount > 0) {
Status = ProcessHotAddedCpus (mPluggedApicIds, PluggedCount);
+ } else if (ToUnplugCount > 0) {
+Status = UnplugCpus (mToUnplugApicIds, ToUnplugCount);
}
if (EFI_ERROR(Status)) {
(6) Hmm... What's the reason for the exclusivity?
Why is the following not better:
if (PluggedCount > 0) {
Status = ProcessHotAddedCpus (mPluggedApicIds, PluggedCount);
if (EFI_ERROR (Status)) {
goto Fatal;
}
}
if (ToUnplugCount > 0) {
Status = UnplugCpus (mToUnplugApicIds, ToUnplugCount);
if (EFI_ERROR (Status)) {
goto Fatal;
}
}
QemuCpuhpCollectApicIds() intentionally populates both arrays in a
single go. As I suggeste
Re: [edk2-devel] [PATCH v6 1/9] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
On 2021-01-29 5:15 p.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
ProcessHotAddedCpus(). This is in preparation for supporting CPU
hot-unplug.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
> + if (EFI_ERROR(Status)) {
> +goto Fatal;
>}
(13) Without having seen the rest of the patches, I think this error
check should be nested under the same (PluggedCount > 0) condition; in
other words, I think it only makes sense to check Status after we
actually call ProcessHotAddedCpus().
Addresses all comments from v5, except for this one, since the (lack) of
nesting makes more sense after patch 4, "OvmfPkg/CpuHotplugSmm: introduce
UnplugCpus()".
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 214 ++---
1 file changed, 129 insertions(+), 85 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..05b1f8cb63a6 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,130 @@ STATIC UINT32 mPostSmmPenAddress;
//
STATIC EFI_HANDLE mDispatchHandle;
+/**
+ Process CPUs that have been hot-added, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, relocate the SMBASE, and report the CPU to PiSmmCpuDxeSmm
+ via EFI_SMM_CPU_SERVICE_PROTOCOL. If the supposedly hot-added CPU is already
+ known, skip it silently.
+
+ @param[in] PluggedApicIdsThe APIC IDs of the CPUs that have been
+ hot-plugged.
+
+ @param[in] PluggedCount The number of filled-in APIC IDs in
+ PluggedApicIds.
+
+ @retval EFI_SUCCESS CPUs corresponding to all the APIC IDs are
+ populated.
+
+ @retval EFI_OUT_OF_RESOURCES Out of APIC ID space in "mCpuHotPlugData".
+
+ @return Error codes propagated from SmbaseRelocate()
+ and mMmCpuService->AddProcessor().
+
+**/
+STATIC
+EFI_STATUS
+ProcessHotAddedCpus (
+ IN APIC_ID *PluggedApicIds,
+ IN UINT32 PluggedCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 PluggedIdx;
+ UINT32 NewSlot;
+
+ //
+ // The Post-SMM Pen need not be reinstalled multiple times within a single
+ // root MMI handling. Even reinstalling once per root MMI is only prudence;
+ // in theory installing the pen in the driver's entry point function should
+ // suffice.
+ //
+ SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+
+ PluggedIdx = 0;
+ NewSlot = 0;
+ while (PluggedIdx < PluggedCount) {
+APIC_ID NewApicId;
+UINT32 CheckSlot;
+UINTN NewProcessorNumberByProtocol;
+
+NewApicId = PluggedApicIds[PluggedIdx];
+
+//
+// Check if the supposedly hot-added CPU is already known to us.
+//
+for (CheckSlot = 0;
+ CheckSlot < mCpuHotPlugData->ArrayLength;
+ CheckSlot++) {
+ if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
+break;
+ }
+}
+if (CheckSlot < mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
+"before; ignoring it\n", __FUNCTION__, NewApicId));
+ PluggedIdx++;
+ continue;
+}
+
+//
+// Find the first empty slot in CPU_HOT_PLUG_DATA.
+//
+while (NewSlot < mCpuHotPlugData->ArrayLength &&
+ mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
+ NewSlot++;
+}
+if (NewSlot == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
+__FUNCTION__, NewApicId));
+ return EFI_OUT_OF_RESOURCES;
+}
+
+//
+// Store the APIC ID of the new processor to the slot.
+//
+mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
+
+//
+// Relocate the SMBASE of the new CPU.
+//
+Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
+ mPostSmmPenAddress);
+if (EFI_ERROR (Status)) {
+ goto RevokeNewSlot;
+}
+
+//
+// Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
+//
+Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
+ );
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, NewApicId, Status));
+ goto RevokeNewSlot;
+}
+
+DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
+ "EFI_SMM_CPU_SERVICE_PROTOCOL a
Re: [edk2-devel] [PATCH v6 5/9] OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
On 2021-01-31 8:53 p.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/OvmfPkg.dec | 10 +
OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf | 1 +
OvmfPkg/Include/Library/CpuHotEjectData.h | 35 +++
3 files changed, 46 insertions(+)
create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 4348bb45c64a..1a2debb821d7 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -106,6 +106,10 @@ [LibraryClasses]
#
XenPlatformLib|Include/Library/XenPlatformLib.h
+ ## @libraryclass Share CPU hot-eject state
+ #
+ CpuHotEjectData|Include/Library/CpuHotEjectData.h
+
[Guids]
gUefiOvmfPkgTokenSpaceGuid= {0x93bb96af, 0xb9f2, 0x4eb8, {0x94,
0x62, 0xe0, 0xba, 0x74, 0x56, 0x42, 0x36}}
gEfiXenInfoGuid = {0xd3b46f3b, 0xd441, 0x1244, {0x9a,
0x12, 0x0, 0x12, 0x27, 0x3f, 0xc1, 0x4d}}
(1) Please drop this hunk -- the [LibraryClasses] section should not be
modified, as we're not introducing a new library class.
@@ -352,6 +356,12 @@ [PcdsDynamic, PcdsDynamicEx]
# This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+ ## This PCD adds a communication channel between PiSmmCpuDxeSmm and
+ # CpuHotplugSmm.
(2) I suggest:
## This PCD adds a communication channel between OVMF's SmmCpuFeaturesLib
# instance in PiSmmCpuDxeSmm, and CpuHotplugSmm.
+ #
+ # Only accessed if PcdCpuHotPlugSupport is TRUE
(3) This statement is technically true, but I suggest dropping it. In my
opinion, it is not useful (it's a superfluous statement). Here's why:
- We set the "PcdCpuHotPlugSupport" feature flag to TRUE in the OVMF DSC
files exactly when the SMM_REQUIRE feature test macro is set on the
"build" command line.
- The whole SMM infrastructure is included in the firmware binary
exactly when SMM_REQUIRE is set.
In other words, PcdCpuHotPlugSupport is *equivalent* with
SmmCpuFeaturesLib, PiSmmCpuDxeSmm, and CpuHotplugSmm being included in
the firmware binary.
Given that the first comment already declares the PCD as an info channel
between SmmCpuFeaturesLib (as built into PiSmmCpuDxeSmm) and
CpuHotplugSmm, the second comment adds nothing.
That makes sense.
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
+
[PcdsFeatureFlag]
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
index 04322b0d7855..e08b572ef169 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
@@ -54,6 +54,7 @@ [Protocols]
[Pcd]
gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress##
CONSUMES
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress ## CONSUMES
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase ##
CONSUMES
[FeaturePcd]
(4) Please move this hunk to patch#7 (subject: "OvmfPkg/CpuHotplugSmm:
add CpuEject()"). That's where CpuHotplugSmm first needs the new PCD.
diff --git a/OvmfPkg/Include/Library/CpuHotEjectData.h
b/OvmfPkg/Include/Library/CpuHotEjectData.h
new file mode 100644
index ..b6fb629a1283
--- /dev/null
+++ b/OvmfPkg/Include/Library/CpuHotEjectData.h
@@ -0,0 +1,35 @@
+/** @file
+ Definition for a CPU hot-eject state sharing structure.
+
(5a) I suggest the following language:
Definition of the CPU_HOT_EJECT_DATA structure, which shares CPU hot-eject
state between OVMF's SmmCpuFeaturesLib instance in PiSmmCpuDxeSmm, and
CpuHotplugSmm.
CPU_HOT_EJECT_DATA is allocated in SMRAM, and pointed-to by
PcdCpuHotEjectDataAddress.
(5b) Please append at least one more sentence to state the condition
when the PCD is *not* NULL.
(6) This new header file should be located at:
OvmfPkg/Include/Pcd/CpuHotEjectData.h
Your explanation below makes sense. OvmfPkg/Include/Library did not
quite seem like the right place but I wasn't sure what would be
better.
Will fix.
please.
The (more or less) general rule is this:
- if you have a macro definition or a structure type that is accessible
through a Pcd, a Protocol, a Guid -- HOB, VenHw() devpath node etc --,
a Library, a Register, etc,
- and the Pcd, Protocol, Guid, Library etc in question is declared in
Re: [edk2-devel] [PATCH v6 3/9] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
On 2021-01-29 6:36 p.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 22 ++
2 files changed, 28 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..804809846890 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -30,6 +30,12 @@ QemuCpuhpReadCpuStatus (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
);
+VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ );
+
UINT32
QemuCpuhpReadCommandData (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index f871e50c377b..ed44264de934 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -67,6 +67,28 @@ QemuCpuhpReadCpuStatus (
return CpuStatus;
}
+VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ )
+{
+ EFI_STATUS Status;
+
+ Status = MmCpuIo->Io.Write (
+ MmCpuIo,
+ MM_IO_UINT8,
+ ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+ 1,
+
+ );
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+ASSERT (FALSE);
+CpuDeadLoop ();
+ }
+}
+
UINT32
QemuCpuhpReadCommandData (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
The code is fine, but please move the new function (both declaration and
definition) between QemuCpuhpWriteCpuSelector() and QemuCpuhpWriteCommand().
Reason: the pre-patch order of the functions matches the order of the
register descriptions in QEMU's "docs/specs/acpi_cpu_hotplug.txt".
There, we first have a section called "read access", then another called
"write access". And in each section, registers are listed in increasing
offset order, within the hotplug register block.
Will fix.
Thanks
Ankur
Thanks!
Laszlo
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71048): https://edk2.groups.io/g/devel/message/71048
Mute This Topic: https://groups.io/mt/80199903/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 2/9] OvmfPkg/CpuHotplugSmm: collect hot-unplug events
On 2021-01-29 6:18 p.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Process fw_remove events in QemuCpuhpCollectApicIds() and collect
corresponding APIC IDs for CPUs that are being hot-unplugged.
In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.
This is based on the QEMU hot-unplug protocol documented here:
https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imamm...@redhat.com/
Also define QEMU_CPUHP_STAT_EJECTED while we are at it.
(1) Please move the addition of QEMU_CPUHP_STAT_EJECTED to patch 8
("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection"), where you
first use it.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
I'm treating events (insert=1, fw_remove=1) below as invalid (return
EFI_PROTOCOL_ERROR, which ends up as an assert), but I'm not sure
that is correct:
if ((CpuStatus & QEMU_CPUHP_STAT_INSERT) != 0) {
//
// The "insert" event guarantees the "enabled" status; plus it
excludes
- // the "remove" event.
+ // the "fw_remove" event.
//
if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0 ||
- (CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+ (CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
"inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
CpuStatus));
QEMU's handling in cpu_hotplug_rd() can return both of these:
cpu_hotplug_rd() {
...
case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
val |= cdev->cpu ? 1 : 0;
val |= cdev->is_inserting ? 2 : 0;
val |= cdev->is_removing ? 4 : 0;
val |= cdev->fw_remove ? 16 : 0;
...
}
and I don't see any code that treats is_inserting and is_removing as
exclusive.
One specific case where this looks it might be a problem is if the user
unplugs a CPU and right after that plugs it.
As part of the unplug handling, the ACPI AML would, in the scan loop,
asynchronously trigger the notify, which would do the OS unplug, set
"fw_remove" and then call the SMI_CMD.
The subsequent plug could then come and set the "insert" bit.
Assuming what I'm describing could happen, I'm not sure what's the right
handling: QEMU could treat these bits as exclusive and then OVMF could
justifiably treat it as a protocol error?
I'm OK with the related part of your patch (i.e., returning
EFI_PROTOCOL_ERROR for (insert=1, fw_remove=1)).
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 29 +++
2 files changed, 26 insertions(+), 5 deletions(-)
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..692e3072598c 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,8 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_EJECTEDBIT3
+#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 8d4a6693c8d6..f871e50c377b 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -245,10 +245,10 @@ QemuCpuhpCollectApicIds (
if ((CpuStatus & QEMU_CPUHP_STAT_INSERT) != 0) {
//
// The "insert" event guarantees the "enabled" status; plus it excludes
- // the "remove" event.
+ // the "fw_remove" event.
//
if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0 ||
- (CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+ (CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
"inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
CpuStatus));
@@ -260,12 +260,31 @@ QemuCpuhpCollectApicIds (
ExtendIds = PluggedApicIds;
ExtendCount = PluggedCount;
-} else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
- DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: remove\n", __FUNCTION__,
-CurrentSelector));
+} else if ((CpuStatus & QEMU_CPUHP_STAT_
Re: [edk2-devel] [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
On 2021-02-01 11:08 a.m., Laszlo Ersek wrote:
apologies, I've got more comments here:
On 01/29/21 01:59, Ankur Arora wrote:
/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(),
+ on each CPU at exit from SMM.
+
+ If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is being ejected, wait in a CpuDeadLoop()
+ until ejected.
+
+ @param[in] ProcessorNum Index of executing CPU.
+
+**/
+VOID
+EFIAPI
+CpuEject (
+ IN UINTN ProcessorNum
+ )
+{
+ //
+ // APIC ID is UINT32, but mCpuHotEjectData->ApicIdMap[] is UINT64
+ // so use UINT64 throughout.
+ //
+ UINT64 ApicId;
+
+ ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
+ if (ApicId == CPU_EJECT_INVALID) {
+return;
+ }
+
+ //
+ // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
+ // after having been cleared to exit the SMI by the monarch and thus have
+ // no SMM processing remaining.
+ //
+ // Given that we cannot allow them to escape to the guest, we pen them
+ // here until the SMM monarch tells the HW to unplug them.
+ //
+ CpuDeadLoop ();
+}
(15) There is no such function as SmmCpuFeaturesSmiRendezvousExit() --
it's SmmCpuFeaturesRendezvousExit().
(16) This function uses a data structure for communication between BSP
and APs -- mCpuHotEjectData->ApicIdMap is modified in UnplugCpus() on
the BSP, and checked above by the APs (too).
What guarantees the visibility of mCpuHotEjectData->ApicIdMap?
I was banking on SmiRendezvous() explicitly signalling that all
processing on the BSP was done before any AP will look at
mCpuHotEjectData in SmmCpuFeaturesRendezvousExit().
1716 //
1717 // Wait for BSP's signal to exit SMI
1718 //
1719 while (*mSmmMpSyncData->AllCpusInSync) {
1720 CpuPause ();
1721 }
1722 }
1723
1724 Exit:
1725 SmmCpuFeaturesRendezvousExit (CpuIndex);
I think we might want to use InterlockedCompareExchange64() in both
EjectCpu() and UnplugCpus() (and make "ApicIdMap" volatile, in
addition). InterlockedCompareExchange64() can be used just for
comparison as well, by passing ExchangeValue=CompareValue.
Speaking specifically about the ApicIdMap, I'm not sure I fully
agree (assuming my comment just above is correct.)
The only AP (reader) ApicIdMap deref is here:
CpuEject():
218 ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
For the to-be-ejected-AP, this value can only move from
valid-APIC-ID (=> wait in CpuDeadLoop()) -> CPU_EJECT_INVALID.
Given that, by the time the worker does the write on line 254, this
AP is guaranteed to be dead already, I don't think there's any
scenario where the to-be-ejected-AP can see anything other than
a valid-APIC-ID.
241 QemuCpuhpWriteCpuSelector (mMmCpuIo, (APIC_ID) RemoveApicId);
242 QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
243
244 //
245 // Compiler barrier to ensure the next store isn't reordered
246 //
247 MemoryFence ();
248
249 //
250 // Clear the eject status for CpuIndex to ensure that an invalid
251 // SMI later does not end up trying to eject it or a newly
252 // hotplugged CpuIndex does not go into the dead loop.
253 //
254 mCpuHotEjectData->ApicIdMap[CpuIndex] = CPU_EJECT_INVALID;
For APs that are not being ejected, they will always see CPU_EJECT_INVALID
since the writer never changes that.
The one scenario in which bad things could happen is if entries in the
ApicIdMap are unaligned (or if the compiler or cpu-arch tears aligned
writes).
(17) I think a similar observation applies to the "Handler" field too,
as APs call it, while the BSP keeps flipping it between NULL and a real
function address. We might have to turn that field into an
From a real function address, to NULL is the problem part right?
(Same argument as above for the transition in UnplugCpus() from
NULL -> function-address.)
EFI_PHYSICAL_ADDRESS (just a fancy name for UINT64), and use
InterlockedCompareExchange64() again.
AFAICS, these are the problematic derefs:
SmmCpuFeaturesRendezvousExit():
450 if (mCpuHotEjectData == NULL ||
451 mCpuHotEjectData->Handler == NULL) {
452 return;
and problematic assignments:
266 //
267 // We are done until the next hot-unplug; clear the handler.
268 //
269 mCpuHotEjectData->Handler = NULL;
270 return;
271 }
Here as well, I've been banking on aligned writes such that the APs would
only see the before or after value not an intermediate value.
Thanks
Ankur
Thanks
Laszlo
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#71031): https://edk2.groups.io/g/devel/message/71031
Mute This Topic: https://groups.io/mt/80199926/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v6 8/9] OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
On 2021-02-01 9:22 a.m., Laszlo Ersek wrote:
On 01/29/21 01:59, Ankur Arora wrote:
Designate a worker CPU (we use the one executing the root MMI
handler), which will do the actual ejection via QEMU in CpuEject().
CpuEject(), on the worker CPU, ejects each marked CPU by first
selecting its APIC ID and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
it out of the SMI.
CpuEject(), on the CPU being ejected, spins around in its holding
area until this final context-switch. This does mean that there is
some CPU state that would ordinarily be restored (in SmiRendezvous()
and in SmiEntry.nasm::CommonHandler), but will not be anymore.
This unrestored state includes FPU state, CET enable, stuffing of
RSB and the final RSM. Since the CPU state is destroyed by QEMU,
this should be okay.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 73 ++
1 file changed, 67 insertions(+), 6 deletions(-)
(1) s/CpuEject/EjectCpu/g, per previous request (affects commit message
and code too).
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 526f51faf070..bf91344eef9c 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -193,9 +193,12 @@ RevokeNewSlot:
CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(),
on each CPU at exit from SMM.
- If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is neither a worker, nor being ejected, nothing
+ to be done.
If, the executing CPU is being ejected, wait in a CpuDeadLoop()
until ejected.
+ If, the executing CPU is a worker CPU, set QEMU CPU status to eject
+ for CPUs being ejected.
@param[in] ProcessorNum Index of executing CPU.
@@ -217,6 +220,56 @@ CpuEject (
return;
}
+ if (ApicId == CPU_EJECT_WORKER) {
(2) The CPU_EJECT_WORKER approach is needlessly complicated (speculative
generality). I wish I understood this idea earlier in the patch set.
(2a) In patch #5 (subject "OvmfPkg/CpuHotplugSmm: define
CPU_HOT_EJECT_DATA"), the CPU_EJECT_WORKER macro definition should be
dropped.
(2b) In this patch, the question whether the executing CPU is the BSP or
not, should be decided with the same logic that is visible in
PlatformSmmBspElection()
[OvmfPkg/Library/SmmCpuPlatformHookLibQemu/SmmCpuPlatformHookLibQemu.c]:
MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr;
BOOLEAN IsBsp;
ApicBaseMsr.Uint64 = AsmReadMsr64 (MSR_IA32_APIC_BASE);
IsBsp = (BOOLEAN)(ApicBaseMsr.Bits.BSP == 1);
(2c) Point (2b) obviates the explicit "mark as worker" logic entirely,
in UnplugCpus() below.
(2d) The "is worker" language (in comments etc) should be replaced with
direct "is BSP" language.
+UINT32 CpuIndex;
+
+for (CpuIndex = 0; CpuIndex < mCpuHotEjectData->ArrayLength; CpuIndex++) {
+ UINT64 RemoveApicId;
+
+ RemoveApicId = mCpuHotEjectData->ApicIdMap[CpuIndex];
+
+ if ((RemoveApicId != CPU_EJECT_INVALID &&
+ RemoveApicId != CPU_EJECT_WORKER)) {
+//
+// This to-be-ejected-CPU has already received the BSP's SMI exit
+// signal and, will execute SmmCpuFeaturesSmiRendezvousExit()
+// followed by this callback or is already waiting in the
+// CpuDeadLoop() below.
+//
+// Tell QEMU to context-switch it out.
+//
+QemuCpuhpWriteCpuSelector (mMmCpuIo, (APIC_ID) RemoveApicId);
+QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
(3) While the QEMU CPU selector value *usually* matches the APIC ID,
it's not an invariant. APIC IDs have an internal structure, composed of
bit-fields, where each bit-field accommodates one hierarchy level in the
CPU topology (thread, core, die (maybe), and socket).
However, this mapping need not be surjective. QEMU lets you create
"pathological" CPU topologies, for example one with:
- 3 threads/core,
- 5 cores/socket,
- (say) 2 sockets.
Under that example, the bit-field standing for the "thread number" level
would have 2 bits, theoretically permitting *4* threads/core, and the
bit-field standing for the "core number" level would have 3 bits,
theoretically allowing for *8* cores/socket.
Considering the fully populated topology, you'd see the CPU selector
range from 0 to (3*5*2-1)=29, inclusive (corresponding to 30 logical
processors in total). However, the APIC ID *image* of this CPU selector
*domain* would not be "contiguous" -- the APIC ID space, with the
above-described structure, would accommodate 4*8*2=64 logical
processors. For example, each APIC ID that stood for the none
[edk2-devel] [PATCH v6 9/9] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
As part of the negotiation treat ICH9_LPC_SMI_F_CPU_HOT_UNPLUG as a
subfeature of feature flag ICH9_LPC_SMI_F_CPU_HOTPLUG, so enable it
only if the other is also being negotiated.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 25 ++---
1 file changed, 22 insertions(+), 3 deletions(-)
diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
index c9d875543205..e70f3f8b58cb 100644
--- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
+++ b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
@@ -29,6 +29,13 @@
//
#define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
+// The following bit value stands for "enable CPU hot unplug, and inject an SMI
+// with control value ICH9_APM_CNT_CPU_HOT_UNPLUG upon hot unplug", in the
+// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
+// Is only negotiated alongside ICH9_LPC_SMI_F_CPU_HOTPLUG.
+//
+#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
+
//
// Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
// for the S3 boot script fragment to write to and read from.
@@ -112,7 +119,8 @@ NegotiateSmiFeatures (
QemuFwCfgReadBytes (sizeof mSmiFeatures, );
//
- // We want broadcast SMI, SMI on CPU hotplug, and nothing else.
+ // We want broadcast SMI, SMI on CPU hotplug, on CPU hot-unplug
+ // and nothing else.
//
RequestedFeaturesMask = ICH9_LPC_SMI_F_BROADCAST;
if (!MemEncryptSevIsEnabled ()) {
@@ -120,8 +128,18 @@ NegotiateSmiFeatures (
// For now, we only support hotplug with SEV disabled.
//
RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOTPLUG;
+RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
}
mSmiFeatures &= RequestedFeaturesMask;
+
+ if (!(mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) &&
+ (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG)) {
+DEBUG ((DEBUG_WARN, "%a CPU host-features %Lx, requested mask %Lx\n",
+ __FUNCTION__, mSmiFeatures, RequestedFeaturesMask));
+
+mSmiFeatures &= ~ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
+ }
+
QemuFwCfgSelectItem (mRequestedFeaturesItem);
QemuFwCfgWriteBytes (sizeof mSmiFeatures, );
@@ -162,8 +180,9 @@ NegotiateSmiFeatures (
if ((mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) == 0) {
DEBUG ((DEBUG_INFO, "%a: CPU hotplug not negotiated\n", __FUNCTION__));
} else {
-DEBUG ((DEBUG_INFO, "%a: CPU hotplug with SMI negotiated\n",
- __FUNCTION__));
+DEBUG ((DEBUG_INFO, "%a: CPU hotplug%s with SMI negotiated\n",
+ __FUNCTION__,
+ (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG) ? ", unplug" : ""));
}
//
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70878): https://edk2.groups.io/g/devel/message/70878
Mute This Topic: https://groups.io/mt/80199973/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v6 4/9] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().
With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 84 ++
1 file changed, 84 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 05b1f8cb63a6..70d69f6ed65b 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -188,6 +188,88 @@ RevokeNewSlot:
}
/**
+ Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, report the CPU to PiSmmCpuDxeSmm via
+ EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is
+ unknown, skip it silently.
+
+ @param[in] ToUnplugApicIdsThe APIC IDs of the CPUs that are about to be
+hot-unplugged.
+
+ @param[in] ToUnplugCount The number of filled-in APIC IDs in
+ToUnplugApicIds.
+
+ @retval EFI_SUCCESS Known APIC IDs have been removed from SMM data
+structures.
+
+ @return Error codes propagated from
+mMmCpuService->RemoveProcessor().
+
+**/
+STATIC
+EFI_STATUS
+UnplugCpus (
+ IN APIC_ID *ToUnplugApicIds,
+ IN UINT32 ToUnplugCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 ToUnplugIdx;
+ UINTN ProcessorNum;
+
+ ToUnplugIdx = 0;
+ while (ToUnplugIdx < ToUnplugCount) {
+APIC_IDRemoveApicId;
+
+RemoveApicId = ToUnplugApicIds[ToUnplugIdx];
+
+//
+// mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+// the ProcessorNum for the APIC ID to be removed.
+//
+for (ProcessorNum = 0;
+ ProcessorNum < mCpuHotPlugData->ArrayLength;
+ ProcessorNum++) {
+ if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+break;
+ }
+}
+
+//
+// Ignore the unplug if APIC ID not found
+//
+if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_INFO, "%a: did not find APIC ID " FMT_APIC_ID
+ " to unplug\n", __FUNCTION__, RemoveApicId));
+ ToUnplugIdx++;
+ continue;
+}
+
+//
+// Mark ProcessorNum for removal from SMM data structures
+//
+Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, RemoveApicId, Status));
+ goto Fatal;
+}
+
+ToUnplugIdx++;
+ }
+
+ //
+ // We've removed this set of APIC IDs from SMM data structures.
+ //
+ return EFI_SUCCESS;
+
+Fatal:
+ return Status;
+}
+
+/**
CPU Hotplug MMI handler function.
This is a root MMI handler.
@@ -303,6 +385,8 @@ CpuHotplugMmi (
if (PluggedCount > 0) {
Status = ProcessHotAddedCpus (mPluggedApicIds, PluggedCount);
+ } else if (ToUnplugCount > 0) {
+Status = UnplugCpus (mToUnplugApicIds, ToUnplugCount);
}
if (EFI_ERROR(Status)) {
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70877): https://edk2.groups.io/g/devel/message/70877
Mute This Topic: https://groups.io/mt/80199964/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v6 2/9] OvmfPkg/CpuHotplugSmm: collect hot-unplug events
Process fw_remove events in QemuCpuhpCollectApicIds() and collect
corresponding APIC IDs for CPUs that are being hot-unplugged.
In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.
This is based on the QEMU hot-unplug protocol documented here:
https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imamm...@redhat.com/
Also define QEMU_CPUHP_STAT_EJECTED while we are at it.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
I'm treating events (insert=1, fw_remove=1) below as invalid (return
EFI_PROTOCOL_ERROR, which ends up as an assert), but I'm not sure
that is correct:
if ((CpuStatus & QEMU_CPUHP_STAT_INSERT) != 0) {
//
// The "insert" event guarantees the "enabled" status; plus it
excludes
- // the "remove" event.
+ // the "fw_remove" event.
//
if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0 ||
- (CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+ (CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
"inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
CpuStatus));
QEMU's handling in cpu_hotplug_rd() can return both of these:
cpu_hotplug_rd() {
...
case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
val |= cdev->cpu ? 1 : 0;
val |= cdev->is_inserting ? 2 : 0;
val |= cdev->is_removing ? 4 : 0;
val |= cdev->fw_remove ? 16 : 0;
...
}
and I don't see any code that treats is_inserting and is_removing as
exclusive.
One specific case where this looks it might be a problem is if the user
unplugs a CPU and right after that plugs it.
As part of the unplug handling, the ACPI AML would, in the scan loop,
asynchronously trigger the notify, which would do the OS unplug, set
"fw_remove" and then call the SMI_CMD.
The subsequent plug could then come and set the "insert" bit.
Assuming what I'm describing could happen, I'm not sure what's the right
handling: QEMU could treat these bits as exclusive and then OVMF could
justifiably treat it as a protocol error?
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 29 +++
2 files changed, 26 insertions(+), 5 deletions(-)
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..692e3072598c 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,8 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_EJECTEDBIT3
+#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 8d4a6693c8d6..f871e50c377b 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -245,10 +245,10 @@ QemuCpuhpCollectApicIds (
if ((CpuStatus & QEMU_CPUHP_STAT_INSERT) != 0) {
//
// The "insert" event guarantees the "enabled" status; plus it excludes
- // the "remove" event.
+ // the "fw_remove" event.
//
if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0 ||
- (CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+ (CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
"inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
CpuStatus));
@@ -260,12 +260,31 @@ QemuCpuhpCollectApicIds (
ExtendIds = PluggedApicIds;
ExtendCount = PluggedCount;
-} else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
- DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: remove\n", __FUNCTION__,
-CurrentSelector));
+} else if ((CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
+ //
+ // "fw_remove" event guarantees "enabled".
+ //
+ if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0) {
+DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
+ "inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
+ CpuStatus));
+retur
[edk2-devel] [PATCH v6 0/9] support CPU hot-unplug
Hi,
This series adds support for CPU hot-unplug with OVMF.
Please see this in conjunction with the QEMU secureboot hot-unplug v2
series posted here (now upstreamed):
https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/
Patches 1, and 3,
("OvmfPkg/CpuHotplugSmm: refactor hotplug logic"),
("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
are either refactors or add support functions.
Patch 2, and 9,
("OvmfPkg/CpuHotplugSmm: collect hot-unplug events"),
("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
handle the QEMU protocol logic for collection of CPU hot-unplug
events or the protocol negotiation.
Patch 4,
("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
adds the MMI logic for CPU hot-unplug handling and informing
the PiSmmCpuDxeSmm of CPU removal.
Patches 5, and 6,
("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA"),
("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
sets up state for doing the CPU ejection as part of hot-unplug.
Patches 7, and 8,
("OvmfPkg/CpuHotplugSmm: add CpuEject()"),
("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
add the CPU ejection logic.
Testing (with QEMU 5.2.50):
- Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
- Synthetic tests with simultaneous multi CPU hot-unplug
- Negotiation with/without CPU hotplug enabled
Also at:
github.com/terminus/edk2/ hot-unplug-v6
Changelog:
v6:
- addresses v5 review comments.
v5:
- fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add
add Qemu Cpu Status helper").
URL:
https://patchew.org/EDK2/20210126064440.299596-1-ankur.a.ar...@oracle.com/
v4:
- Gets rid of unnecessary UefiCpuPkg changes
URL:
https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.ar...@oracle.com/
v3:
- Use a saner PCD based interface to share state between PiSmmCpuDxeSmm
and OvmfPkg/CpuHotplugSmm
- Cleaner split of the hot-unplug code
URL:
https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/
v2:
- Do the ejection via SmmCpuFeaturesRendezvousExit()
URL:
https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/
RFC:
URL:
https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/
Please review.
Thanks
Ankur
Ankur Arora (9):
OvmfPkg/CpuHotplugSmm: refactor hotplug logic
OvmfPkg/CpuHotplugSmm: collect hot-unplug events
OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
OvmfPkg/CpuHotplugSmm: add CpuEject()
OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
OvmfPkg/OvmfPkg.dec| 10 +
OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf| 1 +
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 3 +
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 +
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 +
OvmfPkg/Include/Library/CpuHotEjectData.h | 35 ++
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 472 +
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 51 ++-
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 78
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 25 +-
10 files changed, 584 insertions(+), 99 deletions(-)
create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70875): https://edk2.groups.io/g/devel/message/70875
Mute This Topic: https://groups.io/mt/80199948/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v6 6/9] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection state
between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
CpuHotplugSmm also sets up the CPU ejection mechanism via
CPU_HOT_EJECT_DATA->Handler.
Additionally, expose CPU_HOT_EJECT_DATA via PcdCpuHotEjectDataAddress.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 3 +
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 78 ++
2 files changed, 81 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..32c63722ee62 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -35,4 +35,7 @@ [LibraryClasses]
UefiBootServicesTableLib
[Pcd]
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugSupport
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..33dd5da92432 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -14,7 +14,9 @@
#include
#include
#include
+#include
#include
+#include
#include
#include
#include
@@ -171,6 +173,70 @@ SmmCpuFeaturesHookReturnFromSmm (
return OriginalInstructionPointer;
}
+GLOBAL_REMOVE_IF_UNREFERENCED
+CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+/**
+ Initialize CpuHotEjectData if PcdCpuHotPlugSupport is enabled
+ and, if more than 1 CPU is configured.
+
+ Also sets up the corresponding PcdCpuHotEjectDataAddress.
+**/
+STATIC
+VOID
+SmmCpuFeaturesSmmInitHotEject (
+ VOID
+ )
+{
+ UINT32 mMaxNumberOfCpus;
+ EFI_STATUS Status;
+
+ if (!FeaturePcdGet (PcdCpuHotPlugSupport)) {
+return;
+ }
+
+ // PcdCpuHotPlugSupport => PcdCpuMaxLogicalProcessorNumber
+ mMaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+
+ // No spare CPUs to hot-eject
+ if (mMaxNumberOfCpus == 1) {
+return;
+ }
+
+ mCpuHotEjectData =
+(CPU_HOT_EJECT_DATA *)AllocatePool (sizeof (*mCpuHotEjectData));
+ ASSERT (mCpuHotEjectData != NULL);
+
+ //
+ // Allocate buffer for pointers to array in CPU_HOT_EJECT_DATA.
+ //
+
+ // Revision
+ mCpuHotEjectData->Revision = CPU_HOT_EJECT_DATA_REVISION_1;
+
+ // Array Length of APIC ID
+ mCpuHotEjectData->ArrayLength = mMaxNumberOfCpus;
+
+ // CpuIndex -> APIC ID map
+ mCpuHotEjectData->ApicIdMap = (UINT64 *)AllocatePool
+ (sizeof (*mCpuHotEjectData->ApicIdMap) * mCpuHotEjectData->ArrayLength);
+
+ // Hot-eject handler
+ mCpuHotEjectData->Handler = NULL;
+
+ // Reserved
+ mCpuHotEjectData->Reserved = 0;
+
+ ASSERT (mCpuHotEjectData->ApicIdMap != NULL);
+
+ //
+ // Expose address of CPU Hot eject Data structure
+ //
+ Status = PcdSet64S (PcdCpuHotEjectDataAddress,
+ (UINT64)(VOID *)mCpuHotEjectData);
+ ASSERT_EFI_ERROR (Status);
+}
+
/**
Hook point in normal execution mode that allows the one CPU that was elected
as monarch during System Management Mode initialization to perform additional
@@ -188,6 +254,9 @@ SmmCpuFeaturesSmmRelocationComplete (
UINTN MapPagesBase;
UINTN MapPagesCount;
+
+ SmmCpuFeaturesSmmInitHotEject ();
+
if (!MemEncryptSevIsEnabled ()) {
return;
}
@@ -375,6 +444,15 @@ SmmCpuFeaturesRendezvousExit (
IN UINTN CpuIndex
)
{
+ //
+ // CPU Hot-eject not enabled.
+ //
+ if (mCpuHotEjectData == NULL ||
+ mCpuHotEjectData->Handler == NULL) {
+return;
+ }
+
+ mCpuHotEjectData->Handler (CpuIndex);
}
/**
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70872): https://edk2.groups.io/g/devel/message/70872
Mute This Topic: https://groups.io/mt/80199922/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v6 8/9] OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
Designate a worker CPU (we use the one executing the root MMI
handler), which will do the actual ejection via QEMU in CpuEject().
CpuEject(), on the worker CPU, ejects each marked CPU by first
selecting its APIC ID and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
it out of the SMI.
CpuEject(), on the CPU being ejected, spins around in its holding
area until this final context-switch. This does mean that there is
some CPU state that would ordinarily be restored (in SmiRendezvous()
and in SmiEntry.nasm::CommonHandler), but will not be anymore.
This unrestored state includes FPU state, CET enable, stuffing of
RSB and the final RSM. Since the CPU state is destroyed by QEMU,
this should be okay.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 73 ++
1 file changed, 67 insertions(+), 6 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 526f51faf070..bf91344eef9c 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -193,9 +193,12 @@ RevokeNewSlot:
CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(),
on each CPU at exit from SMM.
- If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is neither a worker, nor being ejected, nothing
+ to be done.
If, the executing CPU is being ejected, wait in a CpuDeadLoop()
until ejected.
+ If, the executing CPU is a worker CPU, set QEMU CPU status to eject
+ for CPUs being ejected.
@param[in] ProcessorNum Index of executing CPU.
@@ -217,6 +220,56 @@ CpuEject (
return;
}
+ if (ApicId == CPU_EJECT_WORKER) {
+UINT32 CpuIndex;
+
+for (CpuIndex = 0; CpuIndex < mCpuHotEjectData->ArrayLength; CpuIndex++) {
+ UINT64 RemoveApicId;
+
+ RemoveApicId = mCpuHotEjectData->ApicIdMap[CpuIndex];
+
+ if ((RemoveApicId != CPU_EJECT_INVALID &&
+ RemoveApicId != CPU_EJECT_WORKER)) {
+//
+// This to-be-ejected-CPU has already received the BSP's SMI exit
+// signal and, will execute SmmCpuFeaturesSmiRendezvousExit()
+// followed by this callback or is already waiting in the
+// CpuDeadLoop() below.
+//
+// Tell QEMU to context-switch it out.
+//
+QemuCpuhpWriteCpuSelector (mMmCpuIo, (APIC_ID) RemoveApicId);
+QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
+
+//
+// Compiler barrier to ensure the next store isn't reordered
+//
+MemoryFence ();
+
+//
+// Clear the eject status for CpuIndex to ensure that an invalid
+// SMI later does not end up trying to eject it or a newly
+// hotplugged CpuIndex does not go into the dead loop.
+//
+mCpuHotEjectData->ApicIdMap[CpuIndex] = CPU_EJECT_INVALID;
+
+DEBUG ((DEBUG_INFO, "%a: Unplugged CPU %u -> " FMT_APIC_ID "\n",
+ __FUNCTION__, CpuIndex, RemoveApicId));
+ }
+}
+
+//
+// Clear our own worker status.
+//
+mCpuHotEjectData->ApicIdMap[ProcessorNum] = CPU_EJECT_INVALID;
+
+//
+// We are done until the next hot-unplug; clear the handler.
+//
+mCpuHotEjectData->Handler = NULL;
+return;
+ }
+
//
// CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
// after having been cleared to exit the SMI by the monarch and thus have
@@ -327,6 +380,19 @@ UnplugCpus (
}
if (EjectCount != 0) {
+UINTN Worker;
+
+Status = mMmCpuService->WhoAmI (mMmCpuService, );
+ASSERT_EFI_ERROR (Status);
+//
+// UnplugCpus() is called via the root MMI handler and thus we are
+// executing in the BSP context.
+//
+// Mark ourselves as the worker CPU.
+//
+ASSERT (mCpuHotEjectData->ApicIdMap[Worker] == CPU_EJECT_INVALID);
+mCpuHotEjectData->ApicIdMap[Worker] = CPU_EJECT_WORKER;
+
//
// We have processors to be ejected; install the handler.
//
@@ -451,11 +517,6 @@ CpuHotplugMmi (
if (EFI_ERROR (Status)) {
goto Fatal;
}
- if (ToUnplugCount > 0) {
-DEBUG ((DEBUG_ERROR, "%a: hot-unplug is not supported yet\n",
- __FUNCTION__));
-goto Fatal;
- }
if (PluggedCount > 0) {
Status = ProcessHotAddedCpus (mPluggedApicIds, PluggedCount);
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70874): https://edk2.groups.io/g/devel/message/70874
Mute This Topic: https://groups.io/mt/80199930/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
Add CpuEject(), which handles the CPU ejection, and provides a holding
area for said CPUs. It is called via SmmCpuFeaturesRendezvousExit(),
at the tail end of the SMI handling.
Also UnplugCpus() now stashes APIC IDs of CPUs which need to be
ejected in CPU_HOT_EJECT_DATA.ApicIdMap. These are used by CpuEject()
to identify such CPUs.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 109 +++--
1 file changed, 105 insertions(+), 4 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 70d69f6ed65b..526f51faf070 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -14,6 +14,7 @@
#include // gMmst
#include // PcdGetBool()
#include // SafeUintnSub()
+#include // CPU_HOT_EJECT_DATA
#include // EFI_MM_CPU_IO_PROTOCOL
#include // EFI_SMM_CPU_SERVICE_PROTOCOL
#include// EFI_STATUS
@@ -32,11 +33,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo;
//
STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService;
//
-// This structure is a communication side-channel between the
+// These structures serve as communication side-channels between the
// EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider
// (i.e., PiSmmCpuDxeSmm).
//
STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData;
//
// SMRAM arrays for fetching the APIC IDs of processors with pending events (of
// known event types), for the time of just one MMI.
@@ -188,11 +190,53 @@ RevokeNewSlot:
}
/**
+ CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(),
+ on each CPU at exit from SMM.
+
+ If, the executing CPU is not being ejected, nothing to be done.
+ If, the executing CPU is being ejected, wait in a CpuDeadLoop()
+ until ejected.
+
+ @param[in] ProcessorNum Index of executing CPU.
+
+**/
+VOID
+EFIAPI
+CpuEject (
+ IN UINTN ProcessorNum
+ )
+{
+ //
+ // APIC ID is UINT32, but mCpuHotEjectData->ApicIdMap[] is UINT64
+ // so use UINT64 throughout.
+ //
+ UINT64 ApicId;
+
+ ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
+ if (ApicId == CPU_EJECT_INVALID) {
+return;
+ }
+
+ //
+ // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
+ // after having been cleared to exit the SMI by the monarch and thus have
+ // no SMM processing remaining.
+ //
+ // Given that we cannot allow them to escape to the guest, we pen them
+ // here until the SMM monarch tells the HW to unplug them.
+ //
+ CpuDeadLoop ();
+}
+
+/**
Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds().
For each such CPU, report the CPU to PiSmmCpuDxeSmm via
- EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is
- unknown, skip it silently.
+ EFI_SMM_CPU_SERVICE_PROTOCOL and stash the APIC ID for later ejection.
+ If the to be hot-unplugged CPU is unknown, skip it silently.
+
+ Additonally, if we do stash any APIC IDs, also install a CPU eject handler
+ which would handle the ejection.
@param[in] ToUnplugApicIdsThe APIC IDs of the CPUs that are about to be
hot-unplugged.
@@ -216,9 +260,11 @@ UnplugCpus (
{
EFI_STATUS Status;
UINT32 ToUnplugIdx;
+ UINT32 EjectCount;
UINTN ProcessorNum;
ToUnplugIdx = 0;
+ EjectCount = 0;
while (ToUnplugIdx < ToUnplugCount) {
APIC_IDRemoveApicId;
@@ -255,13 +301,41 @@ UnplugCpus (
DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
__FUNCTION__, RemoveApicId, Status));
goto Fatal;
+} else {
+ //
+ // Stash the APIC IDs so we can do the actual ejection later.
+ //
+ if (mCpuHotEjectData->ApicIdMap[ProcessorNum] != CPU_EJECT_INVALID) {
+//
+// Since ProcessorNum and APIC-ID map 1-1, so a valid
+// mCpuHotEjectData->ApicIdMap[ProcessorNum] means something
+// is horribly wrong.
+//
+DEBUG ((DEBUG_ERROR, "%a: ProcessorNum %u maps to %llx, cannot "
+"map to " FMT_APIC_ID "\n", __FUNCTION__, ProcessorNum,
+mCpuHotEjectData->ApicIdMap[ProcessorNum], RemoveApicId));
+
+Status = EFI_INVALID_PARAMETER;
+goto Fatal;
+ }
+
+ mCpuHotEjectData->ApicIdMap[ProcessorNum] = (UINT64)RemoveApicId;
+ EjectCount++;
}
ToUnplugIdx++;
}
+ if (EjectCount != 0) {
+//
+// We have processors to be ejected; install the handler.
+//
+mCpuHotEjectData->Handler = CpuEject;
+ }
+
//
- // We've removed this set of APIC IDs from SMM data structures.
+ // We've removed this set
[edk2-devel] [PATCH v6 5/9] OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/OvmfPkg.dec | 10 +
OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf | 1 +
OvmfPkg/Include/Library/CpuHotEjectData.h | 35 +++
3 files changed, 46 insertions(+)
create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 4348bb45c64a..1a2debb821d7 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -106,6 +106,10 @@ [LibraryClasses]
#
XenPlatformLib|Include/Library/XenPlatformLib.h
+ ## @libraryclass Share CPU hot-eject state
+ #
+ CpuHotEjectData|Include/Library/CpuHotEjectData.h
+
[Guids]
gUefiOvmfPkgTokenSpaceGuid= {0x93bb96af, 0xb9f2, 0x4eb8, {0x94,
0x62, 0xe0, 0xba, 0x74, 0x56, 0x42, 0x36}}
gEfiXenInfoGuid = {0xd3b46f3b, 0xd441, 0x1244, {0x9a,
0x12, 0x0, 0x12, 0x27, 0x3f, 0xc1, 0x4d}}
@@ -352,6 +356,12 @@ [PcdsDynamic, PcdsDynamicEx]
# This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+ ## This PCD adds a communication channel between PiSmmCpuDxeSmm and
+ # CpuHotplugSmm.
+ #
+ # Only accessed if PcdCpuHotPlugSupport is TRUE
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
+
[PcdsFeatureFlag]
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
index 04322b0d7855..e08b572ef169 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
@@ -54,6 +54,7 @@ [Protocols]
[Pcd]
gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress## CONSUMES
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress ## CONSUMES
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase ## CONSUMES
[FeaturePcd]
diff --git a/OvmfPkg/Include/Library/CpuHotEjectData.h
b/OvmfPkg/Include/Library/CpuHotEjectData.h
new file mode 100644
index ..b6fb629a1283
--- /dev/null
+++ b/OvmfPkg/Include/Library/CpuHotEjectData.h
@@ -0,0 +1,35 @@
+/** @file
+ Definition for a CPU hot-eject state sharing structure.
+
+ Copyright (C) 2021, Oracle Corporation.
+
+ SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#ifndef _CPU_HOT_EJECT_DATA_H_
+#define _CPU_HOT_EJECT_DATA_H_
+
+typedef
+VOID
+(EFIAPI *CPU_HOT_EJECT_FN)(
+ IN UINTN ProcessorNum
+ );
+
+#define CPU_EJECT_INVALID (MAX_UINT64)
+#define CPU_EJECT_WORKER(MAX_UINT64-1)
+
+#define CPU_HOT_EJECT_DATA_REVISION_1 0x0001
+
+typedef struct {
+ UINT32 Revision; // Used for version identification of
+ // this structure
+ UINT32 ArrayLength; // Entries in the ApicIdMap array
+
+ UINT64 *ApicIdMap;// Pointer to CpuIndex->ApicId map for
+ // pending hot-ejects
+ CPU_HOT_EJECT_FN Handler; // Handler to do the CPU ejection
+
+ UINT64 Reserved;
+} CPU_HOT_EJECT_DATA;
+
+#endif /* _CPU_HOT_EJECT_DATA_H_ */
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70871): https://edk2.groups.io/g/devel/message/70871
Mute This Topic: https://groups.io/mt/80199916/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v6 3/9] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 22 ++
2 files changed, 28 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..804809846890 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -30,6 +30,12 @@ QemuCpuhpReadCpuStatus (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
);
+VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ );
+
UINT32
QemuCpuhpReadCommandData (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index f871e50c377b..ed44264de934 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -67,6 +67,28 @@ QemuCpuhpReadCpuStatus (
return CpuStatus;
}
+VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ )
+{
+ EFI_STATUS Status;
+
+ Status = MmCpuIo->Io.Write (
+ MmCpuIo,
+ MM_IO_UINT8,
+ ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+ 1,
+
+ );
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+ASSERT (FALSE);
+CpuDeadLoop ();
+ }
+}
+
UINT32
QemuCpuhpReadCommandData (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70870): https://edk2.groups.io/g/devel/message/70870
Mute This Topic: https://groups.io/mt/80199903/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v6 1/9] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
ProcessHotAddedCpus(). This is in preparation for supporting CPU
hot-unplug.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
Notes:
> + if (EFI_ERROR(Status)) {
> +goto Fatal;
>}
(13) Without having seen the rest of the patches, I think this error
check should be nested under the same (PluggedCount > 0) condition; in
other words, I think it only makes sense to check Status after we
actually call ProcessHotAddedCpus().
Addresses all comments from v5, except for this one, since the (lack) of
nesting makes more sense after patch 4, "OvmfPkg/CpuHotplugSmm: introduce
UnplugCpus()".
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 214 ++---
1 file changed, 129 insertions(+), 85 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..05b1f8cb63a6 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,130 @@ STATIC UINT32 mPostSmmPenAddress;
//
STATIC EFI_HANDLE mDispatchHandle;
+/**
+ Process CPUs that have been hot-added, per QemuCpuhpCollectApicIds().
+
+ For each such CPU, relocate the SMBASE, and report the CPU to PiSmmCpuDxeSmm
+ via EFI_SMM_CPU_SERVICE_PROTOCOL. If the supposedly hot-added CPU is already
+ known, skip it silently.
+
+ @param[in] PluggedApicIdsThe APIC IDs of the CPUs that have been
+ hot-plugged.
+
+ @param[in] PluggedCount The number of filled-in APIC IDs in
+ PluggedApicIds.
+
+ @retval EFI_SUCCESS CPUs corresponding to all the APIC IDs are
+ populated.
+
+ @retval EFI_OUT_OF_RESOURCES Out of APIC ID space in "mCpuHotPlugData".
+
+ @return Error codes propagated from SmbaseRelocate()
+ and mMmCpuService->AddProcessor().
+
+**/
+STATIC
+EFI_STATUS
+ProcessHotAddedCpus (
+ IN APIC_ID *PluggedApicIds,
+ IN UINT32 PluggedCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 PluggedIdx;
+ UINT32 NewSlot;
+
+ //
+ // The Post-SMM Pen need not be reinstalled multiple times within a single
+ // root MMI handling. Even reinstalling once per root MMI is only prudence;
+ // in theory installing the pen in the driver's entry point function should
+ // suffice.
+ //
+ SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+
+ PluggedIdx = 0;
+ NewSlot = 0;
+ while (PluggedIdx < PluggedCount) {
+APIC_ID NewApicId;
+UINT32 CheckSlot;
+UINTN NewProcessorNumberByProtocol;
+
+NewApicId = PluggedApicIds[PluggedIdx];
+
+//
+// Check if the supposedly hot-added CPU is already known to us.
+//
+for (CheckSlot = 0;
+ CheckSlot < mCpuHotPlugData->ArrayLength;
+ CheckSlot++) {
+ if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
+break;
+ }
+}
+if (CheckSlot < mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
+"before; ignoring it\n", __FUNCTION__, NewApicId));
+ PluggedIdx++;
+ continue;
+}
+
+//
+// Find the first empty slot in CPU_HOT_PLUG_DATA.
+//
+while (NewSlot < mCpuHotPlugData->ArrayLength &&
+ mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
+ NewSlot++;
+}
+if (NewSlot == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
+__FUNCTION__, NewApicId));
+ return EFI_OUT_OF_RESOURCES;
+}
+
+//
+// Store the APIC ID of the new processor to the slot.
+//
+mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
+
+//
+// Relocate the SMBASE of the new CPU.
+//
+Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
+ mPostSmmPenAddress);
+if (EFI_ERROR (Status)) {
+ goto RevokeNewSlot;
+}
+
+//
+// Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
+//
+Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
+ );
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, NewApicId, Status));
+ goto RevokeNewSlot;
+}
+
+DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
+ "EFI_SMM_CPU_SERVICE_PROTOCOL assigned number %Lu\n", __FUNCTION__,
+ NewApicId, (UINT64)mCpuHotPlugData->SmBase[NewSlot],
+ (U
Re: [edk2-devel] [PATCH v5 1/9] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
On 2021-01-26 1:32 p.m., Laszlo Ersek wrote: On 01/26/21 22:17, Ankur Arora wrote: On 2021-01-26 1:07 p.m., Laszlo Ersek wrote: On 01/26/21 20:15, Ankur Arora wrote: On 2021-01-26 11:01 a.m., Laszlo Ersek wrote: I'll continue the review later this week. Acking the comments above. Thank you! Meanwhile let me reprocess the series in light of the comments above. I'm at such a point now, during the v5 review, that I think I can easily re-sync. In general, I don't mind the posting of a new version of a series mid-review, *IF* we agree about it in advance. If you prefer to post a v6, for addressing the comments I've made thus far, I'm OK with that. If you'd like me to continue reviewing v5, I'm also OK with that. So it's up to you -- please state your decision, so that I know if I should proceed with v5 (later this week), or wait for v6. I think I would prefer to send v6. Looking at the v5 comments so far, I'm sure that there's a lot of non conforming coding style issues. Addressing them now (or at least a hopefully significant subset) would probably save time. I agree; thank you. (And, for all the yelling that ECC does, I'm really surprised it didn't catch the "missing space between function designator and opening paren" wart!) Heh. Yeah, I was surprised at how little fault ECC found in the code. Thanks for reviewing. Ankur I'm looking at sending these out by Thursday morning PT, and given that you plan to continue later this week, sounds like it might not lose too much review time either. Yes, that should work fine. Thank you, Ankur! Laszlo -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#70814): https://edk2.groups.io/g/devel/message/70814 Mute This Topic: https://groups.io/mt/80125308/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v5 1/9] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
On 2021-01-26 1:07 p.m., Laszlo Ersek wrote: On 01/26/21 20:15, Ankur Arora wrote: On 2021-01-26 11:01 a.m., Laszlo Ersek wrote: I'll continue the review later this week. Acking the comments above. Thank you! Meanwhile let me reprocess the series in light of the comments above. I'm at such a point now, during the v5 review, that I think I can easily re-sync. In general, I don't mind the posting of a new version of a series mid-review, *IF* we agree about it in advance. If you prefer to post a v6, for addressing the comments I've made thus far, I'm OK with that. If you'd like me to continue reviewing v5, I'm also OK with that. So it's up to you -- please state your decision, so that I know if I should proceed with v5 (later this week), or wait for v6. I think I would prefer to send v6. Looking at the v5 comments so far, I'm sure that there's a lot of non conforming coding style issues. Addressing them now (or at least a hopefully significant subset) would probably save time. I'm looking at sending these out by Thursday morning PT, and given that you plan to continue later this week, sounds like it might not lose too much review time either. Thanks Ankur Thanks! Laszlo -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#70812): https://edk2.groups.io/g/devel/message/70812 Mute This Topic: https://groups.io/mt/80125308/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v5 0/9] support CPU hot-unplug
On 2021-01-26 10:03 a.m., Laszlo Ersek wrote:
Hi Ankur,
On 01/26/21 07:44, Ankur Arora wrote:
Hi,
This series adds support for CPU hot-unplug with OVMF.
Please see this in conjunction with the QEMU secureboot hot-unplug v2
series posted here (now upstreamed):
https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/
Patches 1 and 3,
("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
are either refactors or add support functions.
OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
OvmfPkg/CpuHotplugSmm: add CpuEject()
OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
Patch 2 and 9,
("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
handle the QEMU protocol logic for collection of CPU hot-unplug events
or the protocol negotiation.
Patch 4,
("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
adds the MMI logic for CPU hot-unplug handling and informing
the PiSmmCpuDxeSmm of CPU removal.
Patches 5 and 6,
("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
sets up state for doing the CPU ejection as part of hot-unplug.
Patches 7, and 8,
("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
add the CPU ejection logic.
Testing (with QEMU 5.2.50):
- Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
- Synthetic tests with simultaneous multi CPU hot-unplug
- Negotiation with/without CPU hotplug enabled
Also at:
github.com/terminus/edk2/ hot-unplug-v5
Changelog:
v5:
- fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add
add Qemu Cpu Status helper").
v4:
- Gets rid of unnecessary UefiCpuPkg changes
URL:
https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.ar...@oracle.com/
v3:
- Use a saner PCD based interface to share state between PiSmmCpuDxeSmm
and OvmfPkg/CpuHotplugSmm
- Cleaner split of the hot-unplug code
URL:
https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/
v2:
- Do the ejection via SmmCpuFeaturesRendezvousExit()
URL:
https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/
RFC:
URL:
https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/
Please review.
Thanks
Ankur
Ankur Arora (9):
OvmfPkg/CpuHotplugSmm: refactor hotplug logic
OvmfPkg/CpuHotplugSmm: collect hot-unplug events
OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
OvmfPkg/CpuHotplugSmm: add CpuEject()
OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
OvmfPkg/OvmfPkg.dec| 10 +
OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf| 1 +
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 3 +
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 +
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 +
OvmfPkg/Include/Library/CpuHotEjectData.h | 32 ++
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 450 -
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 57 ++-
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 68
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 25 +-
10 files changed, 552 insertions(+), 102 deletions(-)
create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
This series is still mal-formatted; for some reason it has one too many
CRs per line.
Did you try to set up the "base64" or the "8bit"
content-transfer-encoding? The emails reflected through the list are
still "quoted-printable".
Confirming that this was the problem. And, thanks for debugging this.
Ankur
Anyway, I managed to apply the patches with some trickery on a local branch.
Thanks
Laszlo
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70808): https://edk2.groups.io/g/devel/message/70808
Mute This Topic: https://groups.io/mt/80125307/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v5 1/9] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
On 2021-01-26 11:01 a.m., Laszlo Ersek wrote:
On 01/26/21 07:44, Ankur Arora wrote:
Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
PlugCpus(). This is in preparation for supporting CPU hot-unplug.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 208 ++---
1 file changed, 123 insertions(+), 85 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..a5052a501e5a 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,124 @@ STATIC UINT32 mPostSmmPenAddress;
//
STATIC EFI_HANDLE mDispatchHandle;
+/**
+ CPU Hotplug handler function.
+
+ @param[in] PluggedApicIds List of APIC IDs to be plugged.
+
+ @param[in] PluggedCountCount of APIC IDs to be plugged.
(1) These comments are not optimal.
The variable names say "Plugged", meaning that the CPUs have already
been plugged, from the QEMU perspective. The purpose of this function is
to go over those CPUs whose APIC IDs have been collected with events
pending, relocate the SMBASE on each, and then expose each such CPU to
EFI_SMM_CPU_SERVICE_PROTOCOL. For a given CPU, I think the comment on
the EFI_SMM_ADD_PROCESSOR prototype
[UefiCpuPkg/Include/Protocol/SmmCpuService.h] best expresses the goal:
"Notify that a new processor has been added to the system ... The SMM
CPU driver should add the processor to the SMM CPU list".
See also the comment on QemuCpuhpCollectApicIds():
"""
Collect the APIC IDs of
- the CPUs that have been hot-plugged,
- the CPUs that are about to be hot-unplugged.
"""
This closely reflects which agent (firmware vs. QEMU) is driving each
particular operation / direction.
(1a) So please replace the leading comment with:
Process those CPUs that have been hot-added, according to
QemuCpuhpCollectApicIds().
For each such CPU, relocate the SMBASE, and report the CPU to PiSmmCpuDxeSmm
via EFI_SMM_CPU_SERVICE_PROTOCOL. If a supposedly hot-added CPU is already
known, skip it silently.
(1b) Similarly, in the parameter comments, "to be plugged" is wrong. I
suggest copying the parameter descriptions from
QemuCpuhpCollectApicIds():
@param[out] PluggedApicIds The APIC IDs of the CPUs that have been
hot-plugged.
@param[out] PluggedCount The number of filled-in APIC IDs in
PluggedApicIds.
+
+ @retval EFI_SUCCESSSome of the requested APIC IDs were
hot-plugged.
(2) This is inexact; on successful return, each one of the collected
CPUs has been relocated and exposed to the SMM CPU driver. (Either in
this particular invocation, or in an earlier invocation, but on success,
there is no entry in PluggedApicIds that is *not known* to the SMM CPU
driver, or whose SMBASE is not relocated.)
+
+ @retval EFI_INTERRUPT_PENDING Fatal error while hot-plugging.
(3) This error code is very uncommon, and it is mostly/only required
from the function -- CpuHotplugMmi() -- that is registered with
gMmst->MmiHandlerRegister(). The meaning of the error code is, "The MMI
source could not be quiesced", which is a situation that can be
identified at the level of CpuHotplugMmi(), but not at the level of
PlugCpus().
Please list the following return values instead:
@retval EFI_OUT_OF_RESOURCES Out of APIC ID space in "mCpuHotPlugData".
@return Error codes propagated from SmbaseRelocate()
and mMmCpuService->AddProcessor().
(General remark, in addition: please note the difference between
"@retval" and "@return". The latter does not name a particular value;
the set of values is described in natural language instead.)
+
+**/
+STATIC
+EFI_STATUS
+EFIAPI
(4) There is no need to make this function EFIAPI.
+PlugCpus(
(5) Space character missing between function name and opening
parenthesis.
Please check every function prototype and function call for this -- one
space char before the opening paren is required, except in the
definition of function-like macros (where the language standard requires
the "(" to be directly adjacent).
(6) According to the discussion above, this function should be called
ProcessHotAddedCpus().
... The best hint for this function name is actually the comment that
sits atop the stretch of code you are extracting, namely:
//
// Process hot-added CPUs.
//
+ IN APIC_ID *PluggedApicIds,
+ IN UINT32 PluggedCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 PluggedIdx;
+ UINT32 NewSlot;
+
+ //
+ // Process hot-added C
[edk2-devel] [PATCH v5 9/9] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
As part of the negotiation treat ICH9_LPC_SMI_F_CPU_HOT_UNPLUG as a
subfeature of feature flag ICH9_LPC_SMI_F_CPU_HOTPLUG, so enable it
only if the other is also being negotiated.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 25 ++---
1 file changed, 22 insertions(+), 3 deletions(-)
diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
index c9d875543205..af01104d69c6 100644
--- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
+++ b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
@@ -29,6 +29,13 @@
//
#define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
+// The following bit value stands for "enable CPU hot unplug, and inject an SMI
+// with control value ICH9_APM_CNT_CPU_HOT_UNPLUG upon hot unplug", in the
+// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
+// Can only be negotiated alongside ICH9_LPC_SMI_F_CPU_HOTPLUG.
+//
+#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
+
//
// Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
// for the S3 boot script fragment to write to and read from.
@@ -112,7 +119,8 @@ NegotiateSmiFeatures (
QemuFwCfgReadBytes (sizeof mSmiFeatures, );
//
- // We want broadcast SMI, SMI on CPU hotplug, and nothing else.
+ // We want broadcast SMI, SMI on CPU hotplug, on CPU hot-unplug
+ // and nothing else.
//
RequestedFeaturesMask = ICH9_LPC_SMI_F_BROADCAST;
if (!MemEncryptSevIsEnabled ()) {
@@ -120,8 +128,18 @@ NegotiateSmiFeatures (
// For now, we only support hotplug with SEV disabled.
//
RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOTPLUG;
+RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
}
mSmiFeatures &= RequestedFeaturesMask;
+
+ if (!(mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) &&
+ (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG)) {
+DEBUG ((DEBUG_WARN, "%a CPU host-features %Lx, requested mask %Lx\n",
+ __FUNCTION__, mSmiFeatures, RequestedFeaturesMask));
+
+mSmiFeatures &= ~ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
+ }
+
QemuFwCfgSelectItem (mRequestedFeaturesItem);
QemuFwCfgWriteBytes (sizeof mSmiFeatures, );
@@ -162,8 +180,9 @@ NegotiateSmiFeatures (
if ((mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) == 0) {
DEBUG ((DEBUG_INFO, "%a: CPU hotplug not negotiated\n", __FUNCTION__));
} else {
-DEBUG ((DEBUG_INFO, "%a: CPU hotplug with SMI negotiated\n",
- __FUNCTION__));
+DEBUG ((DEBUG_INFO, "%a: CPU hotplug%s with SMI negotiated\n",
+ __FUNCTION__,
+ (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG) ? ", unplug" : ""));
}
//
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70768): https://edk2.groups.io/g/devel/message/70768
Mute This Topic: https://groups.io/mt/80125324/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v5 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()
Add CpuEject(), which handles the CPU ejection, and provides
a holding area for said CPUs. It is called via
SmmCpuFeaturesRendezvousExit(), at the tail end of the SMI
handling.
Also UnplugCpus() now stashes APIC IDs of CPUs which need to
be ejected in CPU_HOT_EJECT_DATA.ApicIdMap. These are used by
CpuEject() to identify such CPUs.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 94 +-
1 file changed, 93 insertions(+), 1 deletion(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index d165d6ccea0d..99e6845a12d9 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -14,6 +14,7 @@
#include // gMmst
#include // PcdGetBool()
#include // SafeUintnSub()
+#include // CPU_HOT_EJECT_DATA
#include // EFI_MM_CPU_IO_PROTOCOL
#include // EFI_SMM_CPU_SERVICE_PROTOCOL
#include// EFI_STATUS
@@ -32,11 +33,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo;
//
STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService;
//
-// This structure is a communication side-channel between the
+// These structures serve as communication side-channels between the
// EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider
// (i.e., PiSmmCpuDxeSmm).
//
STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData;
//
// SMRAM arrays for fetching the APIC IDs of processors with pending events (of
// known event types), for the time of just one MMI.
@@ -182,6 +184,43 @@ Fatal:
}
/**
+ CPU Hot-eject handler function.
+
+ If the executing CPU is not being ejected: NOP.
+ If the executing CPU is being ejected: wait in a CpuDeadLoop()
+ until ejected.
+
+ @param[in] ProcessorNum Index of executing CPU.
+**/
+VOID
+EFIAPI
+CpuEject(
+ IN UINTN ProcessorNum
+ )
+{
+ //
+ // APIC ID is UINT32, but mCpuHotEjectData->ApicIdMap[] is UINT64
+ // so use UINT64 throughout.
+ //
+ UINT64 ApicId;
+
+ ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
+ if (ApicId == CPU_EJECT_INVALID) {
+return;
+ }
+
+ //
+ // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
+ // after having been cleared to exit the SMI by the monarch and thus have
+ // no SMM processing remaining.
+ //
+ // Given that we cannot allow them to escape to the guest, we pen them
+ // here until the SMM monarch tells the HW to unplug them.
+ //
+ CpuDeadLoop ();
+}
+
+/**
CPU Hot-unplug MMI handler function.
@param[in] UnplugApicIds List of APIC IDs to be unplugged.
@@ -203,9 +242,11 @@ UnplugCpus(
{
EFI_STATUS Status;
UINT32 ToUnplugIdx;
+ UINT32 EjectCount;
UINTN ProcessorNum;
ToUnplugIdx = 0;
+ EjectCount = 0;
while (ToUnplugIdx < ToUnplugCount) {
APIC_IDRemoveApicId;
@@ -242,11 +283,35 @@ UnplugCpus(
DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
__FUNCTION__, RemoveApicId, Status));
goto Fatal;
+} else {
+ //
+ // Stash the APIC IDs so we can do the actual unplug later.
+ //
+ if (mCpuHotEjectData->ApicIdMap[ProcessorNum] != CPU_EJECT_INVALID) {
+//
+// Since ProcessorNum and APIC-ID map 1-1, so a valid
+// mCpuHotEjectData->ApicIdMap[ProcessorNum] means something
+// is horribly wrong.
+//
+DEBUG ((DEBUG_ERROR, "%a: ProcessorNum %u maps to %llx, cannot map to
" FMT_APIC_ID "\n",
+ __FUNCTION__, ProcessorNum,
mCpuHotEjectData->ApicIdMap[ProcessorNum], RemoveApicId));
+goto Fatal;
+ }
+
+ mCpuHotEjectData->ApicIdMap[ProcessorNum] = (UINT64)RemoveApicId;
+ EjectCount++;
}
ToUnplugIdx++;
}
+ if (EjectCount != 0) {
+//
+// We have processors to be ejected; install the handler.
+//
+mCpuHotEjectData->Handler = CpuEject;
+ }
+
//
// We've handled this unplug.
//
@@ -445,7 +510,13 @@ CpuHotplugEntry (
// Our DEPEX on EFI_SMM_CPU_SERVICE_PROTOCOL guarantees that PiSmmCpuDxeSmm
// has pointed PcdCpuHotPlugDataAddress to CPU_HOT_PLUG_DATA in SMRAM.
//
+ // Additionally, CPU Hot-unplug is available only if CPU Hotplug is, so
+ // the same DEPEX also guarantees that PcdCpuHotEjectDataAddress points
+ // to CPU_HOT_EJECT_DATA in SMRAM.
+ //
mCpuHotPlugData = (VOID *)(UINTN)PcdGet64 (PcdCpuHotPlugDataAddress);
+ mCpuHotEjectData = (VOID *)(UINTN)PcdGet64 (PcdCpuHotEjectDataAddress);
+
if (mCpuHotPlugData == NULL) {
Status = EFI_NOT_FOUND;
DEBUG ((
[edk2-devel] [PATCH v5 8/9] OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
Designate a worker CPU (we use the one executing the root MMI
handler), which will do the actual ejection via QEMU in CpuEject().
CpuEject(), on the worker CPU, ejects each marked CPU by first
selecting its APIC ID and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
it out of the SMI.
CpuEject(), on the CPU being ejected, spins around in its holding
area until this final context-switch. This does mean that there is
some CPU state that would ordinarily be restored (in SmiRendezvous()
and in SmiEntry.nasm::CommonHandler), but will not be anymore.
This unrestored state includes FPU state, CET enable, stuffing of
RSB and the final RSM. Since the CPU state is destroyed by QEMU,
this should be okay.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 73 ++
1 file changed, 67 insertions(+), 6 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 99e6845a12d9..d4e27d641dc5 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -186,10 +186,13 @@ Fatal:
/**
CPU Hot-eject handler function.
- If the executing CPU is not being ejected: NOP.
+ If the executing CPU is neither a worker, nor being ejected: NOP.
If the executing CPU is being ejected: wait in a CpuDeadLoop()
until ejected.
+ If the executing CPU is a worker CPU: set QEMU Cpu status to eject
+ for CPUs being ejected.
+
@param[in] ProcessorNum Index of executing CPU.
**/
VOID
@@ -209,6 +212,56 @@ CpuEject(
return;
}
+ if (ApicId == CPU_EJECT_WORKER) {
+UINT32 CpuIndex;
+
+for (CpuIndex = 0; CpuIndex < mCpuHotEjectData->ArrayLength; CpuIndex++) {
+ UINT64 RemoveApicId;
+
+ RemoveApicId = mCpuHotEjectData->ApicIdMap[CpuIndex];
+
+ if ((RemoveApicId != CPU_EJECT_INVALID &&
+ RemoveApicId != CPU_EJECT_WORKER)) {
+//
+// This to-be-ejected-CPU has already received the BSP's SMI exit
+// signal and, will execute SmmCpuFeaturesSmiRendezvousExit()
+// followed by this callback or is already waiting in the
+// CpuDeadLoop() below.
+//
+// Tell QEMU to context-switch it out.
+//
+QemuCpuhpWriteCpuSelector (mMmCpuIo, (APIC_ID) RemoveApicId);
+QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
+
+//
+// Compiler barrier to ensure the next store isn't reordered
+//
+MemoryFence();
+
+//
+// Clear the eject status for CpuIndex to ensure that an invalid
+// SMI later does not end up trying to eject it or a newly
+// hotplugged CpuIndex does not go into the dead loop.
+//
+mCpuHotEjectData->ApicIdMap[CpuIndex] = CPU_EJECT_INVALID;
+
+DEBUG ((DEBUG_INFO, "%a: Unplugged CPU %u -> " FMT_APIC_ID "\n",
+ __FUNCTION__, CpuIndex, RemoveApicId));
+ }
+}
+
+//
+// Clear our own worker status.
+//
+mCpuHotEjectData->ApicIdMap[ProcessorNum] = CPU_EJECT_INVALID;
+
+//
+// We are done until the next hot-unplug; clear the handler.
+//
+mCpuHotEjectData->Handler = NULL;
+return;
+ }
+
//
// CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
// after having been cleared to exit the SMI by the monarch and thus have
@@ -306,6 +359,19 @@ UnplugCpus(
}
if (EjectCount != 0) {
+UINTN Worker;
+
+Status = mMmCpuService->WhoAmI(mMmCpuService, );
+ASSERT_EFI_ERROR(Status);
+//
+// UnplugCpus() is called via the root MMI handler and thus we are
+// executing in the BSP context.
+//
+// Mark ourselves as the worker CPU.
+//
+ASSERT (mCpuHotEjectData->ApicIdMap[Worker] == CPU_EJECT_INVALID);
+mCpuHotEjectData->ApicIdMap[Worker] = CPU_EJECT_WORKER;
+
//
// We have processors to be ejected; install the handler.
//
@@ -429,11 +495,6 @@ CpuHotplugMmi (
if (EFI_ERROR (Status)) {
goto Fatal;
}
- if (ToUnplugCount > 0) {
-DEBUG ((DEBUG_ERROR, "%a: hot-unplug is not supported yet\n",
- __FUNCTION__));
-goto Fatal;
- }
if (PluggedCount > 0) {
Status = PlugCpus(mPluggedApicIds, PluggedCount);
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70767): https://edk2.groups.io/g/devel/message/70767
Mute This Topic: https://groups.io/mt/80125315/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v5 6/9] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection state
between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
CpuHotplugSmm also sets up the CPU ejection mechanism via
CPU_HOT_EJECT_DATA->Handler.
Additionally, expose CPU_HOT_EJECT_DATA via PcdCpuHotEjectDataAddress.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 3 +
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 68 ++
2 files changed, 71 insertions(+)
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..32c63722ee62 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -35,4 +35,7 @@ [LibraryClasses]
UefiBootServicesTableLib
[Pcd]
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugSupport
+ gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..5c9cdc6710e4 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -14,7 +14,9 @@
#include
#include
#include
+#include // AllocatePool()
#include
+#include
#include
#include
#include
@@ -171,6 +173,60 @@ SmmCpuFeaturesHookReturnFromSmm (
return OriginalInstructionPointer;
}
+GLOBAL_REMOVE_IF_UNREFERENCED
+CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+/**
+ This function initializes CpuHotEjectData if PcdCpuHotPlugSupport is
+ enabled and if more than 1 CPU is configured.
+
+ Also sets up the corresponding PcdCpuHotEjectDataAddress.
+**/
+STATIC
+VOID
+EFIAPI
+SmmCpuFeaturesSmmInitHotEject(
+ VOID
+ )
+{
+ UINT32 mMaxNumberOfCpus;
+ EFI_STATUS Status;
+
+ if (!FeaturePcdGet (PcdCpuHotPlugSupport)) {
+return;
+ }
+
+ // PcdCpuHotPlugSupport => PcdCpuMaxLogicalProcessorNumber
+ mMaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+
+ // No spare CPUs to eject
+ if (mMaxNumberOfCpus == 1) {
+return;
+ }
+
+ mCpuHotEjectData =
+(CPU_HOT_EJECT_DATA *)AllocatePool (sizeof (*mCpuHotEjectData));
+ ASSERT (mCpuHotEjectData != NULL);
+
+ //
+ // Allocate buffer for pointers to array in CPU_HOT_EJECT_DATA.
+ //
+ mCpuHotEjectData->Revision = CPU_HOT_EJECT_DATA_REVISION_1; // Revision
+ mCpuHotEjectData->ArrayLength = mMaxNumberOfCpus; // Array
Length of APIC ID
+ mCpuHotEjectData->ApicIdMap = // CpuIndex ->
APIC ID map
+(UINT64 *)AllocatePool (sizeof (UINT64) * mCpuHotEjectData->ArrayLength);
+ mCpuHotEjectData->Handler = NULL; // Hot Eject
handler
+ mCpuHotEjectData->Handler = 0;// Reserved
+
+ ASSERT (mCpuHotEjectData->ApicIdMap != NULL);
+
+ //
+ // Expose address of CPU Hot eject Data structure
+ //
+ Status = PcdSet64S (PcdCpuHotEjectDataAddress, (UINT64)(VOID
*)mCpuHotEjectData);
+ ASSERT_EFI_ERROR (Status);
+}
+
/**
Hook point in normal execution mode that allows the one CPU that was elected
as monarch during System Management Mode initialization to perform additional
@@ -188,6 +244,9 @@ SmmCpuFeaturesSmmRelocationComplete (
UINTN MapPagesBase;
UINTN MapPagesCount;
+
+ SmmCpuFeaturesSmmInitHotEject();
+
if (!MemEncryptSevIsEnabled ()) {
return;
}
@@ -375,6 +434,15 @@ SmmCpuFeaturesRendezvousExit (
IN UINTN CpuIndex
)
{
+ //
+ // CPU Hot eject not enabled.
+ //
+ if (mCpuHotEjectData == NULL ||
+ mCpuHotEjectData->Handler == NULL) {
+return;
+ }
+
+ mCpuHotEjectData->Handler(CpuIndex);
}
/**
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70765): https://edk2.groups.io/g/devel/message/70765
Mute This Topic: https://groups.io/mt/80125313/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v5 5/9] OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/OvmfPkg.dec | 10 ++
OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf | 1 +
OvmfPkg/Include/Library/CpuHotEjectData.h | 32 +++
3 files changed, 43 insertions(+)
create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 4348bb45c64a..e79ff28465e3 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -106,6 +106,10 @@ [LibraryClasses]
#
XenPlatformLib|Include/Library/XenPlatformLib.h
+ ## @libraryclass Share CPU hot-eject state
+ #
+ CpuHotEjectData|Include/Library/CpuHotEjectData.h
+
[Guids]
gUefiOvmfPkgTokenSpaceGuid= {0x93bb96af, 0xb9f2, 0x4eb8, {0x94,
0x62, 0xe0, 0xba, 0x74, 0x56, 0x42, 0x36}}
gEfiXenInfoGuid = {0xd3b46f3b, 0xd441, 0x1244, {0x9a,
0x12, 0x0, 0x12, 0x27, 0x3f, 0xc1, 0x4d}}
@@ -352,6 +356,12 @@ [PcdsDynamic, PcdsDynamicEx]
# This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+ ## This PCD adds a communication channel between PiSmmCpuDxe and
+ # CpuHotplugSmm.
+ #
+ # Only accessed if PcdCpuHotPlugSupport is TRUE
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
+
[PcdsFeatureFlag]
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
index 04322b0d7855..e08b572ef169 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
@@ -54,6 +54,7 @@ [Protocols]
[Pcd]
gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress## CONSUMES
+ gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress ## CONSUMES
gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase ## CONSUMES
[FeaturePcd]
diff --git a/OvmfPkg/Include/Library/CpuHotEjectData.h
b/OvmfPkg/Include/Library/CpuHotEjectData.h
new file mode 100644
index ..839ddad6d8f8
--- /dev/null
+++ b/OvmfPkg/Include/Library/CpuHotEjectData.h
@@ -0,0 +1,32 @@
+/** @file
+ Definition for a structure sharing state for CPU hot-eject.
+
+ Copyright (C) 2021, Oracle Corporation.
+
+ SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#ifndef _CPU_HOT_EJECT_DATA_H_
+#define _CPU_HOT_EJECT_DATA_H_
+
+typedef
+VOID
+(EFIAPI *CPU_HOT_EJECT_FN)(
+ IN UINTN ProcessorNum
+ );
+
+#define CPU_EJECT_INVALID (MAX_UINT64)
+#define CPU_EJECT_WORKER(MAX_UINT64-1)
+
+#define CPU_HOT_EJECT_DATA_REVISION_1 0x0001
+
+typedef struct {
+ UINT32 Revision; // Used for version identification for
this structure
+ UINT32 ArrayLength; // Number of entries for the ApicIdMap
array
+
+ UINT64 *ApicIdMap;// Pointer to CpuIndex->ApicId map for
pending ejects
+ CPU_HOT_EJECT_FN Handler; // Handler to do the CPU ejection
+ UINT64 Reserved;
+} CPU_HOT_EJECT_DATA;
+
+#endif /* _CPU_HOT_EJECT_DATA_H_ */
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70764): https://edk2.groups.io/g/devel/message/70764
Mute This Topic: https://groups.io/mt/80125312/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v5 3/9] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 22 ++
2 files changed, 28 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..804809846890 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -30,6 +30,12 @@ QemuCpuhpReadCpuStatus (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
);
+VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ );
+
UINT32
QemuCpuhpReadCommandData (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 2dd783ebf42e..c35ae31d2db0 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -67,6 +67,28 @@ QemuCpuhpReadCpuStatus (
return CpuStatus;
}
+VOID
+QemuCpuhpWriteCpuStatus (
+ IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+ IN UINT8CpuStatus
+ )
+{
+ EFI_STATUS Status;
+
+ Status = MmCpuIo->Io.Write(
+ MmCpuIo,
+ MM_IO_UINT8,
+ ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+ 1,
+
+ );
+ if (EFI_ERROR (Status)) {
+DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+ASSERT (FALSE);
+CpuDeadLoop ();
+ }
+}
+
UINT32
QemuCpuhpReadCommandData (
IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70762): https://edk2.groups.io/g/devel/message/70762
Mute This Topic: https://groups.io/mt/80125310/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v5 4/9] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().
With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 77 ++
1 file changed, 77 insertions(+)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index a5052a501e5a..d165d6ccea0d 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -182,6 +182,81 @@ Fatal:
}
/**
+ CPU Hot-unplug MMI handler function.
+
+ @param[in] UnplugApicIds List of APIC IDs to be unplugged.
+
+ @param[in] ToUnplugCount Count of APIC IDs to be unplugged.
+
+ @retval EFI_SUCCESSSome of the requested APIC IDs will be
hot-unplugged.
+
+ @retval EFI_INTERRUPT_PENDING Fatal error while hot-plugging.
+
+**/
+STATIC
+EFI_STATUS
+EFIAPI
+UnplugCpus(
+ IN APIC_ID *UnplugApicIds,
+ IN UINT32 ToUnplugCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 ToUnplugIdx;
+ UINTN ProcessorNum;
+
+ ToUnplugIdx = 0;
+ while (ToUnplugIdx < ToUnplugCount) {
+APIC_IDRemoveApicId;
+
+RemoveApicId = UnplugApicIds[ToUnplugIdx];
+
+//
+// mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+// the ProcessorNum for the APIC ID to be removed.
+//
+for (ProcessorNum = 0;
+ ProcessorNum < mCpuHotPlugData->ArrayLength;
+ ProcessorNum++) {
+ if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+break;
+ }
+}
+
+//
+// Ignore the unplug if APIC ID not found
+//
+if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_INFO, "%a: did not find APIC ID " FMT_APIC_ID " to
unplug\n",
+__FUNCTION__, RemoveApicId));
+ ToUnplugIdx++;
+ continue;
+}
+
+//
+// Mark ProcessorNum for removal from SMM data structures
+//
+Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+
+if (EFI_ERROR(Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, RemoveApicId, Status));
+ goto Fatal;
+}
+
+ToUnplugIdx++;
+ }
+
+ //
+ // We've handled this unplug.
+ //
+ return EFI_SUCCESS;
+
+Fatal:
+ return EFI_INTERRUPT_PENDING;
+}
+
+/**
CPU Hotplug MMI handler function.
This is a root MMI handler.
@@ -297,6 +372,8 @@ CpuHotplugMmi (
if (PluggedCount > 0) {
Status = PlugCpus(mPluggedApicIds, PluggedCount);
+ } else if (ToUnplugCount > 0) {
+Status = UnplugCpus(mToUnplugApicIds, ToUnplugCount);
}
if (EFI_ERROR(Status)) {
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70763): https://edk2.groups.io/g/devel/message/70763
Mute This Topic: https://groups.io/mt/80125311/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v5 1/9] OvmfPkg/CpuHotplugSmm: refactor hotplug logic
Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
PlugCpus(). This is in preparation for supporting CPU hot-unplug.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 208 ++---
1 file changed, 123 insertions(+), 85 deletions(-)
diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..a5052a501e5a 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,124 @@ STATIC UINT32 mPostSmmPenAddress;
//
STATIC EFI_HANDLE mDispatchHandle;
+/**
+ CPU Hotplug handler function.
+
+ @param[in] PluggedApicIds List of APIC IDs to be plugged.
+
+ @param[in] PluggedCountCount of APIC IDs to be plugged.
+
+ @retval EFI_SUCCESSSome of the requested APIC IDs were
hot-plugged.
+
+ @retval EFI_INTERRUPT_PENDING Fatal error while hot-plugging.
+
+**/
+STATIC
+EFI_STATUS
+EFIAPI
+PlugCpus(
+ IN APIC_ID *PluggedApicIds,
+ IN UINT32 PluggedCount
+ )
+{
+ EFI_STATUS Status;
+ UINT32 PluggedIdx;
+ UINT32 NewSlot;
+
+ //
+ // Process hot-added CPUs.
+ //
+ // The Post-SMM Pen need not be reinstalled multiple times within a single
+ // root MMI handling. Even reinstalling once per root MMI is only prudence;
+ // in theory installing the pen in the driver's entry point function should
+ // suffice.
+ //
+ SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+
+ PluggedIdx = 0;
+ NewSlot = 0;
+ while (PluggedIdx < PluggedCount) {
+APIC_ID NewApicId;
+UINT32 CheckSlot;
+UINTN NewProcessorNumberByProtocol;
+
+NewApicId = PluggedApicIds[PluggedIdx];
+
+//
+// Check if the supposedly hot-added CPU is already known to us.
+//
+for (CheckSlot = 0;
+ CheckSlot < mCpuHotPlugData->ArrayLength;
+ CheckSlot++) {
+ if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
+break;
+ }
+}
+if (CheckSlot < mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
+"before; ignoring it\n", __FUNCTION__, NewApicId));
+ PluggedIdx++;
+ continue;
+}
+
+//
+// Find the first empty slot in CPU_HOT_PLUG_DATA.
+//
+while (NewSlot < mCpuHotPlugData->ArrayLength &&
+ mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
+ NewSlot++;
+}
+if (NewSlot == mCpuHotPlugData->ArrayLength) {
+ DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
+__FUNCTION__, NewApicId));
+ goto Fatal;
+}
+
+//
+// Store the APIC ID of the new processor to the slot.
+//
+mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
+
+//
+// Relocate the SMBASE of the new CPU.
+//
+Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
+ mPostSmmPenAddress);
+if (EFI_ERROR (Status)) {
+ goto RevokeNewSlot;
+}
+
+//
+// Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
+//
+Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
+ );
+if (EFI_ERROR (Status)) {
+ DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
+__FUNCTION__, NewApicId, Status));
+ goto RevokeNewSlot;
+}
+
+DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
+ "EFI_SMM_CPU_SERVICE_PROTOCOL assigned number %Lu\n", __FUNCTION__,
+ NewApicId, (UINT64)mCpuHotPlugData->SmBase[NewSlot],
+ (UINT64)NewProcessorNumberByProtocol));
+
+NewSlot++;
+PluggedIdx++;
+ }
+
+ //
+ // We've handled this hotplug.
+ //
+ return EFI_SUCCESS;
+
+RevokeNewSlot:
+ mCpuHotPlugData->ApicId[NewSlot] = MAX_UINT64;
+
+Fatal:
+ return EFI_INTERRUPT_PENDING;
+}
/**
CPU Hotplug MMI handler function.
@@ -122,8 +240,6 @@ CpuHotplugMmi (
UINT8 ApmControl;
UINT32 PluggedCount;
UINT32 ToUnplugCount;
- UINT32 PluggedIdx;
- UINT32 NewSlot;
//
// Assert that we are entering this function due to our root MMI handler
@@ -179,87 +295,12 @@ CpuHotplugMmi (
goto Fatal;
}
- //
- // Process hot-added CPUs.
- //
- // The Post-SMM Pen need not be reinstalled multiple times within a single
- // root MMI handling. Even reinstalling once per root MMI is only prudence;
- // in theory installing the pen in the driver's entry point function should
- // suffice.
- //
- Smbase
[edk2-devel] [PATCH v5 2/9] OvmfPkg/CpuHotplugSmm: collect hot-unplug events
Process fw_remove events in QemuCpuhpCollectApicIds() and collect
corresponding APIC IDs for CPUs that are being hot-unplugged.
In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.
This is based on the QEMU hot-unplug protocol documented here:
https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imamm...@redhat.com/
Also define QEMU_CPUHP_STAT_EJECTED while we are at it.
Cc: Laszlo Ersek
Cc: Jordan Justen
Cc: Ard Biesheuvel
Cc: Igor Mammedov
Cc: Boris Ostrovsky
Cc: Aaron Young
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora
---
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 ++
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 35 +--
2 files changed, 29 insertions(+), 8 deletions(-)
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..692e3072598c 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,8 @@
#define QEMU_CPUHP_STAT_ENABLEDBIT0
#define QEMU_CPUHP_STAT_INSERT BIT1
#define QEMU_CPUHP_STAT_REMOVE BIT2
+#define QEMU_CPUHP_STAT_EJECTEDBIT3
+#define QEMU_CPUHP_STAT_FW_REMOVE BIT4
#define QEMU_CPUHP_RW_CMD_DATA 0x8
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 8d4a6693c8d6..2dd783ebf42e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -205,7 +205,7 @@ QemuCpuhpCollectApicIds (
UINT8 CpuStatus;
APIC_ID *ExtendIds;
UINT32 *ExtendCount;
-APIC_ID NewApicId;
+APIC_ID OpApicId;
//
// Write CurrentSelector (which is valid) to the CPU selector register.
@@ -245,10 +245,10 @@ QemuCpuhpCollectApicIds (
if ((CpuStatus & QEMU_CPUHP_STAT_INSERT) != 0) {
//
// The "insert" event guarantees the "enabled" status; plus it excludes
- // the "remove" event.
+ // the "fw_remove" event.
//
if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0 ||
- (CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+ (CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
"inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
CpuStatus));
@@ -260,12 +260,31 @@ QemuCpuhpCollectApicIds (
ExtendIds = PluggedApicIds;
ExtendCount = PluggedCount;
-} else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
- DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: remove\n", __FUNCTION__,
+} else if ((CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
+ //
+ // "fw_remove" event guarantees "enabled".
+ //
+ if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0) {
+DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
+ "inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
+ CpuStatus));
+return EFI_PROTOCOL_ERROR;
+ }
+
+ DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: fw_remove\n",
__FUNCTION__,
CurrentSelector));
ExtendIds = ToUnplugApicIds;
ExtendCount = ToUnplugCount;
+} else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+ //
+ // Let the OSPM deal with the "remove" event.
+ //
+ DEBUG ((DEBUG_INFO, "%a: CurrentSelector=%u: remove (ignored)\n",
__FUNCTION__,
+CurrentSelector));
+
+ CurrentSelector++;
+ continue;
} else {
DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: no event\n",
__FUNCTION__, CurrentSelector));
@@ -281,10 +300,10 @@ QemuCpuhpCollectApicIds (
return EFI_BUFFER_TOO_SMALL;
}
QemuCpuhpWriteCommand (MmCpuIo, QEMU_CPUHP_CMD_GET_ARCH_ID);
-NewApicId = QemuCpuhpReadCommandData (MmCpuIo);
+OpApicId = QemuCpuhpReadCommandData (MmCpuIo);
DEBUG ((DEBUG_VERBOSE, "%a: ApicId=" FMT_APIC_ID "\n", __FUNCTION__,
- NewApicId));
-ExtendIds[(*ExtendCount)++] = NewApicId;
+ OpApicId, ExtendCount));
+ExtendIds[(*ExtendCount)++] = OpApicId;
//
// We've processed the CPU with (known) pending events, but we must never
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70761): https://edk2.groups.io/g/devel/message/70761
Mute This Topic: https://groups.io/mt/80125309/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[edk2-devel] [PATCH v5 0/9] support CPU hot-unplug
Hi,
This series adds support for CPU hot-unplug with OVMF.
Please see this in conjunction with the QEMU secureboot hot-unplug v2
series posted here (now upstreamed):
https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/
Patches 1 and 3,
("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
are either refactors or add support functions.
OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
OvmfPkg/CpuHotplugSmm: add CpuEject()
OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
Patch 2 and 9,
("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
handle the QEMU protocol logic for collection of CPU hot-unplug events
or the protocol negotiation.
Patch 4,
("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
adds the MMI logic for CPU hot-unplug handling and informing
the PiSmmCpuDxeSmm of CPU removal.
Patches 5 and 6,
("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
sets up state for doing the CPU ejection as part of hot-unplug.
Patches 7, and 8,
("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
add the CPU ejection logic.
Testing (with QEMU 5.2.50):
- Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
- Synthetic tests with simultaneous multi CPU hot-unplug
- Negotiation with/without CPU hotplug enabled
Also at:
github.com/terminus/edk2/ hot-unplug-v5
Changelog:
v5:
- fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add
add Qemu Cpu Status helper").
v4:
- Gets rid of unnecessary UefiCpuPkg changes
URL:
https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.ar...@oracle.com/
v3:
- Use a saner PCD based interface to share state between PiSmmCpuDxeSmm
and OvmfPkg/CpuHotplugSmm
- Cleaner split of the hot-unplug code
URL:
https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/
v2:
- Do the ejection via SmmCpuFeaturesRendezvousExit()
URL:
https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/
RFC:
URL:
https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/
Please review.
Thanks
Ankur
Ankur Arora (9):
OvmfPkg/CpuHotplugSmm: refactor hotplug logic
OvmfPkg/CpuHotplugSmm: collect hot-unplug events
OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
OvmfPkg/CpuHotplugSmm: add CpuEject()
OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
OvmfPkg/OvmfPkg.dec| 10 +
OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf| 1 +
.../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf| 3 +
OvmfPkg/CpuHotplugSmm/QemuCpuhp.h | 6 +
OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 2 +
OvmfPkg/Include/Library/CpuHotEjectData.h | 32 ++
OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 450 -
OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 57 ++-
.../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 68
OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 25 +-
10 files changed, 552 insertions(+), 102 deletions(-)
create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
--
2.9.3
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70759): https://edk2.groups.io/g/devel/message/70759
Mute This Topic: https://groups.io/mt/80125307/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug
On 2021-01-21 11:51 a.m., Laszlo Ersek wrote:
On 01/21/21 20:11, Ankur Arora wrote:
On 2021-01-21 4:29 a.m., Laszlo Ersek wrote:
For catching such issues early on, please run "PatchCheck.py" before
posting, e.g. as in:
$ python3 BaseTools/Scripts/PatchCheck.py master..topic_branch
(In the present case, PatchCheck reports lots of "not CRLF" errors.)
PatchCheck.py is great. Thanks.
An even better idea is to push your topic branch (which you're about to
post to the list) to your edk2 fork on github.com, and then submit a
pull request against the "tianocore/edk2" project. The pull request will
be auto-closed in the end (it will never be merged), but the goal is to
trigger a CI run on the patch set, and to give you the error messages if
any. PatchCheck runs as a part of CI, too.
(github.com PRs are used for actual merging by edk2 maintainers, but for
that, they set the "push" label on the subject PRs, and the "push" label
is restricted to maintainers.)
I apologize about the extra delay. Would you be OK posting v5?
Sure.
Thanks for your understanding, I'm relieved.
Just a side question which I should have asked you earlier: are
the coding standards listed somewhere?
Yes, see "C Coding Standards" at
<https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Documentation#specifications>.
Although...
I had looked for answers to similar questions but the "TianoCore C style
guide" doesn't mention either PatchCheck.py or rules around CRLF.
... you may have found exactly that, already. Unfortunately, like all
documentation in the world, this document is somewhat out of date.
There is also a tool, called ECC, that contains a builtin C language
parser, that enforces various style requirements. It's built into the CI
system and runs on github -- however, we have it disabled for OVMF, in
commit ef3e73c6a0c0 ("OvmfPkg: Disable EccCheck CI until EccCheck issues
are fixed", 2020-12-14). There are two reasons for that:
- Some stuff that ECC enforces is just overzealous -- in some cases, we
relax the coding style under OvmfPkg specifically, but ECC cannot easily
be configured to tolerate those relaxations (for example, you cannot
configure it to ignore a particular error code *regardless* of where it
occurs under OvmfPkg).
- In some cases, ECC rejects C language constructs that are both valid
and conformant to the coding style (i.e., ECC has its own bugs).
So we usually "teach" these quirks to contributors during review. It's
quite a lot of investment, which is part of why we hope that
contributors stick around.
A very understandable sentiment :).
An alternative is this: in your topic branch that you intend to post to
edk2-devel, include an *extra* patch (not to be posted to the list) that
*reverts* commit ef3e73c6a0c0. And submit a github.com PR with this
variant of your topic branch. As a result, ECC will be unleashed on (the
end-stage of) your patch series. If you fix all of the ECC issues, then
reviewers should make next to zero *style* complaints -- on the other
hand, you could be updating code that ECC complains about *incorrectly*
(due to an ECC bug, or because the subject style rule is something we
deem superfluous, or at least "possible to bend", under OvmfPkg).
So perhaps try to investigate this latter avenue, and see if you are
willing to deal with what ECC throws at you. :)
Can't guarantee that I'll see it through but my curiosity on what ECC
will throw at me is piqued. Will try.
Thanks for the comments and my apologies for making you review all of
these non-substantive changes.
It's a standard part of reviewing the first few contributions of a new
community member; it's too bad we don't have better stuff in place. We
do have docs but they are slightly outdated, and we do have ECC but with
warts.
We also have recommendations (not official requirements!) on setting up
git, as a part of
<https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers>
-- which is likely out of date somewhat --; the
I guess that's the nature of non-executable bits... out of date
soon as they are written. Thanks, this and the scripts below both
were pretty useful.
BaseTools/Scripts/SetupGit.py
utility automates more or less the same settings.
We also have
BaseTools/Scripts/GetMaintainer.py
which lets you determine the Cc:'s you should put in each commit message...
And when you dump all this *process* on a new contributor, they run away
screaming -- kind of justifiedly. :/
Heh, I think I see the appeal of this gradual introduction of process.
Personally speaking, it is after working on this feature and seeing
things like SafeUintnMult(), that you see that FW generally and EDK2
specifically really does need to be written differently from, for
instance the kernel.
Thanks
Ankur
Thanks
Laszlo
Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug
On 2021-01-21 4:29 a.m., Laszlo Ersek wrote:
Hi Ankur,
On 01/18/21 19:35, Ankur Arora wrote:
On 2021-01-18 9:09 a.m., Laszlo Ersek wrote:
On 01/18/21 07:34, Ankur Arora wrote:
Hi,
This series adds support for CPU hot-unplug with OVMF.
Please see this in conjunction with the QEMU secureboot hot-unplug v2
series posted here (now upstreamed):
https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imamm...@redhat.com/
Patches 1 and 3,
("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
are either refactors or add support functions.
OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
OvmfPkg/CpuHotplugSmm: add CpuEject()
OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
Patch 2 and 9,
("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
handle the QEMU protocol logic for collection of CPU hot-unplug events
or the protocol negotiation.
Patch 4,
("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
adds the MMI logic for CPU hot-unplug handling and informing
the PiSmmCpuDxeSmm of CPU removal.
Patches 5 and 6,
("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
sets up state for doing the CPU ejection as part of hot-unplug.
Patches 7, and 8,
("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
add the CPU ejection logic.
Testing (with QEMU 5.2.50):
- Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
- Synthetic tests with simultaneous multi CPU hot-unplug
- Negotiation with/without CPU hotplug enabled
Also at:
github.com/terminus/edk2/ hot-unplug-v4
Changelog:
v4:
- Gets rid of unnecessary UefiCpuPkg changes
v3:
- Use a saner PCD based interface to share state between
PiSmmCpuDxeSmm
and OvmfPkg/CpuHotplugSmm
- Cleaner split of the hot-unplug code
URL:
https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.ar...@oracle.com/
v2:
- Do the ejection via SmmCpuFeaturesRendezvousExit()
URL:
https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.ar...@oracle.com/
RFC:
URL:
https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.ar...@oracle.com/
Please review.
I've got this series in my review queue (confirming).
I'd like to finish review on the
<https://bugzilla.tianocore.org/show_bug.cgi?id=3059> series first,
since that's what I've got mostly in my mind at this point.
I hope to start reviewing the unplug series in a few days.
Sounds good. Thanks.
I apologize for coming back with more formalities.
The patches are somewhat incorrectly composed. Looking at the very first
patch email for example, the quoted-printable encoding of the email shows:
- a hard \r in each context line of the patch (encoded as "=0D"),
- no \r character in any newly added line of the patch.
This *inconsistency* is a problem -- once I apply the patch with git-am,
it creates files with mixed LF / CRLF line terminators.
Every source file (new files in particular) should be purely CRLF.
Please update your editor's settings to stick with the line terminator
style of the file that's being edited. (When creating new files, you may
have to switch to CRLF explicitly.)
Furthermore, please convert all of the patches to purely CRLF as follows:
- run a git-rebase with "edit" actions
- at each stage, determine the set of files updated/created by the commit
- run "unix2dos" on all those files
- squash the result into the commit
- continue
(You could script this too, using the "exec" git-rebase action, but
doing it manually could be tolerable as well.)
Now, of course I can do this myself with the patches, on my end, but
(assuming at least one more version of the patch set is going to be
necessary), fixing your settings and the patches both, for future
manipulation, would now be timely.
... For reviewing non-trivial patch series, I tend to apply them first
on a local branch (nothing beats the power of the full git toolkit
during a review); that's how I found this.
For catching such issues early on, please run "PatchCheck.py" before
posting, e.g. as in:
$ python3 BaseTools/Scripts/PatchCheck.py master..topic_branch
(In the present case, PatchCheck reports lots of "not CRLF" errors.)
PatchCheck.py is great. Thanks.
An even better idea is to push your topic branch (which you're about to
post to the list) to your edk2 fork on github.com, and then submit a
pull request against the "tianocore/edk2" project. The pull request will
be auto-closed in the end (it will never be merged), but the goal is to
trigger a CI run on the patch set, and to give you the error messages if
any. PatchCheck runs as a part of CI, too.
(github.com P
