Re: [edk2-devel] [PATCH 1/2] ArmPkg: implement EFI_MP_SERVICES_PROTOCOL based on PSCI calls

2022-11-29 Thread Kun Qin
Thanks for the update, Rebecca. I will also keep an eye on the cache 
issue when I have a chance to run this on FVP models.


Regards,
Kun

On 11/28/2022 4:04 PM, Rebecca Cran wrote:

On 11/28/22 15:59, Kun Qin wrote:

Hi Rebecca/Ard,

I was trying to reach out regarding the original patches earlier (see 
below) but it might fell off your stacks
due to high traffic on the mailing list. Could you please kindly 
review the questions when you have a chance?


In addition, I found another edge case of the MP service: when the AP 
routine hits a timeout, the metadata
will be left in the unfinished states. If the AP routine eventually 
completes and return, this AP will stay in "finished"
but never become "ready" during this boot. I tried to add below 
change, which seems to work. But please let me

know if you have other concerns:
https://github.com/kuqin12/mu_silicon_arm_tiano/commit/c76072b37018276f2fec2582d0c540be5b40d0f2 



Thanks, I'll take a look and integrate the fix into the next revision 
of the patch series.




Lastly, do you plan to merge these patches in the near future? This 
will be a great add-on for ARM platforms.


The issue that's currently preventing them from being merged is a 
failure I noticed on the Neoverse N2 FVP: for some reason despite 
enabling the MMU and caches, manual cache flushes are still required 
for the data to be seen between CPUs. I don't know if that's a bug in 
the code or in the FVP model.





-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96718): https://edk2.groups.io/g/devel/message/96718
Mute This Topic: https://groups.io/mt/93329492/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-




Re: [edk2-devel] [PATCH 1/2] ArmPkg: implement EFI_MP_SERVICES_PROTOCOL based on PSCI calls

2022-11-28 Thread Rebecca Cran

On 11/28/22 15:59, Kun Qin wrote:

Hi Rebecca/Ard,

I was trying to reach out regarding the original patches earlier (see 
below) but it might fell off your stacks
due to high traffic on the mailing list. Could you please kindly review 
the questions when you have a chance?


In addition, I found another edge case of the MP service: when the AP 
routine hits a timeout, the metadata
will be left in the unfinished states. If the AP routine eventually 
completes and return, this AP will stay in "finished"
but never become "ready" during this boot. I tried to add below change, 
which seems to work. But please let me

know if you have other concerns:
https://github.com/kuqin12/mu_silicon_arm_tiano/commit/c76072b37018276f2fec2582d0c540be5b40d0f2


Thanks, I'll take a look and integrate the fix into the next revision of 
the patch series.




Lastly, do you plan to merge these patches in the near future? This will 
be a great add-on for ARM platforms.


The issue that's currently preventing them from being merged is a 
failure I noticed on the Neoverse N2 FVP: for some reason despite 
enabling the MMU and caches, manual cache flushes are still required for 
the data to be seen between CPUs. I don't know if that's a bug in the 
code or in the FVP model.


--
Rebecca Cran



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96663): https://edk2.groups.io/g/devel/message/96663
Mute This Topic: https://groups.io/mt/93329492/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-




Re: [edk2-devel] [PATCH 1/2] ArmPkg: implement EFI_MP_SERVICES_PROTOCOL based on PSCI calls

2022-11-28 Thread Rebecca Cran

Sorry, I missed this originally. I've added my replies inline.][

On 9/29/22 12:45, Kun Qin wrote:

Hi Rebecca,

Thanks for sending this patch. I have a few questions inline "[KQ]". 
Could you please help me to

understand the patch better? Thanks in advance.

Regards,
Kun

On 8/29/2022 8:59 AM, Rebecca Cran wrote:



+#define GET_MPIDR_AFFINITY_BITS(x)  ((x) & 0xFF00FF)
+
+#define MPIDR_MT_BIT  BIT24
[KQ]: Is there any reason why this is not defined in a more formal 
place? I think that will allow the platform to use as well.


That's a good point. I can certainly move it somewhere more proper.


+  BOOLEAN IsBsp;
[KQ]: Can we add a check to see if BSP is found during this routine? I 
think failing to identify a BSP might cause other issues in the 
following services?


We could certainly log an error and abort if IsBsp is never set to TRUE.


+  for (Index = 0; Index < mCpuMpData.NumberOfProcessors; Index++) {
+    if (GET_MPIDR_AFFINITY_BITS (ArmReadMpidr ()) == 
CoreInfo[Index].Mpidr) {
[KQ]: Does this mean BSP should never set the `MPIDR_MT_BIT` in the 
gArmMpCoreInfoGuid HOB data? Otherwise, this logic will never be able to 
find the BSP?


No, here we assume we're running on the BSP, and the code is looking for 
the entry in the CoreInfo array that matches the BSP's affinity fields.



+  switch (GetApState (CpuData)) {
+    case CpuStateReady:
+  SetApProcedure (CpuData, Procedure, ProcedureArgument);
+  Status = DispatchCpu (Index);
+  if (EFI_ERROR (Status)) {
+    CpuData->State = CpuStateIdle;
+    Status = EFI_NOT_READY;
+    goto Done;
[KQ]: Is it really necessary to use "goto Done"? This might cause all 
the CPUs after this failing index to not reset the CPU state.
That way, all the subsequent "StartupAllAps" will fail due to 
"CheckAllCpusReady" returning FALSE on those "dirty cores". Please

let me know if that is by design or not.


Good point. I'll fix it.


--
Rebecca Cran


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96662): https://edk2.groups.io/g/devel/message/96662
Mute This Topic: https://groups.io/mt/93329492/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-




Re: [edk2-devel] [PATCH 1/2] ArmPkg: implement EFI_MP_SERVICES_PROTOCOL based on PSCI calls

2022-11-28 Thread Kun Qin

Hi Rebecca/Ard,

I was trying to reach out regarding the original patches earlier (see 
below) but it might fell off your stacks
due to high traffic on the mailing list. Could you please kindly review 
the questions when you have a chance?


In addition, I found another edge case of the MP service: when the AP 
routine hits a timeout, the metadata
will be left in the unfinished states. If the AP routine eventually 
completes and return, this AP will stay in "finished"
but never become "ready" during this boot. I tried to add below change, 
which seems to work. But please let me

know if you have other concerns:
https://github.com/kuqin12/mu_silicon_arm_tiano/commit/c76072b37018276f2fec2582d0c540be5b40d0f2

Lastly, do you plan to merge these patches in the near future? This will 
be a great add-on for ARM platforms.


Thanks,
Kun

On 9/29/2022 11:45 AM, Kun Qin wrote:

Hi Rebecca,

Thanks for sending this patch. I have a few questions inline "[KQ]". 
Could you please help me to

understand the patch better? Thanks in advance.

Regards,
Kun

On 8/29/2022 8:59 AM, Rebecca Cran wrote:

Add support for EFI_MP_SERVICES_PROTOCOL during the DXE phase under
AArch64.

PSCI_CPU_ON is called to power on the core, the supplied procedure is
executed and PSCI_CPU_OFF is called to power off the core.

Fixes contributed by Ard Biesheuvel.

Signed-off-by: Rebecca Cran 
---
  ArmPkg/ArmPkg.dsc |    1 +
  ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf |   55 +
  ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h |  351 
  ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.c   | 1774 


  ArmPkg/Drivers/ArmPsciMpServicesDxe/MpFuncs.S |   57 +
  5 files changed, 2238 insertions(+)

diff --git a/ArmPkg/ArmPkg.dsc b/ArmPkg/ArmPkg.dsc
index 59fd8f295d4f..4716789402fc 100644
--- a/ArmPkg/ArmPkg.dsc
+++ b/ArmPkg/ArmPkg.dsc
@@ -125,6 +125,7 @@ [Components.common]
    ArmPkg/Drivers/CpuPei/CpuPei.inf
    ArmPkg/Drivers/ArmGic/ArmGicDxe.inf
    ArmPkg/Drivers/ArmGic/ArmGicLib.inf
+  ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf
    ArmPkg/Drivers/GenericWatchdogDxe/GenericWatchdogDxe.inf
    ArmPkg/Drivers/TimerDxe/TimerDxe.inf
  diff --git 
a/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf 
b/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf

new file mode 100644
index ..2b109c72e0a0
--- /dev/null
+++ b/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf
@@ -0,0 +1,55 @@
+## @file
+#  ARM MP services protocol driver
+#
+#  Copyright (c) 2022, Qualcomm Innovation Center, Inc. All rights 
reserved.

+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+[Defines]
+  INF_VERSION    = 1.27
+  BASE_NAME  = ArmPsciMpServicesDxe
+  FILE_GUID  = 007ab472-dc4a-4df8-a5c2-abb4a327278c
+  MODULE_TYPE    = DXE_DRIVER
+  VERSION_STRING = 1.0
+
+  ENTRY_POINT    = ArmPsciMpServicesDxeInitialize
+
+[Sources.Common]
+  ArmPsciMpServicesDxe.c
+  MpFuncs.S
+  MpServicesInternal.h
+
+[Packages]
+  ArmPkg/ArmPkg.dec
+  ArmPlatformPkg/ArmPlatformPkg.dec
+  EmbeddedPkg/EmbeddedPkg.dec
+  MdePkg/MdePkg.dec
+  MdeModulePkg/MdeModulePkg.dec
+
+[LibraryClasses]
+  ArmLib
+  ArmMmuLib
+  ArmSmcLib
+  BaseMemoryLib
+  CacheMaintenanceLib
+  DebugLib
+  HobLib
+  MemoryAllocationLib
+  UefiBootServicesTableLib
+  UefiDriverEntryPoint
+  UefiLib
+
+[Protocols]
+  gEfiMpServiceProtocolGuid    ## PRODUCES
+  gEfiLoadedImageProtocolGuid  ## CONSUMES
+
+[Guids]
+  gArmMpCoreInfoGuid
+
+[Depex]
+  TRUE
+
+[BuildOptions]
+  GCC:*_*_*_CC_FLAGS = -mstrict-align
diff --git a/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h 
b/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h

new file mode 100644
index ..ee13816ef64c
--- /dev/null
+++ b/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h
@@ -0,0 +1,351 @@
+/** @file
+
+Copyright (c) 2022, Qualcomm Innovation Center, Inc. All rights 
reserved.

+Copyright (c) 2006 - 2011, Intel Corporation. All rights reserved.
+Portions copyright (c) 2011, Apple Inc. All rights reserved.
+
+SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#ifndef MP_SERVICES_INTERNAL_H_
+#define MP_SERVICES_INTERNAL_H_
+
+#include 
+#include 
+
+#include 
+#include 
+
+#define AP_STACK_SIZE  0x1000
+
+//
+// Internal Data Structures
+//
+
+//
+// AP state
+//
+// The state transitions for an AP when it process a procedure are:
+//  Idle > Ready > Busy > Idle
+//   [BSP]   [AP]   [AP]
+//
+typedef enum {
+  CpuStateIdle,
+  CpuStateReady,
+  CpuStateBlocked,
+  CpuStateBusy,
+  CpuStateFinished,
+  CpuStateDisabled
+} CPU_STATE;
+
+//
+// Define Individual Processor Data block.
+//
+typedef struct {
+  EFI_PROCESSOR_INFORMATION    Info;
+  EFI_AP_PROCEDURE Procedure;
+  VOID *Parameter;
+  CPU_STATE    State;
+  

Re: [edk2-devel] [PATCH 1/2] ArmPkg: implement EFI_MP_SERVICES_PROTOCOL based on PSCI calls

2022-09-29 Thread Kun Qin

Hi Rebecca,

Thanks for sending this patch. I have a few questions inline "[KQ]". 
Could you please help me to

understand the patch better? Thanks in advance.

Regards,
Kun

On 8/29/2022 8:59 AM, Rebecca Cran wrote:

Add support for EFI_MP_SERVICES_PROTOCOL during the DXE phase under
AArch64.

PSCI_CPU_ON is called to power on the core, the supplied procedure is
executed and PSCI_CPU_OFF is called to power off the core.

Fixes contributed by Ard Biesheuvel.

Signed-off-by: Rebecca Cran 
---
  ArmPkg/ArmPkg.dsc|1 +
  ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf |   55 +
  ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h |  351 
  ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.c   | 1774 

  ArmPkg/Drivers/ArmPsciMpServicesDxe/MpFuncs.S|   57 +
  5 files changed, 2238 insertions(+)

diff --git a/ArmPkg/ArmPkg.dsc b/ArmPkg/ArmPkg.dsc
index 59fd8f295d4f..4716789402fc 100644
--- a/ArmPkg/ArmPkg.dsc
+++ b/ArmPkg/ArmPkg.dsc
@@ -125,6 +125,7 @@ [Components.common]
ArmPkg/Drivers/CpuPei/CpuPei.inf
ArmPkg/Drivers/ArmGic/ArmGicDxe.inf
ArmPkg/Drivers/ArmGic/ArmGicLib.inf
+  ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf
ArmPkg/Drivers/GenericWatchdogDxe/GenericWatchdogDxe.inf
ArmPkg/Drivers/TimerDxe/TimerDxe.inf
  
diff --git a/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf b/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf

new file mode 100644
index ..2b109c72e0a0
--- /dev/null
+++ b/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf
@@ -0,0 +1,55 @@
+## @file
+#  ARM MP services protocol driver
+#
+#  Copyright (c) 2022, Qualcomm Innovation Center, Inc. All rights 
reserved.
+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+[Defines]
+  INF_VERSION= 1.27
+  BASE_NAME  = ArmPsciMpServicesDxe
+  FILE_GUID  = 007ab472-dc4a-4df8-a5c2-abb4a327278c
+  MODULE_TYPE= DXE_DRIVER
+  VERSION_STRING = 1.0
+
+  ENTRY_POINT= ArmPsciMpServicesDxeInitialize
+
+[Sources.Common]
+  ArmPsciMpServicesDxe.c
+  MpFuncs.S
+  MpServicesInternal.h
+
+[Packages]
+  ArmPkg/ArmPkg.dec
+  ArmPlatformPkg/ArmPlatformPkg.dec
+  EmbeddedPkg/EmbeddedPkg.dec
+  MdePkg/MdePkg.dec
+  MdeModulePkg/MdeModulePkg.dec
+
+[LibraryClasses]
+  ArmLib
+  ArmMmuLib
+  ArmSmcLib
+  BaseMemoryLib
+  CacheMaintenanceLib
+  DebugLib
+  HobLib
+  MemoryAllocationLib
+  UefiBootServicesTableLib
+  UefiDriverEntryPoint
+  UefiLib
+
+[Protocols]
+  gEfiMpServiceProtocolGuid## PRODUCES
+  gEfiLoadedImageProtocolGuid  ## CONSUMES
+
+[Guids]
+  gArmMpCoreInfoGuid
+
+[Depex]
+  TRUE
+
+[BuildOptions]
+  GCC:*_*_*_CC_FLAGS = -mstrict-align
diff --git a/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h 
b/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h
new file mode 100644
index ..ee13816ef64c
--- /dev/null
+++ b/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h
@@ -0,0 +1,351 @@
+/** @file
+
+Copyright (c) 2022, Qualcomm Innovation Center, Inc. All rights reserved.
+Copyright (c) 2006 - 2011, Intel Corporation. All rights reserved.
+Portions copyright (c) 2011, Apple Inc. All rights reserved.
+
+SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#ifndef MP_SERVICES_INTERNAL_H_
+#define MP_SERVICES_INTERNAL_H_
+
+#include 
+#include 
+
+#include 
+#include 
+
+#define AP_STACK_SIZE  0x1000
+
+//
+// Internal Data Structures
+//
+
+//
+// AP state
+//
+// The state transitions for an AP when it process a procedure are:
+//  Idle > Ready > Busy > Idle
+//   [BSP]   [AP]   [AP]
+//
+typedef enum {
+  CpuStateIdle,
+  CpuStateReady,
+  CpuStateBlocked,
+  CpuStateBusy,
+  CpuStateFinished,
+  CpuStateDisabled
+} CPU_STATE;
+
+//
+// Define Individual Processor Data block.
+//
+typedef struct {
+  EFI_PROCESSOR_INFORMATIONInfo;
+  EFI_AP_PROCEDURE Procedure;
+  VOID *Parameter;
+  CPU_STATEState;
+  EFI_EVENTCheckThisAPEvent;
+  UINTNTimeout;
+  UINTNTimeTaken;
+  VOID *Ttbr0;
+  UINTNTcr;
+  UINTNMair;
+} CPU_AP_DATA;
+
+//
+// Define MP data block which consumes individual processor block.
+//
+typedef struct {
+  UINTN   NumberOfProcessors;
+  UINTN   NumberOfEnabledProcessors;
+  EFI_EVENT   CheckAllAPsEvent;
+  EFI_EVENT   WaitEvent;
+  UINTN   FinishCount;
+  UINTN   StartCount;
+  EFI_AP_PROCEDUREProcedure;
+  VOID*ProcedureArgument;
+  BOOLEAN SingleThread;
+  UINTN   StartedNumber;
+  CPU_AP_DATA *CpuData;
+  UINTN   Timeout;
+  UINTN   

[edk2-devel] [PATCH 1/2] ArmPkg: implement EFI_MP_SERVICES_PROTOCOL based on PSCI calls

2022-08-29 Thread Rebecca Cran
Add support for EFI_MP_SERVICES_PROTOCOL during the DXE phase under
AArch64.

PSCI_CPU_ON is called to power on the core, the supplied procedure is
executed and PSCI_CPU_OFF is called to power off the core.

Fixes contributed by Ard Biesheuvel.

Signed-off-by: Rebecca Cran 
---
 ArmPkg/ArmPkg.dsc|1 +
 ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf |   55 +
 ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h |  351 
 ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.c   | 1774 

 ArmPkg/Drivers/ArmPsciMpServicesDxe/MpFuncs.S|   57 +
 5 files changed, 2238 insertions(+)

diff --git a/ArmPkg/ArmPkg.dsc b/ArmPkg/ArmPkg.dsc
index 59fd8f295d4f..4716789402fc 100644
--- a/ArmPkg/ArmPkg.dsc
+++ b/ArmPkg/ArmPkg.dsc
@@ -125,6 +125,7 @@ [Components.common]
   ArmPkg/Drivers/CpuPei/CpuPei.inf
   ArmPkg/Drivers/ArmGic/ArmGicDxe.inf
   ArmPkg/Drivers/ArmGic/ArmGicLib.inf
+  ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf
   ArmPkg/Drivers/GenericWatchdogDxe/GenericWatchdogDxe.inf
   ArmPkg/Drivers/TimerDxe/TimerDxe.inf
 
diff --git a/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf 
b/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf
new file mode 100644
index ..2b109c72e0a0
--- /dev/null
+++ b/ArmPkg/Drivers/ArmPsciMpServicesDxe/ArmPsciMpServicesDxe.inf
@@ -0,0 +1,55 @@
+## @file
+#  ARM MP services protocol driver
+#
+#  Copyright (c) 2022, Qualcomm Innovation Center, Inc. All rights 
reserved.
+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+[Defines]
+  INF_VERSION= 1.27
+  BASE_NAME  = ArmPsciMpServicesDxe
+  FILE_GUID  = 007ab472-dc4a-4df8-a5c2-abb4a327278c
+  MODULE_TYPE= DXE_DRIVER
+  VERSION_STRING = 1.0
+
+  ENTRY_POINT= ArmPsciMpServicesDxeInitialize
+
+[Sources.Common]
+  ArmPsciMpServicesDxe.c
+  MpFuncs.S
+  MpServicesInternal.h
+
+[Packages]
+  ArmPkg/ArmPkg.dec
+  ArmPlatformPkg/ArmPlatformPkg.dec
+  EmbeddedPkg/EmbeddedPkg.dec
+  MdePkg/MdePkg.dec
+  MdeModulePkg/MdeModulePkg.dec
+
+[LibraryClasses]
+  ArmLib
+  ArmMmuLib
+  ArmSmcLib
+  BaseMemoryLib
+  CacheMaintenanceLib
+  DebugLib
+  HobLib
+  MemoryAllocationLib
+  UefiBootServicesTableLib
+  UefiDriverEntryPoint
+  UefiLib
+
+[Protocols]
+  gEfiMpServiceProtocolGuid## PRODUCES
+  gEfiLoadedImageProtocolGuid  ## CONSUMES
+
+[Guids]
+  gArmMpCoreInfoGuid
+
+[Depex]
+  TRUE
+
+[BuildOptions]
+  GCC:*_*_*_CC_FLAGS = -mstrict-align
diff --git a/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h 
b/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h
new file mode 100644
index ..ee13816ef64c
--- /dev/null
+++ b/ArmPkg/Drivers/ArmPsciMpServicesDxe/MpServicesInternal.h
@@ -0,0 +1,351 @@
+/** @file
+
+Copyright (c) 2022, Qualcomm Innovation Center, Inc. All rights reserved.
+Copyright (c) 2006 - 2011, Intel Corporation. All rights reserved.
+Portions copyright (c) 2011, Apple Inc. All rights reserved.
+
+SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#ifndef MP_SERVICES_INTERNAL_H_
+#define MP_SERVICES_INTERNAL_H_
+
+#include 
+#include 
+
+#include 
+#include 
+
+#define AP_STACK_SIZE  0x1000
+
+//
+// Internal Data Structures
+//
+
+//
+// AP state
+//
+// The state transitions for an AP when it process a procedure are:
+//  Idle > Ready > Busy > Idle
+//   [BSP]   [AP]   [AP]
+//
+typedef enum {
+  CpuStateIdle,
+  CpuStateReady,
+  CpuStateBlocked,
+  CpuStateBusy,
+  CpuStateFinished,
+  CpuStateDisabled
+} CPU_STATE;
+
+//
+// Define Individual Processor Data block.
+//
+typedef struct {
+  EFI_PROCESSOR_INFORMATIONInfo;
+  EFI_AP_PROCEDURE Procedure;
+  VOID *Parameter;
+  CPU_STATEState;
+  EFI_EVENTCheckThisAPEvent;
+  UINTNTimeout;
+  UINTNTimeTaken;
+  VOID *Ttbr0;
+  UINTNTcr;
+  UINTNMair;
+} CPU_AP_DATA;
+
+//
+// Define MP data block which consumes individual processor block.
+//
+typedef struct {
+  UINTN   NumberOfProcessors;
+  UINTN   NumberOfEnabledProcessors;
+  EFI_EVENT   CheckAllAPsEvent;
+  EFI_EVENT   WaitEvent;
+  UINTN   FinishCount;
+  UINTN   StartCount;
+  EFI_AP_PROCEDUREProcedure;
+  VOID*ProcedureArgument;
+  BOOLEAN SingleThread;
+  UINTN   StartedNumber;
+  CPU_AP_DATA *CpuData;
+  UINTN   Timeout;
+  UINTN   *FailedList;
+  UINTN   FailedListIndex;
+  BOOLEAN TimeoutActive;
+} CPU_MP_DATA;
+
+/** Secondary core entry point.
+
+**/
+VOID
+ApEntryPoint (
+  VOID
+  );
+
+/** C entry-point for the AP.
+