Re: [PATCH v4 0/5] DOMCTL-based guest magic region allocation for 11 domUs
Hi Stefano, Daniel, On 4/26/2024 6:18 AM, Stefano Stabellini wrote: On Thu, 18 Apr 2024, Daniel P. Smith wrote: On 4/9/24 00:53, Henry Wang wrote: An error message can seen from the init-dom0less application on direct-mapped 1:1 domains: ``` Allocating magic pages memory.c:238:d0v0 mfn 0x39000 doesn't belong to d1 Error on alloc magic pages ``` This is because populate_physmap() automatically assumes gfn == mfn for direct mapped domains. This cannot be true for the magic pages that are allocated later for 1:1 Dom0less DomUs from the init-dom0less helper application executed in Dom0. For domain using statically allocated memory but not 1:1 direct-mapped, similar error "failed to retrieve a reserved page" can be seen as the reserved memory list is empty at that time. This series tries to fix this issue using a DOMCTL-based approach, because for 1:1 direct-mapped domUs, we need to avoid the RAM regions and inform the toolstack about the region found by hypervisor for mapping the magic pages. Patch 1 introduced a new DOMCTL to get the guest memory map, currently only used for the magic page regions. Patch 2 generalized the extended region finding logic so that it can be reused for other use cases such as finding 1:1 domU magic regions. Patch 3 uses the same approach as finding the extended regions to find the guest magic page regions for direct-mapped DomUs. Patch 4 avoids hardcoding all base addresses of guest magic region in the init-dom0less application by consuming the newly introduced DOMCTL. Patch 5 is a simple patch to do some code duplication clean-up in xc. Hey Henry, To help provide some perspective, these issues are not experienced with hyperlaunch. This is because we understood early on that you cannot move a lightweight version of the toolstack into hypervisor init and not provide a mechanism to communicate what it did to the runtime control plane. We evaluated the possible mechanism, to include introducing a new hypercall op, and ultimately settled on using hypfs. The primary reason is this information is static data that, while informative later, is only necessary for the control plane to understand the state of the system. As a result, hyperlaunch is able to allocate any and all special pages required as part of domain construction and communicate their addresses to the control plane. As for XSM, hypfs is already protected and at this time we do not see any domain builder information needing to be restricted separately from the data already present in hypfs. I would like to make the suggestion that instead of continuing down this path, perhaps you might consider adopting the hyperlaunch usage of hypfs. Then adjust dom0less domain construction to allocate the special pages at construction time. The original hyperlaunch series includes a patch that provides the helper app for the xenstore announcement. And I can provide you with updated versions if that would be helpful. I also think that the new domctl is not needed and that the dom0less domain builder should allocate the magic pages. Yes this is indeed much better. Thanks Daniel for suggesting this. On ARM, we already allocate HVM_PARAM_CALLBACK_IRQ during dom0less domain build and set HVM_PARAM_STORE_PFN to ~0ULL. I think it would be only natural to extend that code to also allocate the magic pages and set HVM_PARAM_STORE_PFN (and others) correctly. If we do it that way it is simpler and consistent with the HVM_PARAM_CALLBACK_IRQ allocation, and we don't even need hypfs. Currently we do not enable hypfs in our safety certifiability configuration. It is indeed very important to consider the safety certification (which I completely missed). Therefore I've sent an updated version based on HVMOP [1]. In the future we can switch to hypfs if needed. [1] https://lore.kernel.org/xen-devel/20240426031455.579637-1-xin.wa...@amd.com/ Kind regards, Henry
Re: [PATCH v4 0/5] DOMCTL-based guest magic region allocation for 11 domUs
On Thu, 18 Apr 2024, Daniel P. Smith wrote: > On 4/9/24 00:53, Henry Wang wrote: > > An error message can seen from the init-dom0less application on > > direct-mapped 1:1 domains: > > ``` > > Allocating magic pages > > memory.c:238:d0v0 mfn 0x39000 doesn't belong to d1 > > Error on alloc magic pages > > ``` > > > > This is because populate_physmap() automatically assumes gfn == mfn > > for direct mapped domains. This cannot be true for the magic pages > > that are allocated later for 1:1 Dom0less DomUs from the init-dom0less > > helper application executed in Dom0. For domain using statically > > allocated memory but not 1:1 direct-mapped, similar error "failed to > > retrieve a reserved page" can be seen as the reserved memory list > > is empty at that time. > > > > This series tries to fix this issue using a DOMCTL-based approach, > > because for 1:1 direct-mapped domUs, we need to avoid the RAM regions > > and inform the toolstack about the region found by hypervisor for > > mapping the magic pages. Patch 1 introduced a new DOMCTL to get the > > guest memory map, currently only used for the magic page regions. > > Patch 2 generalized the extended region finding logic so that it can > > be reused for other use cases such as finding 1:1 domU magic regions. > > Patch 3 uses the same approach as finding the extended regions to find > > the guest magic page regions for direct-mapped DomUs. Patch 4 avoids > > hardcoding all base addresses of guest magic region in the init-dom0less > > application by consuming the newly introduced DOMCTL. Patch 5 is a > > simple patch to do some code duplication clean-up in xc. > > Hey Henry, > > To help provide some perspective, these issues are not experienced with > hyperlaunch. This is because we understood early on that you cannot move a > lightweight version of the toolstack into hypervisor init and not provide a > mechanism to communicate what it did to the runtime control plane. We > evaluated the possible mechanism, to include introducing a new hypercall op, > and ultimately settled on using hypfs. The primary reason is this information > is static data that, while informative later, is only necessary for the > control plane to understand the state of the system. As a result, hyperlaunch > is able to allocate any and all special pages required as part of domain > construction and communicate their addresses to the control plane. As for XSM, > hypfs is already protected and at this time we do not see any domain builder > information needing to be restricted separately from the data already present > in hypfs. > > I would like to make the suggestion that instead of continuing down this path, > perhaps you might consider adopting the hyperlaunch usage of hypfs. Then > adjust dom0less domain construction to allocate the special pages at > construction time. The original hyperlaunch series includes a patch that > provides the helper app for the xenstore announcement. And I can provide you > with updated versions if that would be helpful. I also think that the new domctl is not needed and that the dom0less domain builder should allocate the magic pages. On ARM, we already allocate HVM_PARAM_CALLBACK_IRQ during dom0less domain build and set HVM_PARAM_STORE_PFN to ~0ULL. I think it would be only natural to extend that code to also allocate the magic pages and set HVM_PARAM_STORE_PFN (and others) correctly. If we do it that way it is simpler and consistent with the HVM_PARAM_CALLBACK_IRQ allocation, and we don't even need hypfs. Currently we do not enable hypfs in our safety certifiability configuration.
Re: [PATCH v4 0/5] DOMCTL-based guest magic region allocation for 11 domUs
Hi Daniel,
On 4/18/2024 10:16 PM, Daniel P. Smith wrote:
On 4/9/24 00:53, Henry Wang wrote:
An error message can seen from the init-dom0less application on
direct-mapped 1:1 domains:
```
Allocating magic pages
memory.c:238:d0v0 mfn 0x39000 doesn't belong to d1
Error on alloc magic pages
```
This is because populate_physmap() automatically assumes gfn == mfn
for direct mapped domains. This cannot be true for the magic pages
that are allocated later for 1:1 Dom0less DomUs from the init-dom0less
helper application executed in Dom0. For domain using statically
allocated memory but not 1:1 direct-mapped, similar error "failed to
retrieve a reserved page" can be seen as the reserved memory list
is empty at that time.
This series tries to fix this issue using a DOMCTL-based approach,
because for 1:1 direct-mapped domUs, we need to avoid the RAM regions
and inform the toolstack about the region found by hypervisor for
mapping the magic pages.
Hey Henry,
To help provide some perspective, these issues are not experienced
with hyperlaunch. This is because we understood early on that you
cannot move a lightweight version of the toolstack into hypervisor
init and not provide a mechanism to communicate what it did to the
runtime control plane. We evaluated the possible mechanism, to include
introducing a new hypercall op, and ultimately settled on using hypfs.
The primary reason is this information is static data that, while
informative later, is only necessary for the control plane to
understand the state of the system. As a result, hyperlaunch is able
to allocate any and all special pages required as part of domain
construction and communicate their addresses to the control plane. As
for XSM, hypfs is already protected and at this time we do not see any
domain builder information needing to be restricted separately from
the data already present in hypfs.
I would like to make the suggestion that instead of continuing down
this path, perhaps you might consider adopting the hyperlaunch usage
of hypfs. Then adjust dom0less domain construction to allocate the
special pages at construction time.
Thank you for the suggestion. I think your proposal makes sense. However
I am not familiar with the hypfs so may I ask some questions first to
confirm if I understand your proposal correctly: Do you mean I should
firstly find, allocate and create mapping for these special pages at the
dom0less domU's construction time, then store the GPA in hypfs and
extract the GPA from init-dom0less app later on? Should I use existing
interfaces such as xenhypfs_{open,cat,ls, etc} or I may probably need to
add new hypercall ops?
The original hyperlaunch series includes a patch that provides the
helper app for the xenstore announcement. And I can provide you with
updated versions if that would be helpful.
Thank you, yes a pointer to the corresponding series and patch would be
definitely helpful.
Kind regards,
Henry
V/r,
Daniel P. Smith
Re: [PATCH v4 0/5] DOMCTL-based guest magic region allocation for 11 domUs
On 4/9/24 00:53, Henry Wang wrote: An error message can seen from the init-dom0less application on direct-mapped 1:1 domains: ``` Allocating magic pages memory.c:238:d0v0 mfn 0x39000 doesn't belong to d1 Error on alloc magic pages ``` This is because populate_physmap() automatically assumes gfn == mfn for direct mapped domains. This cannot be true for the magic pages that are allocated later for 1:1 Dom0less DomUs from the init-dom0less helper application executed in Dom0. For domain using statically allocated memory but not 1:1 direct-mapped, similar error "failed to retrieve a reserved page" can be seen as the reserved memory list is empty at that time. This series tries to fix this issue using a DOMCTL-based approach, because for 1:1 direct-mapped domUs, we need to avoid the RAM regions and inform the toolstack about the region found by hypervisor for mapping the magic pages. Patch 1 introduced a new DOMCTL to get the guest memory map, currently only used for the magic page regions. Patch 2 generalized the extended region finding logic so that it can be reused for other use cases such as finding 1:1 domU magic regions. Patch 3 uses the same approach as finding the extended regions to find the guest magic page regions for direct-mapped DomUs. Patch 4 avoids hardcoding all base addresses of guest magic region in the init-dom0less application by consuming the newly introduced DOMCTL. Patch 5 is a simple patch to do some code duplication clean-up in xc. Hey Henry, To help provide some perspective, these issues are not experienced with hyperlaunch. This is because we understood early on that you cannot move a lightweight version of the toolstack into hypervisor init and not provide a mechanism to communicate what it did to the runtime control plane. We evaluated the possible mechanism, to include introducing a new hypercall op, and ultimately settled on using hypfs. The primary reason is this information is static data that, while informative later, is only necessary for the control plane to understand the state of the system. As a result, hyperlaunch is able to allocate any and all special pages required as part of domain construction and communicate their addresses to the control plane. As for XSM, hypfs is already protected and at this time we do not see any domain builder information needing to be restricted separately from the data already present in hypfs. I would like to make the suggestion that instead of continuing down this path, perhaps you might consider adopting the hyperlaunch usage of hypfs. Then adjust dom0less domain construction to allocate the special pages at construction time. The original hyperlaunch series includes a patch that provides the helper app for the xenstore announcement. And I can provide you with updated versions if that would be helpful. V/r, Daniel P. Smith
