On 14/09/2023 09:16, Leo Yan wrote:
Hi all,
Hi Leo,
I'd like to discuss for how to handle the reserved memory nodes in DT binding on Xen / Arm64. Note, now I am using DTB when boot Xen but not UEFI/ACPI (ACPI is disabled in this case). ## Failure I ported Xen on a platform, after the kernel booting, the Xen hypervisor reports error: (XEN) arch/arm/p2m.c:2202: d0v0: Failing to acquire the MFN 0x1a02dc This error is caused by kernel using an invalid memory frame number 0x1a02dc, we can convert it to the address: 0x1a02dc << PAGE_SHIFT = 0x1_a02d_c000
This error is coming from get_page_from_gva(). The use of the function is usually an indication that Xen is trying to access the page. Can you use WARN() to provide a full trace?
## Reason Two important things we need to check. One is what's the DT binding passed from the bootloader to Xen hypervisor, and the second thing is what's the DT binding passed from Xen hypervisor to kernel. We can see the bootloader passes below memory nodes to Xen hypervisor: (XEN) RAM: 0000000020000000 - 00000000bfffffff (XEN) RAM: 00000001a0000000 - 00000002ffffffff (XEN) (XEN) MODULE[0]: 0000000020100000 - 0000000020265000 Xen (XEN) MODULE[1]: 0000000023000000 - 0000000023024000 Device Tree (XEN) MODULE[2]: 0000000024000000 - 0000000028000000 Kernel (XEN) RESVD[0]: 0000000020000000 - 000000002000ffff (XEN) RESVD[1]: 0000000040000000 - 000000005fffffff (XEN) RESVD[2]: 00000001a0000000 - 00000001bfffffff (XEN) RESVD[3]: 000000002e000000 - 000000002fffffff We can see the second DDR section is: [0x0000_0001_a000_0000 .. 0x0000_0002_ffff_ffff] And there have reserved memory section is: [0x0000_0001_a000_0000 .. 0x0000_0001_bfff_ffff] When register the boot memory sections, dt_unreserved_regions() will remove all reserved memory sections, which means the section [0x0000_0001_a000_0000 .. 0x0000_0001_bfff_ffff] is not managed by Xen hypervisor at all. If later kernel uses any pages in this range, Xen will report the error.
It is perfectly fine for dom0 to access a page that are not managed by Xen (a good example is MMIO regions are not managed). Now, some problems can occur if the page then needs to be access by Xen (e.g. if they contain hypercall buffers).
[...]
## Fixes I think it's wrong to add the reserved memory regions into the DT binding as normal memory nodes for Dom0 kernel. On the other hand, we cannot simply remove these reserved memory regions and don't pass to Dom0 kernel - we might reserve memory for specific purpose, for example, ramoops [1] for kernel debugging. The right thing to do is to keep these reserved memory nodes to Dom0 kernel. So one task is to record properties for these reserved memory node name and properties and pass to Dom0 kernel. The difficulty is how we can avoid allocate these reserved memory regions in Xen hypervisor. We cannot register the reserved memory into the boot pages, otherwise, the reserved memory might be allocated in the early phase. But we need to register these pages into the frame management framework and reserve them in the first place, so that we can allow Dom0 kernel to use them. (I checked a bit the static memory mechanism, seems to me we cannot use it to resolve this issue).
From my understanding reserved region are normal RAM which have been carved out for specific purpose. They may expect different caching policy (e.g. non-cachable). AFAIK, Xen doesn't have the capability to know the memory attribute (the DT binding only tell whether the region should not mapped. See the property "no-map"), hence why they were excluded from the memory management.
It would be good to understand why Xen may try to get a reference on the page.
Also, can you find the associated reserved-region and content of the Device-Tree?
Cheers, -- Julien Grall