On 7/27/23 14:14, Emanuele Ghidoli wrote:
On 27/07/2023 13:21, Heinrich Schuchardt wrote:
On 27.07.23 13:08, Emanuele Ghidoli wrote:
Efi loader module have its own memory management that flags as reserved the
area between ram_top to ram_end (currently where dt reserved-memory is
falling). uboot lmb updates reserved-memory by adding these
efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in
the device tree (used for comunication with other microcontroller on the SoC,
OPTEE, ...) falling in the DDR range between 456 and 512 MB.
Normally U-Boot relocate itself at the end of DDR so it overlap this
reserved-memory and this of course does not work [1].
In this case U-Boot prints the following errors:
ERROR: reserving fdt memory region failed (addr=9c800000 size=300000)
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000)
...
ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve
memories (lmb_flag LMB_NOMAP).
To fix this I moved the U-Boot relocation address, implementing
board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value
with no overlap (448 MB).
The memory map is:
+---------------+ 512 MB
|DT reserved-mem|
+---------------+ 456 MB
|free space |
+---------------+ 448 MB (ram_top)
| |
|uboot |
| |
+---------------+ 0 MB
This is working fine (the board is able to boot, no memory overlaps) ...
however, in this configuration U-Boot we still have some errors prints while
loading linux dtb:
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000
flags=4)
ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000
flags=4)
ERROR: reserving fdt memory region failed (addr=9da00000 size=100000
flags=4)
ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000
flags=4)
ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4)
ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000
flags=4)
And the reason of these error is complete different: The efi loader lmb memory
reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
1) lib/lmb.c:lmb_reserve_common() reserve these memories:
~415M - 448M u-boot reserved area (lmb_flag LMB_NONE)
456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved
area. This is not coalesced/merged with the u-boot reserved area because it is
not contiguous.
This is the reserved-memory array at this point:
~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE)
~415M - 448M u-boot reserved area (lmb_flag LMB_NONE)
456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
3) lib/lmb.c:efi_lmb_reserve() reserve some more areas.
In the last two steps efi loader add some reserved area that overlap u-boot and
fdt
reserved area BUT they are also contiguous to the "efi loader reserved area".
The very last step reserve also area between ram_top (448M) to ram_end (512M)
We fall in this condition (where we have overlapping areas!):
~414M - 512M efi loader reserved area (lmb_flag LMB_NONE)
~415M - 448M u-boot reserved area (lmb_flag LMB_NONE)
456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
4) Now, while loading the linux fdt the reserved-memory areas are checked
toward efi loader reserved area, they overlap, BUT they have different
lmb_flag. So we get the ERROR print.
Hopefully this is clear, I undestand is not that obvious ...
Thanks Emanuele for reporting the issue.
Could you, please, provide the output of
efidebug memmap
Verdin AM62 # efidebug memmap
MMC: no card present
No EFI system partition
No EFI system partition
Failed to persist EFI variables
Type Start End Attributes
================ ================ ================ ==========
CONVENTIONAL 0000000080000000-0000000098eec000 WB
BOOT DATA 0000000098eec000-0000000098eee000 WB
RUNTIME DATA 0000000098eee000-0000000098eef000 WB|RT
BOOT DATA 0000000098eef000-0000000098ef2000 WB
RUNTIME DATA 0000000098ef2000-0000000098ef3000 WB|RT
BOOT DATA 0000000098ef3000-0000000098ef4000 WB
RUNTIME DATA 0000000098ef4000-0000000098ef6000 WB|RT
BOOT DATA 0000000098ef6000-0000000098ef7000 WB
RUNTIME DATA 0000000098ef7000-0000000098f07000 WB|RT
BOOT DATA 0000000098f07000-0000000098f10000 WB
BOOT CODE 0000000098f10000-000000009bf20000 WB
RUNTIME CODE 000000009bf20000-000000009bf30000 WB|RT
BOOT CODE 000000009bf30000-000000009c000000 WB
BOOT DATA 000000009c000000-00000000a0000000 WB
and of
bdinfo
With this "debug" patch I hope giving you a better picture:
diff --git a/lib/lmb.c b/lib/lmb.c
index b2c233edb64e..1feb7189936e 100644
--- a/lib/lmb.c
+++ b/lib/lmb.c
@@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void
*fdt_blob)
if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob)
boot_fdt_add_mem_rsv_regions(lmb, fdt_blob);
+ printf("%s. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions
and efi_lmb_reserve\n", __func__);
+ lmb_dump_all_force(lmb);
+
if (CONFIG_IS_ENABLED(EFI_LOADER))
efi_lmb_reserve(lmb);
}
Verdin AM62 # bdinfo
boot_params = 0x0000000000000000
DRAM bank = 0x0000000000000000
-> start = 0x0000000080000000
-> size = 0x0000000020000000
flashstart = 0x0000000000000000
flashsize = 0x0000000000000000
flashoffset = 0x0000000000000000
baudrate = 115200 bps
relocaddr = 0x000000009bf22000
reloc off = 0x000000001b722000
Build = 64-bit
current eth = ethernet@8000000port@1
ethaddr = 00:14:2d:e5:73:c1
IP addr = <NULL>
fdt_blob = 0x0000000099f10890
new_fdt = 0x0000000099f10890
fdt_size = 0x000000000000f4e0
multi_dtb_fit= 0x0000000000000000
lmb_reserve_common. Calling lmb_dump_all_force between
boot_fdt_add_mem_rsv_regions and efi_lmb_reserve
lmb_dump_all:
memory.cnt = 0x1 / max = 0x10
memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0
reserved.cnt = 0x3 / max = 0x10
reserved[0] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0
reserved[1] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4
reserved[2] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4
lmb_dump_all:
memory.cnt = 0x1 / max = 0x10
memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0
reserved.cnt = 0x4 / max = 0x10
reserved[0] [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0
reserved[1] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0
reserved[2] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4
reserved[3] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4
devicetree = separate
serial addr = 0x0000000002800000
width = 0x0000000000000000
shift = 0x0000000000000002
offset = 0x0000000000000000
clock = 0x0000000002dc6c00
arch_number = 0x0000000000000000
TLB addr = 0x000000009bff0000
irq_sp = 0x0000000099f10880
sp start = 0x0000000099f10880
Early malloc usage: 2a88 / 8000
Best regards
Heinrich
IMO we have two different bugs:
- there is nothing that prevent that coalesced area may overlap other areas
(while lmb module expect that there aren't overlapping areas)
- efi loader (correctly) reserve between ram_top and ram_end BUT this area is
reserved by fdt.
One potential solution could be to override efi_add_known_memory() and set
ram_top = ram_end, e.g.
- efi_add_conventional_memory_map(ram_start, ram_end, ram_top);
+ efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
but this does not really seems like something that should ve overridden at the
board level.
Any suggestion?
Emanuele
Very interesting the efidebug command...
Thank you.
Emanuele
Hello Emanuele,
I am able to reproduce the issue on a Pine A64 LTS with this tree:
https://source.denx.de/u-boot/custodians/u-boot-efi/-/tree/top_reserved
Best regards
Heinrich
