On 27/07/2023 13:21, Heinrich Schuchardt wrote: > On 27.07.23 13:08, Emanuele Ghidoli wrote: >> Efi loader module have its own memory management that flags as reserved the >> area between ram_top to ram_end (currently where dt reserved-memory is >> falling). uboot lmb updates reserved-memory by adding these >> efi loader module areas (see lmb_reserve_common->efi_lmb_reserve). >> >> >> On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified >> in >> the device tree (used for comunication with other microcontroller on the SoC, >> OPTEE, ...) falling in the DDR range between 456 and 512 MB. >> >> Normally U-Boot relocate itself at the end of DDR so it overlap this >> reserved-memory and this of course does not work [1]. >> >> In this case U-Boot prints the following errors: >> >> ERROR: reserving fdt memory region failed (addr=9c800000 size=300000) >> ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000) >> ... >> ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000) >> >> because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT >> reserve >> memories (lmb_flag LMB_NOMAP). >> >> >> To fix this I moved the U-Boot relocation address, implementing >> board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value >> with no overlap (448 MB). >> >> >> The memory map is: >> +---------------+ 512 MB >> |DT reserved-mem| >> +---------------+ 456 MB >> |free space | >> +---------------+ 448 MB (ram_top) >> | | >> |uboot | >> | | >> +---------------+ 0 MB >> >> This is working fine (the board is able to boot, no memory overlaps) ... >> >> however, in this configuration U-Boot we still have some errors prints while >> loading linux dtb: >> >> ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 >> flags=4) >> >> And the reason of these error is complete different: The efi loader lmb >> memory >> reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area. >> >> >> 1) lib/lmb.c:lmb_reserve_common() reserve these memories: >> ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) >> 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) >> >> 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved >> area. This is not coalesced/merged with the u-boot reserved area because it >> is >> not contiguous. >> >> This is the reserved-memory array at this point: >> ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) >> ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) >> 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) >> >> 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. >> In the last two steps efi loader add some reserved area that overlap u-boot >> and fdt >> reserved area BUT they are also contiguous to the "efi loader reserved area". >> The very last step reserve also area between ram_top (448M) to ram_end (512M) >> >> We fall in this condition (where we have overlapping areas!): >> ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) >> ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) >> 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) >> >> 4) Now, while loading the linux fdt the reserved-memory areas are checked >> toward efi loader reserved area, they overlap, BUT they have different >> lmb_flag. So we get the ERROR print. >> >> Hopefully this is clear, I undestand is not that obvious ... > > Thanks Emanuele for reporting the issue. > > Could you, please, provide the output of > > efidebug memmap Verdin AM62 # efidebug memmap MMC: no card present No EFI system partition No EFI system partition Failed to persist EFI variables Type Start End Attributes ================ ================ ================ ========== CONVENTIONAL 0000000080000000-0000000098eec000 WB BOOT DATA 0000000098eec000-0000000098eee000 WB RUNTIME DATA 0000000098eee000-0000000098eef000 WB|RT BOOT DATA 0000000098eef000-0000000098ef2000 WB RUNTIME DATA 0000000098ef2000-0000000098ef3000 WB|RT BOOT DATA 0000000098ef3000-0000000098ef4000 WB RUNTIME DATA 0000000098ef4000-0000000098ef6000 WB|RT BOOT DATA 0000000098ef6000-0000000098ef7000 WB RUNTIME DATA 0000000098ef7000-0000000098f07000 WB|RT BOOT DATA 0000000098f07000-0000000098f10000 WB BOOT CODE 0000000098f10000-000000009bf20000 WB RUNTIME CODE 000000009bf20000-000000009bf30000 WB|RT BOOT CODE 000000009bf30000-000000009c000000 WB BOOT DATA 000000009c000000-00000000a0000000 WB > > and of > > bdinfo With this "debug" patch I hope giving you a better picture:
diff --git a/lib/lmb.c b/lib/lmb.c index b2c233edb64e..1feb7189936e 100644 --- a/lib/lmb.c +++ b/lib/lmb.c @@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void *fdt_blob) if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob) boot_fdt_add_mem_rsv_regions(lmb, fdt_blob); + printf("%s. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve\n", __func__); + lmb_dump_all_force(lmb); + if (CONFIG_IS_ENABLED(EFI_LOADER)) efi_lmb_reserve(lmb); } Verdin AM62 # bdinfo boot_params = 0x0000000000000000 DRAM bank = 0x0000000000000000 -> start = 0x0000000080000000 -> size = 0x0000000020000000 flashstart = 0x0000000000000000 flashsize = 0x0000000000000000 flashoffset = 0x0000000000000000 baudrate = 115200 bps relocaddr = 0x000000009bf22000 reloc off = 0x000000001b722000 Build = 64-bit current eth = ethernet@8000000port@1 ethaddr = 00:14:2d:e5:73:c1 IP addr = <NULL> fdt_blob = 0x0000000099f10890 new_fdt = 0x0000000099f10890 fdt_size = 0x000000000000f4e0 multi_dtb_fit= 0x0000000000000000 lmb_reserve_common. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x3 / max = 0x10 reserved[0] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[1] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[2] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x4 / max = 0x10 reserved[0] [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0 reserved[1] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[2] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[3] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 devicetree = separate serial addr = 0x0000000002800000 width = 0x0000000000000000 shift = 0x0000000000000002 offset = 0x0000000000000000 clock = 0x0000000002dc6c00 arch_number = 0x0000000000000000 TLB addr = 0x000000009bff0000 irq_sp = 0x0000000099f10880 sp start = 0x0000000099f10880 Early malloc usage: 2a88 / 8000 > > Best regards > > Heinrich > >> >> IMO we have two different bugs: >> - there is nothing that prevent that coalesced area may overlap other areas >> (while lmb module expect that there aren't overlapping areas) >> - efi loader (correctly) reserve between ram_top and ram_end BUT this area >> is >> reserved by fdt. >> >> One potential solution could be to override efi_add_known_memory() and set >> ram_top = ram_end, e.g. >> >> - efi_add_conventional_memory_map(ram_start, ram_end, ram_top); >> + efi_add_conventional_memory_map(ram_start, ram_end, ram_end); >> >> >> but this does not really seems like something that should ve overridden at >> the >> board level. >> >> Any suggestion? >> >> Emanuele > Very interesting the efidebug command... Thank you. Emanuele