Hello all, re-opening this old email thread ... On Sat, Aug 12, 2023 at 08:23:32AM +0200, Heinrich Schuchardt wrote: > On 7/27/23 14:14, Emanuele Ghidoli wrote: > > On 27/07/2023 13:21, Heinrich Schuchardt wrote: > > > On 27.07.23 13:08, Emanuele Ghidoli wrote: > > > > Efi loader module have its own memory management that flags as reserved > > > > the > > > > area between ram_top to ram_end (currently where dt reserved-memory is > > > > falling). uboot lmb updates reserved-memory by adding these > > > > efi loader module areas (see lmb_reserve_common->efi_lmb_reserve). > > > > > > > > > > > > On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory > > > > specified in > > > > the device tree (used for comunication with other microcontroller on > > > > the SoC, > > > > OPTEE, ...) falling in the DDR range between 456 and 512 MB. > > > > > > > > Normally U-Boot relocate itself at the end of DDR so it overlap this > > > > reserved-memory and this of course does not work [1]. > > > > > > > > In this case U-Boot prints the following errors: > > > > > > > > ERROR: reserving fdt memory region failed (addr=9c800000 > > > > size=300000) > > > > ERROR: reserving fdt memory region failed (addr=9cb00000 > > > > size=100000) > > > > ... > > > > ERROR: reserving fdt memory region failed (addr=9e800000 > > > > size=1800000) > > > > > > > > because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT > > > > reserve > > > > memories (lmb_flag LMB_NOMAP). > > > > > > > > > > > > To fix this I moved the U-Boot relocation address, implementing > > > > board_get_usable_ram_top() and therefore setting gd->ram_top to a lower > > > > value > > > > with no overlap (448 MB). > > > > > > > > > > > > The memory map is: > > > > +---------------+ 512 MB > > > > |DT reserved-mem| > > > > +---------------+ 456 MB > > > > |free space | > > > > +---------------+ 448 MB (ram_top) > > > > | | > > > > |uboot | > > > > | | > > > > +---------------+ 0 MB > > > > > > > > This is working fine (the board is able to boot, no memory overlaps) ... > > > > > > > > however, in this configuration U-Boot we still have some errors prints > > > > while > > > > loading linux dtb: > > > > > > > > ERROR: reserving fdt memory region failed (addr=9cb00000 > > > > size=100000 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9cc00000 > > > > size=e00000 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9da00000 > > > > size=100000 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9db00000 > > > > size=c00000 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 > > > > flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9e800000 > > > > size=1800000 flags=4) > > > > > > > > And the reason of these error is complete different: The efi loader lmb > > > > memory > > > > reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some > > > > area. > > > > > > > > > > > > 1) lib/lmb.c:lmb_reserve_common() reserve these memories: > > > > ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) > > > > 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) > > > > > > > > 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot > > > > reserved > > > > area. This is not coalesced/merged with the u-boot reserved area > > > > because it is > > > > not contiguous. > > > > > > > > This is the reserved-memory array at this point: > > > > ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) > > > > ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) > > > > 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) > > > > > > > > 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. > > > > In the last two steps efi loader add some reserved area that overlap > > > > u-boot and fdt > > > > reserved area BUT they are also contiguous to the "efi loader reserved > > > > area". > > > > The very last step reserve also area between ram_top (448M) to ram_end > > > > (512M) > > > > > > > > We fall in this condition (where we have overlapping areas!): > > > > ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) > > > > ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) > > > > 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) > > > > > > > > 4) Now, while loading the linux fdt the reserved-memory areas are > > > > checked > > > > toward efi loader reserved area, they overlap, BUT they have different > > > > lmb_flag. So we get the ERROR print. > > > > > > > > Hopefully this is clear, I undestand is not that obvious ... > > > > > > efidebug memmap > > Verdin AM62 # efidebug memmap > > MMC: no card present > > No EFI system partition > > No EFI system partition > > Failed to persist EFI variables > > Type Start End Attributes > > ================ ================ ================ ========== > > CONVENTIONAL 0000000080000000-0000000098eec000 WB > > BOOT DATA 0000000098eec000-0000000098eee000 WB > > RUNTIME DATA 0000000098eee000-0000000098eef000 WB|RT > > BOOT DATA 0000000098eef000-0000000098ef2000 WB > > RUNTIME DATA 0000000098ef2000-0000000098ef3000 WB|RT > > BOOT DATA 0000000098ef3000-0000000098ef4000 WB > > RUNTIME DATA 0000000098ef4000-0000000098ef6000 WB|RT > > BOOT DATA 0000000098ef6000-0000000098ef7000 WB > > RUNTIME DATA 0000000098ef7000-0000000098f07000 WB|RT > > BOOT DATA 0000000098f07000-0000000098f10000 WB > > BOOT CODE 0000000098f10000-000000009bf20000 WB > > RUNTIME CODE 000000009bf20000-000000009bf30000 WB|RT > > BOOT CODE 000000009bf30000-000000009c000000 WB > > BOOT DATA 000000009c000000-00000000a0000000 WB > > > > > > and of > > > > > > bdinfo > > With this "debug" patch I hope giving you a better picture: > > > > diff --git a/lib/lmb.c b/lib/lmb.c > > index b2c233edb64e..1feb7189936e 100644 > > --- a/lib/lmb.c > > +++ b/lib/lmb.c > > @@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void > > *fdt_blob) > > if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob) > > boot_fdt_add_mem_rsv_regions(lmb, fdt_blob); > > > > + printf("%s. Calling lmb_dump_all_force between > > boot_fdt_add_mem_rsv_regions and efi_lmb_reserve\n", __func__); > > + lmb_dump_all_force(lmb); > > + > > if (CONFIG_IS_ENABLED(EFI_LOADER)) > > efi_lmb_reserve(lmb); > > } > > > > Verdin AM62 # bdinfo > > boot_params = 0x0000000000000000 > > DRAM bank = 0x0000000000000000 > > -> start = 0x0000000080000000 > > -> size = 0x0000000020000000 > > flashstart = 0x0000000000000000 > > flashsize = 0x0000000000000000 > > flashoffset = 0x0000000000000000 > > baudrate = 115200 bps > > relocaddr = 0x000000009bf22000 > > reloc off = 0x000000001b722000 > > Build = 64-bit > > current eth = ethernet@8000000port@1 > > ethaddr = 00:14:2d:e5:73:c1 > > IP addr = <NULL> > > fdt_blob = 0x0000000099f10890 > > new_fdt = 0x0000000099f10890 > > fdt_size = 0x000000000000f4e0 > > multi_dtb_fit= 0x0000000000000000 > > lmb_reserve_common. Calling lmb_dump_all_force between > > boot_fdt_add_mem_rsv_regions and efi_lmb_reserve > > lmb_dump_all: > > memory.cnt = 0x1 / max = 0x10 > > memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 > > reserved.cnt = 0x3 / max = 0x10 > > reserved[0] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 > > reserved[1] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 > > reserved[2] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 > > lmb_dump_all: > > memory.cnt = 0x1 / max = 0x10 > > memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 > > reserved.cnt = 0x4 / max = 0x10 > > reserved[0] [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0 > > reserved[1] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 > > reserved[2] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 > > reserved[3] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 > > devicetree = separate > > serial addr = 0x0000000002800000 > > width = 0x0000000000000000 > > shift = 0x0000000000000002 > > offset = 0x0000000000000000 > > clock = 0x0000000002dc6c00 > > arch_number = 0x0000000000000000 > > TLB addr = 0x000000009bff0000 > > irq_sp = 0x0000000099f10880 > > sp start = 0x0000000099f10880 > > Early malloc usage: 2a88 / 8000 > > > > IMO we have two different bugs: > > > > - there is nothing that prevent that coalesced area may overlap > > > > other areas > > > > (while lmb module expect that there aren't overlapping areas) > > > > - efi loader (correctly) reserve between ram_top and ram_end BUT > > > > this area is > > > > reserved by fdt. > > > > > > > > One potential solution could be to override efi_add_known_memory() and > > > > set > > > > ram_top = ram_end, e.g. > > > > > > > > - efi_add_conventional_memory_map(ram_start, ram_end, > > > > ram_top); > > > > + efi_add_conventional_memory_map(ram_start, ram_end, > > > > ram_end); > > > > > > > > > > > > but this does not really seems like something that should ve overridden > > > > at the > > > > board level. > > I am able to reproduce the issue on a Pine A64 LTS with this tree: > https://source.denx.de/u-boot/custodians/u-boot-efi/-/tree/top_reserved
Patrice send this patch https://lore.kernel.org/all/20240308101230.2595220-1-patrice.chot...@foss.st.com/ to fix what it looks like the same issue on STM32MP1. Should we have a similar patch for TI K3? What do you think? Francesco