Re: Proposal: U-Boot memory management
Hi Tom, On Mon, 10 Jun 2024 at 10:29, Tom Rini wrote: > > On Mon, Jun 10, 2024 at 07:55:44AM -0600, Simon Glass wrote: > > Hi Sughosh, > > > > On Thu, 6 Jun 2024 at 13:18, Sughosh Ganu wrote: > > > > > > hi Simon, > > > > > > On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > > > > > > > +Sughosh Ganu for reference > > > > > > > > > > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > > > > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini > > > > > > : > > > > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > > > > >> Hi Tom, > > > > > > >> > > > > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > > > > >> > > Hi, > > > > > > >> > > > > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis > > > > > > >> > > > wrote: > > > > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > > >> > > > > > From: Tom Rini > > > > > > >> > > > > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > > > > >> > > > > > Schuchardt wrote: > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > > >> > > > > > > : > > > > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon > > > > > > >> > > > > > > >Glass wrote: > > > > > > >> > > > > > > >> Hi, > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > > >> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > Hi, > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > This records my thoughts after a discussion > > > > > > >> > > > > > > >> > with Ilias & Heinrich re > > > > > > >> > > > > > > >> > memory allocation in U-Boot. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > 1. malloc() > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > malloc() is used for programmatic memory > > > > > > >> > > > > > > >> > allocation. It allows memory > > > > > > >> > > > > > > >> > to be freed. It is not designed for very large > > > > > > >> > > > > > > >> > allocations (e.g. a > > > > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > 2. lmb > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as > > > > > > >> > > > > > > >> > those needed for a > > > > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only > > > > > > >> > > > > > > >> > transitory, for the purposes of > > > > > > >> > > > > > > >> > loading some images and booting. If the boot > > > > > > >> > > > > > > >> > fails, then all lmb > > > > > > >> > > > > > > >> > allocations go away. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > lmb is set up by getting all available memory > > > > > > >> > > > > > > >> > and then removing what > > > > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, > > > > > > >> > > > > > > >> > etc.) > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > lmb reservations have a few flags so that areas > > > > > > >> > > > > > > >> > of memory can be > > > > > > >> > > > > > > >> > provided with attributes > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a > > > > > > >> > > > > > > >> > file does an lmb > > > > > > >> > > > > > > >> > allocation but only for the purpose of avoiding > > > > > > >> > > > > > > >> > a file being loaded > > > > > > >> > > > > > > >> > over U-Boot code/data. The allocation is > > > > > > >> > > > > > > >> > dropped immediately after the > > > > > > >> > > > > > > >> > file is loaded. Within the bootm command, or > > > > > > >> > > > > > > >> > when using standard boot, > > > > > > >> > > > > > > >> > this would be fairly easy to solve. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should > > > > > > >> > > > > > > >> > consider doing the same. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > 3. EFI > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large > > > > > > >> > > > > > > >> > allocations. But via a 'pool' > > > > > > >> > > > > > > >> > function it can also do smaller allocations > > > > > > >> > > > > > > >> > similar to malloc(), > > > > > > >> > > > > > > >> > although each one uses at least 4KB at present. > > > > > > >> > > >
Re: Proposal: U-Boot memory management
On Mon, Jun 10, 2024 at 07:55:44AM -0600, Simon Glass wrote: > Hi Sughosh, > > On Thu, 6 Jun 2024 at 13:18, Sughosh Ganu wrote: > > > > hi Simon, > > > > On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > > > > > +Sughosh Ganu for reference > > > > > > > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini > > > > > : > > > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > > > >> Hi Tom, > > > > > >> > > > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini > > > > > >> wrote: > > > > > >> > > > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > > > >> > > Hi, > > > > > >> > > > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > > > >> > > wrote: > > > > > >> > > > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis > > > > > >> > > > wrote: > > > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > >> > > > > > From: Tom Rini > > > > > >> > > > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > > > >> > > > > > Schuchardt wrote: > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > >> > > > > > > : > > > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass > > > > > >> > > > > > > >wrote: > > > > > >> > > > > > > >> Hi, > > > > > >> > > > > > > >> > > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > >> > > > > > > >> wrote: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > Hi, > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > This records my thoughts after a discussion with > > > > > >> > > > > > > >> > Ilias & Heinrich re > > > > > >> > > > > > > >> > memory allocation in U-Boot. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > 1. malloc() > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > malloc() is used for programmatic memory > > > > > >> > > > > > > >> > allocation. It allows memory > > > > > >> > > > > > > >> > to be freed. It is not designed for very large > > > > > >> > > > > > > >> > allocations (e.g. a > > > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > 2. lmb > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as > > > > > >> > > > > > > >> > those needed for a > > > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only transitory, > > > > > >> > > > > > > >> > for the purposes of > > > > > >> > > > > > > >> > loading some images and booting. If the boot > > > > > >> > > > > > > >> > fails, then all lmb > > > > > >> > > > > > > >> > allocations go away. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > lmb is set up by getting all available memory and > > > > > >> > > > > > > >> > then removing what > > > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, > > > > > >> > > > > > > >> > etc.) > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > lmb reservations have a few flags so that areas > > > > > >> > > > > > > >> > of memory can be > > > > > >> > > > > > > >> > provided with attributes > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a file > > > > > >> > > > > > > >> > does an lmb > > > > > >> > > > > > > >> > allocation but only for the purpose of avoiding a > > > > > >> > > > > > > >> > file being loaded > > > > > >> > > > > > > >> > over U-Boot code/data. The allocation is dropped > > > > > >> > > > > > > >> > immediately after the > > > > > >> > > > > > > >> > file is loaded. Within the bootm command, or when > > > > > >> > > > > > > >> > using standard boot, > > > > > >> > > > > > > >> > this would be fairly easy to solve. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should > > > > > >> > > > > > > >> > consider doing the same. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > 3. EFI > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large > > > > > >> > > > > > > >> > allocations. But via a 'pool' > > > > > >> > > > > > > >> > function it can also do smaller allocations > > > > > >> > > > > > > >> > similar to malloc(), > > > > > >> > > > > > > >> > although each one uses at least 4KB at present. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > EFI allocations do not go away when a boot fails. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > With EFI it is possible to add allocations post > > > > > >> > > > > > > >> > facto, in which case > > > > > >> >
Re: Proposal: U-Boot memory management
Hi Sughosh, On Mon, 10 Jun 2024 at 09:17, Sughosh Ganu wrote: > > hi Simon, > > On Mon, 10 Jun 2024 at 20:33, Simon Glass wrote: > > > > Hi Sughosh, > > > > On Mon, 10 Jun 2024 at 08:40, Sughosh Ganu wrote: > > > > > > hi Simon, > > > > > > On Mon, 10 Jun 2024 at 19:25, Simon Glass wrote: > > > > > > > > Hi Sughosh, > > > > > > > > On Thu, 6 Jun 2024 at 13:18, Sughosh Ganu > > > > wrote: > > > > > > > > > > hi Simon, > > > > > > > > > > On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > > > > > > > > > > > +Sughosh Ganu for reference > > > > > > > > > > > > > > > > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > > > > > > > > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini > > > > > > > > : > > > > > > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > > > > > > >> Hi Tom, > > > > > > > > >> > > > > > > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass > > > > > > > > >> > wrote: > > > > > > > > >> > > Hi, > > > > > > > > >> > > > > > > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark > > > > > > > > >> > > > Kettenis wrote: > > > > > > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > > > > >> > > > > > From: Tom Rini > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > > > > > > >> > > > > > Schuchardt wrote: > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom > > > > > > > > >> > > > > > > Rini : > > > > > > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon > > > > > > > > >> > > > > > > >Glass wrote: > > > > > > > > >> > > > > > > >> Hi, > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > > > > >> > > > > > > >> wrote: > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > Hi, > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > This records my thoughts after a discussion > > > > > > > > >> > > > > > > >> > with Ilias & Heinrich re > > > > > > > > >> > > > > > > >> > memory allocation in U-Boot. > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > 1. malloc() > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > malloc() is used for programmatic memory > > > > > > > > >> > > > > > > >> > allocation. It allows memory > > > > > > > > >> > > > > > > >> > to be freed. It is not designed for very > > > > > > > > >> > > > > > > >> > large allocations (e.g. a > > > > > > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > 2. lmb > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, > > > > > > > > >> > > > > > > >> > such as those needed for a > > > > > > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only > > > > > > > > >> > > > > > > >> > transitory, for the purposes of > > > > > > > > >> > > > > > > >> > loading some images and booting. If the > > > > > > > > >> > > > > > > >> > boot fails, then all lmb > > > > > > > > >> > > > > > > >> > allocations go away. > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > lmb is set up by getting all available > > > > > > > > >> > > > > > > >> > memory and then removing what > > > > > > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() > > > > > > > > >> > > > > > > >> > space, etc.) > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > lmb reservations have a few flags so that > > > > > > > > >> > > > > > > >> > areas of memory can be > > > > > > > > >> > > > > > > >> > provided with attributes > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading > > > > > > > > >> > > > > > > >> > a file does an lmb > > > > > > > > >> > > > > > > >> > allocation but only for the purpose of > > > > > > > > >> > > > > > > >> > avoiding a file being loaded > > > > > > > > >> > > > > > > >> > over U-Boot code/data. The allocation is > > > > > > > > >> > > > > > > >> > dropped immediately after the > > > > > > > > >> > > > > > > >> > file is loaded. Within the bootm command, > > > > > > > > >> > > > > > > >> > or when using standard boot, > > > > > > > > >> > > > > > > >> > this would be fairly easy to solve. > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We > > > > > > > > >> > > > > > > >> > should
Re: Proposal: U-Boot memory management
hi Simon, On Mon, 10 Jun 2024 at 20:33, Simon Glass wrote: > > Hi Sughosh, > > On Mon, 10 Jun 2024 at 08:40, Sughosh Ganu wrote: > > > > hi Simon, > > > > On Mon, 10 Jun 2024 at 19:25, Simon Glass wrote: > > > > > > Hi Sughosh, > > > > > > On Thu, 6 Jun 2024 at 13:18, Sughosh Ganu wrote: > > > > > > > > hi Simon, > > > > > > > > On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > > > > > > > > > +Sughosh Ganu for reference > > > > > > > > > > > > > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > > > > > > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini > > > > > > > : > > > > > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > > > > > >> Hi Tom, > > > > > > > >> > > > > > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > > > > > >> > > Hi, > > > > > > > >> > > > > > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis > > > > > > > >> > > > wrote: > > > > > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > > > >> > > > > > From: Tom Rini > > > > > > > >> > > > > > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > > > > > >> > > > > > Schuchardt wrote: > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > > > >> > > > > > > : > > > > > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon > > > > > > > >> > > > > > > >Glass wrote: > > > > > > > >> > > > > > > >> Hi, > > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > > > >> > > > > > > >> wrote: > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > Hi, > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > This records my thoughts after a discussion > > > > > > > >> > > > > > > >> > with Ilias & Heinrich re > > > > > > > >> > > > > > > >> > memory allocation in U-Boot. > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > 1. malloc() > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > malloc() is used for programmatic memory > > > > > > > >> > > > > > > >> > allocation. It allows memory > > > > > > > >> > > > > > > >> > to be freed. It is not designed for very > > > > > > > >> > > > > > > >> > large allocations (e.g. a > > > > > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > 2. lmb > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such > > > > > > > >> > > > > > > >> > as those needed for a > > > > > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only > > > > > > > >> > > > > > > >> > transitory, for the purposes of > > > > > > > >> > > > > > > >> > loading some images and booting. If the boot > > > > > > > >> > > > > > > >> > fails, then all lmb > > > > > > > >> > > > > > > >> > allocations go away. > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > lmb is set up by getting all available memory > > > > > > > >> > > > > > > >> > and then removing what > > > > > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() > > > > > > > >> > > > > > > >> > space, etc.) > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > lmb reservations have a few flags so that > > > > > > > >> > > > > > > >> > areas of memory can be > > > > > > > >> > > > > > > >> > provided with attributes > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a > > > > > > > >> > > > > > > >> > file does an lmb > > > > > > > >> > > > > > > >> > allocation but only for the purpose of > > > > > > > >> > > > > > > >> > avoiding a file being loaded > > > > > > > >> > > > > > > >> > over U-Boot code/data. The allocation is > > > > > > > >> > > > > > > >> > dropped immediately after the > > > > > > > >> > > > > > > >> > file is loaded. Within the bootm command, or > > > > > > > >> > > > > > > >> > when using standard boot, > > > > > > > >> > > > > > > >> > this would be fairly easy to solve. > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should > > > > > > > >> > > > > > > >> > consider doing the same. > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > 3. EFI > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large >
Re: Proposal: U-Boot memory management
Hi Sughosh, On Mon, 10 Jun 2024 at 08:40, Sughosh Ganu wrote: > > hi Simon, > > On Mon, 10 Jun 2024 at 19:25, Simon Glass wrote: > > > > Hi Sughosh, > > > > On Thu, 6 Jun 2024 at 13:18, Sughosh Ganu wrote: > > > > > > hi Simon, > > > > > > On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > > > > > > > +Sughosh Ganu for reference > > > > > > > > > > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > > > > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini > > > > > > : > > > > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > > > > >> Hi Tom, > > > > > > >> > > > > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > > > > >> > > Hi, > > > > > > >> > > > > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis > > > > > > >> > > > wrote: > > > > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > > >> > > > > > From: Tom Rini > > > > > > >> > > > > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > > > > >> > > > > > Schuchardt wrote: > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > > >> > > > > > > : > > > > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon > > > > > > >> > > > > > > >Glass wrote: > > > > > > >> > > > > > > >> Hi, > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > > >> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > Hi, > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > This records my thoughts after a discussion > > > > > > >> > > > > > > >> > with Ilias & Heinrich re > > > > > > >> > > > > > > >> > memory allocation in U-Boot. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > 1. malloc() > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > malloc() is used for programmatic memory > > > > > > >> > > > > > > >> > allocation. It allows memory > > > > > > >> > > > > > > >> > to be freed. It is not designed for very large > > > > > > >> > > > > > > >> > allocations (e.g. a > > > > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > 2. lmb > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as > > > > > > >> > > > > > > >> > those needed for a > > > > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only > > > > > > >> > > > > > > >> > transitory, for the purposes of > > > > > > >> > > > > > > >> > loading some images and booting. If the boot > > > > > > >> > > > > > > >> > fails, then all lmb > > > > > > >> > > > > > > >> > allocations go away. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > lmb is set up by getting all available memory > > > > > > >> > > > > > > >> > and then removing what > > > > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, > > > > > > >> > > > > > > >> > etc.) > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > lmb reservations have a few flags so that areas > > > > > > >> > > > > > > >> > of memory can be > > > > > > >> > > > > > > >> > provided with attributes > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a > > > > > > >> > > > > > > >> > file does an lmb > > > > > > >> > > > > > > >> > allocation but only for the purpose of avoiding > > > > > > >> > > > > > > >> > a file being loaded > > > > > > >> > > > > > > >> > over U-Boot code/data. The allocation is > > > > > > >> > > > > > > >> > dropped immediately after the > > > > > > >> > > > > > > >> > file is loaded. Within the bootm command, or > > > > > > >> > > > > > > >> > when using standard boot, > > > > > > >> > > > > > > >> > this would be fairly easy to solve. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should > > > > > > >> > > > > > > >> > consider doing the same. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > 3. EFI > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large > > > > > > >> > > > > > > >> > allocations. But via a 'pool' > > > > > > >> > > > > > > >> > function it can also do smaller allocations > > > > > > >> > > > > > > >> > similar to malloc(), > > > > > > >> > > > > > > >> > although each one uses at least 4KB at present. > > >
Re: Proposal: U-Boot memory management
hi Simon, On Mon, 10 Jun 2024 at 19:25, Simon Glass wrote: > > Hi Sughosh, > > On Thu, 6 Jun 2024 at 13:18, Sughosh Ganu wrote: > > > > hi Simon, > > > > On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > > > > > +Sughosh Ganu for reference > > > > > > > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini > > > > > : > > > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > > > >> Hi Tom, > > > > > >> > > > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini > > > > > >> wrote: > > > > > >> > > > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > > > >> > > Hi, > > > > > >> > > > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > > > >> > > wrote: > > > > > >> > > > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis > > > > > >> > > > wrote: > > > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > >> > > > > > From: Tom Rini > > > > > >> > > > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > > > >> > > > > > Schuchardt wrote: > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > >> > > > > > > : > > > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass > > > > > >> > > > > > > >wrote: > > > > > >> > > > > > > >> Hi, > > > > > >> > > > > > > >> > > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > >> > > > > > > >> wrote: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > Hi, > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > This records my thoughts after a discussion with > > > > > >> > > > > > > >> > Ilias & Heinrich re > > > > > >> > > > > > > >> > memory allocation in U-Boot. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > 1. malloc() > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > malloc() is used for programmatic memory > > > > > >> > > > > > > >> > allocation. It allows memory > > > > > >> > > > > > > >> > to be freed. It is not designed for very large > > > > > >> > > > > > > >> > allocations (e.g. a > > > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > 2. lmb > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as > > > > > >> > > > > > > >> > those needed for a > > > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only transitory, > > > > > >> > > > > > > >> > for the purposes of > > > > > >> > > > > > > >> > loading some images and booting. If the boot > > > > > >> > > > > > > >> > fails, then all lmb > > > > > >> > > > > > > >> > allocations go away. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > lmb is set up by getting all available memory and > > > > > >> > > > > > > >> > then removing what > > > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, > > > > > >> > > > > > > >> > etc.) > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > lmb reservations have a few flags so that areas > > > > > >> > > > > > > >> > of memory can be > > > > > >> > > > > > > >> > provided with attributes > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a file > > > > > >> > > > > > > >> > does an lmb > > > > > >> > > > > > > >> > allocation but only for the purpose of avoiding a > > > > > >> > > > > > > >> > file being loaded > > > > > >> > > > > > > >> > over U-Boot code/data. The allocation is dropped > > > > > >> > > > > > > >> > immediately after the > > > > > >> > > > > > > >> > file is loaded. Within the bootm command, or when > > > > > >> > > > > > > >> > using standard boot, > > > > > >> > > > > > > >> > this would be fairly easy to solve. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should > > > > > >> > > > > > > >> > consider doing the same. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > 3. EFI > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large > > > > > >> > > > > > > >> > allocations. But via a 'pool' > > > > > >> > > > > > > >> > function it can also do smaller allocations > > > > > >> > > > > > > >> > similar to malloc(), > > > > > >> > > > > > > >> > although each one uses at least 4KB at present. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > EFI allocations do not go away when a boot fails. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > With EFI it is possible to add allocations post > > > > > >> > > > > > > >> > facto, in which case > > > > > >> >
Re: Proposal: U-Boot memory management
Hi Sughosh, On Thu, 6 Jun 2024 at 13:18, Sughosh Ganu wrote: > > hi Simon, > > On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > > > +Sughosh Ganu for reference > > > > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini : > > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > > >> Hi Tom, > > > > >> > > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini wrote: > > > > >> > > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > > >> > > Hi, > > > > >> > > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > > >> > > wrote: > > > > >> > > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > >> > > > > > From: Tom Rini > > > > >> > > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > > >> > > > > > Schuchardt wrote: > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > >> > > > > > > : > > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass > > > > >> > > > > > > >wrote: > > > > >> > > > > > > >> Hi, > > > > >> > > > > > > >> > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > >> > > > > > > >> wrote: > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > Hi, > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > This records my thoughts after a discussion with > > > > >> > > > > > > >> > Ilias & Heinrich re > > > > >> > > > > > > >> > memory allocation in U-Boot. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > 1. malloc() > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > malloc() is used for programmatic memory > > > > >> > > > > > > >> > allocation. It allows memory > > > > >> > > > > > > >> > to be freed. It is not designed for very large > > > > >> > > > > > > >> > allocations (e.g. a > > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > 2. lmb > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as > > > > >> > > > > > > >> > those needed for a > > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only transitory, > > > > >> > > > > > > >> > for the purposes of > > > > >> > > > > > > >> > loading some images and booting. If the boot fails, > > > > >> > > > > > > >> > then all lmb > > > > >> > > > > > > >> > allocations go away. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > lmb is set up by getting all available memory and > > > > >> > > > > > > >> > then removing what > > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > lmb reservations have a few flags so that areas of > > > > >> > > > > > > >> > memory can be > > > > >> > > > > > > >> > provided with attributes > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a file > > > > >> > > > > > > >> > does an lmb > > > > >> > > > > > > >> > allocation but only for the purpose of avoiding a > > > > >> > > > > > > >> > file being loaded > > > > >> > > > > > > >> > over U-Boot code/data. The allocation is dropped > > > > >> > > > > > > >> > immediately after the > > > > >> > > > > > > >> > file is loaded. Within the bootm command, or when > > > > >> > > > > > > >> > using standard boot, > > > > >> > > > > > > >> > this would be fairly easy to solve. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should > > > > >> > > > > > > >> > consider doing the same. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > 3. EFI > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large > > > > >> > > > > > > >> > allocations. But via a 'pool' > > > > >> > > > > > > >> > function it can also do smaller allocations similar > > > > >> > > > > > > >> > to malloc(), > > > > >> > > > > > > >> > although each one uses at least 4KB at present. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > EFI allocations do not go away when a boot fails. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > With EFI it is possible to add allocations post > > > > >> > > > > > > >> > facto, in which case > > > > >> > > > > > > >> > they are added to the allocation table just as if > > > > >> > > > > > > >> > the memory was > > > > >> > > > > > > >> > allocated with EFI to begin with. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > The EFI allocations and the lmb allocations use the > > > > >> > > > > > > >> > same memory, so in > > > > >> > > > > >
Re: Proposal: U-Boot memory management
hi Simon, On Wed, 29 May 2024 at 22:00, Simon Glass wrote: > > +Sughosh Ganu for reference > > > On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini : > > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > > >> Hi Tom, > > > >> > > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini wrote: > > > >> > > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > >> > > Hi, > > > >> > > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini > > > >> > > wrote: > > > >> > > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > >> > > > > > From: Tom Rini > > > >> > > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich > > > >> > > > > > Schuchardt wrote: > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > >> > > > > > > : > > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass > > > >> > > > > > > >wrote: > > > >> > > > > > > >> Hi, > > > >> > > > > > > >> > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > >> > > > > > > >> wrote: > > > >> > > > > > > >> > > > > >> > > > > > > >> > Hi, > > > >> > > > > > > >> > > > > >> > > > > > > >> > This records my thoughts after a discussion with > > > >> > > > > > > >> > Ilias & Heinrich re > > > >> > > > > > > >> > memory allocation in U-Boot. > > > >> > > > > > > >> > > > > >> > > > > > > >> > 1. malloc() > > > >> > > > > > > >> > > > > >> > > > > > > >> > malloc() is used for programmatic memory allocation. > > > >> > > > > > > >> > It allows memory > > > >> > > > > > > >> > to be freed. It is not designed for very large > > > >> > > > > > > >> > allocations (e.g. a > > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > >> > > > > > > >> > > > > >> > > > > > > >> > 2. lmb > > > >> > > > > > > >> > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as those > > > >> > > > > > > >> > needed for a > > > >> > > > > > > >> > kernel or ramdisk. Allocation is only transitory, for > > > >> > > > > > > >> > the purposes of > > > >> > > > > > > >> > loading some images and booting. If the boot fails, > > > >> > > > > > > >> > then all lmb > > > >> > > > > > > >> > allocations go away. > > > >> > > > > > > >> > > > > >> > > > > > > >> > lmb is set up by getting all available memory and > > > >> > > > > > > >> > then removing what > > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > > >> > > > > > > >> > > > > >> > > > > > > >> > lmb reservations have a few flags so that areas of > > > >> > > > > > > >> > memory can be > > > >> > > > > > > >> > provided with attributes > > > >> > > > > > > >> > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a file > > > >> > > > > > > >> > does an lmb > > > >> > > > > > > >> > allocation but only for the purpose of avoiding a > > > >> > > > > > > >> > file being loaded > > > >> > > > > > > >> > over U-Boot code/data. The allocation is dropped > > > >> > > > > > > >> > immediately after the > > > >> > > > > > > >> > file is loaded. Within the bootm command, or when > > > >> > > > > > > >> > using standard boot, > > > >> > > > > > > >> > this would be fairly easy to solve. > > > >> > > > > > > >> > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should consider > > > >> > > > > > > >> > doing the same. > > > >> > > > > > > >> > > > > >> > > > > > > >> > 3. EFI > > > >> > > > > > > >> > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > >> > > > > > > >> > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large allocations. > > > >> > > > > > > >> > But via a 'pool' > > > >> > > > > > > >> > function it can also do smaller allocations similar > > > >> > > > > > > >> > to malloc(), > > > >> > > > > > > >> > although each one uses at least 4KB at present. > > > >> > > > > > > >> > > > > >> > > > > > > >> > EFI allocations do not go away when a boot fails. > > > >> > > > > > > >> > > > > >> > > > > > > >> > With EFI it is possible to add allocations post > > > >> > > > > > > >> > facto, in which case > > > >> > > > > > > >> > they are added to the allocation table just as if the > > > >> > > > > > > >> > memory was > > > >> > > > > > > >> > allocated with EFI to begin with. > > > >> > > > > > > >> > > > > >> > > > > > > >> > The EFI allocations and the lmb allocations use the > > > >> > > > > > > >> > same memory, so in > > > >> > > > > > > >> > principle could conflict. > > > >> > > > > > > >> > > > > >> > > > > > > >> > EFI allocations are sometimes used to allocate > > > >> > > > > > > >> > internal U-Boot data as > > > >> > > > > > > >> > well, if needed by the EFI app. For example, while > > > >> > > > >
Re: Proposal: U-Boot memory management
+Sughosh Ganu for reference On Sun, 31 Dec 2023 at 09:16, Tom Rini wrote: > > On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini : > > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > > >> Hi Tom, > > >> > > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini wrote: > > >> > > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > >> > > Hi, > > >> > > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini wrote: > > >> > > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > >> > > > > > From: Tom Rini > > >> > > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt > > >> > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > >> > > > > > > : > > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > > >> > > > > > > >> Hi, > > >> > > > > > > >> > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > >> > > > > > > >> wrote: > > >> > > > > > > >> > > > >> > > > > > > >> > Hi, > > >> > > > > > > >> > > > >> > > > > > > >> > This records my thoughts after a discussion with Ilias > > >> > > > > > > >> > & Heinrich re > > >> > > > > > > >> > memory allocation in U-Boot. > > >> > > > > > > >> > > > >> > > > > > > >> > 1. malloc() > > >> > > > > > > >> > > > >> > > > > > > >> > malloc() is used for programmatic memory allocation. It > > >> > > > > > > >> > allows memory > > >> > > > > > > >> > to be freed. It is not designed for very large > > >> > > > > > > >> > allocations (e.g. a > > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > > >> > > > > > > >> > > > >> > > > > > > >> > 2. lmb > > >> > > > > > > >> > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as those > > >> > > > > > > >> > needed for a > > >> > > > > > > >> > kernel or ramdisk. Allocation is only transitory, for > > >> > > > > > > >> > the purposes of > > >> > > > > > > >> > loading some images and booting. If the boot fails, > > >> > > > > > > >> > then all lmb > > >> > > > > > > >> > allocations go away. > > >> > > > > > > >> > > > >> > > > > > > >> > lmb is set up by getting all available memory and then > > >> > > > > > > >> > removing what > > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > >> > > > > > > >> > > > >> > > > > > > >> > lmb reservations have a few flags so that areas of > > >> > > > > > > >> > memory can be > > >> > > > > > > >> > provided with attributes > > >> > > > > > > >> > > > >> > > > > > > >> > There are some corner cases...e.g. loading a file does > > >> > > > > > > >> > an lmb > > >> > > > > > > >> > allocation but only for the purpose of avoiding a file > > >> > > > > > > >> > being loaded > > >> > > > > > > >> > over U-Boot code/data. The allocation is dropped > > >> > > > > > > >> > immediately after the > > >> > > > > > > >> > file is loaded. Within the bootm command, or when using > > >> > > > > > > >> > standard boot, > > >> > > > > > > >> > this would be fairly easy to solve. > > >> > > > > > > >> > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should consider > > >> > > > > > > >> > doing the same. > > >> > > > > > > >> > > > >> > > > > > > >> > 3. EFI > > >> > > > > > > >> > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > >> > > > > > > >> > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large allocations. > > >> > > > > > > >> > But via a 'pool' > > >> > > > > > > >> > function it can also do smaller allocations similar to > > >> > > > > > > >> > malloc(), > > >> > > > > > > >> > although each one uses at least 4KB at present. > > >> > > > > > > >> > > > >> > > > > > > >> > EFI allocations do not go away when a boot fails. > > >> > > > > > > >> > > > >> > > > > > > >> > With EFI it is possible to add allocations post facto, > > >> > > > > > > >> > in which case > > >> > > > > > > >> > they are added to the allocation table just as if the > > >> > > > > > > >> > memory was > > >> > > > > > > >> > allocated with EFI to begin with. > > >> > > > > > > >> > > > >> > > > > > > >> > The EFI allocations and the lmb allocations use the > > >> > > > > > > >> > same memory, so in > > >> > > > > > > >> > principle could conflict. > > >> > > > > > > >> > > > >> > > > > > > >> > EFI allocations are sometimes used to allocate internal > > >> > > > > > > >> > U-Boot data as > > >> > > > > > > >> > well, if needed by the EFI app. For example, while > > >> > > > > > > >> > efi_image_parse() > > >> > > > > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since > > >> > > > > > > >> > the code runs > > >> > > > > > > >> > in the app context and may need to access the memory > > >> > > > > > > >> > after U-Boot has > > >> > > > > > > >> > exited. Also efi_smb
Re: Proposal: U-Boot memory management
On Sun, Dec 31, 2023 at 04:40:06PM +0100, Heinrich Schuchardt wrote: > > > Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini : > >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > >> Hi Tom, > >> > >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini wrote: > >> > > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > >> > > Hi, > >> > > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini wrote: > >> > > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > >> > > > > > From: Tom Rini > >> > > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt > >> > > > > > wrote: > >> > > > > > > > >> > > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > >> > > > > > > : > >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > >> > > > > > > >> Hi, > >> > > > > > > >> > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > >> > > > > > > >> wrote: > >> > > > > > > >> > > >> > > > > > > >> > Hi, > >> > > > > > > >> > > >> > > > > > > >> > This records my thoughts after a discussion with Ilias & > >> > > > > > > >> > Heinrich re > >> > > > > > > >> > memory allocation in U-Boot. > >> > > > > > > >> > > >> > > > > > > >> > 1. malloc() > >> > > > > > > >> > > >> > > > > > > >> > malloc() is used for programmatic memory allocation. It > >> > > > > > > >> > allows memory > >> > > > > > > >> > to be freed. It is not designed for very large > >> > > > > > > >> > allocations (e.g. a > >> > > > > > > >> > 10MB kernel or 100MB ramdisk). > >> > > > > > > >> > > >> > > > > > > >> > 2. lmb > >> > > > > > > >> > > >> > > > > > > >> > lmb is used for large blocks of memory, such as those > >> > > > > > > >> > needed for a > >> > > > > > > >> > kernel or ramdisk. Allocation is only transitory, for the > >> > > > > > > >> > purposes of > >> > > > > > > >> > loading some images and booting. If the boot fails, then > >> > > > > > > >> > all lmb > >> > > > > > > >> > allocations go away. > >> > > > > > > >> > > >> > > > > > > >> > lmb is set up by getting all available memory and then > >> > > > > > > >> > removing what > >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > >> > > > > > > >> > > >> > > > > > > >> > lmb reservations have a few flags so that areas of memory > >> > > > > > > >> > can be > >> > > > > > > >> > provided with attributes > >> > > > > > > >> > > >> > > > > > > >> > There are some corner cases...e.g. loading a file does an > >> > > > > > > >> > lmb > >> > > > > > > >> > allocation but only for the purpose of avoiding a file > >> > > > > > > >> > being loaded > >> > > > > > > >> > over U-Boot code/data. The allocation is dropped > >> > > > > > > >> > immediately after the > >> > > > > > > >> > file is loaded. Within the bootm command, or when using > >> > > > > > > >> > standard boot, > >> > > > > > > >> > this would be fairly easy to solve. > >> > > > > > > >> > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should consider > >> > > > > > > >> > doing the same. > >> > > > > > > >> > > >> > > > > > > >> > 3. EFI > >> > > > > > > >> > > >> > > > > > > >> > EFI has its own memory-allocation tables. > >> > > > > > > >> > > >> > > > > > > >> > Like lmb, EFI is able to deal with large allocations. But > >> > > > > > > >> > via a 'pool' > >> > > > > > > >> > function it can also do smaller allocations similar to > >> > > > > > > >> > malloc(), > >> > > > > > > >> > although each one uses at least 4KB at present. > >> > > > > > > >> > > >> > > > > > > >> > EFI allocations do not go away when a boot fails. > >> > > > > > > >> > > >> > > > > > > >> > With EFI it is possible to add allocations post facto, in > >> > > > > > > >> > which case > >> > > > > > > >> > they are added to the allocation table just as if the > >> > > > > > > >> > memory was > >> > > > > > > >> > allocated with EFI to begin with. > >> > > > > > > >> > > >> > > > > > > >> > The EFI allocations and the lmb allocations use the same > >> > > > > > > >> > memory, so in > >> > > > > > > >> > principle could conflict. > >> > > > > > > >> > > >> > > > > > > >> > EFI allocations are sometimes used to allocate internal > >> > > > > > > >> > U-Boot data as > >> > > > > > > >> > well, if needed by the EFI app. For example, while > >> > > > > > > >> > efi_image_parse() > >> > > > > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since > >> > > > > > > >> > the code runs > >> > > > > > > >> > in the app context and may need to access the memory > >> > > > > > > >> > after U-Boot has > >> > > > > > > >> > exited. Also efi_smbios.c uses allocate_pages() and then > >> > > > > > > >> > adds a new > >> > > > > > > >> > mapping as well. > >> > > > > > > >> > > >> > > > > > > >> > EFI memory has attributes, including what the memory is > >> > > > > > > >> > used for (to > >> > > > > > > >> > some degree of granu
Re: Proposal: U-Boot memory management
Am 31. Dezember 2023 16:11:44 MEZ schrieb Tom Rini : >On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: >> Hi Tom, >> >> On Sun, Dec 31, 2023 at 6:54 AM Tom Rini wrote: >> > >> > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: >> > > Hi, >> > > >> > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini wrote: >> > > > >> > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: >> > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 >> > > > > > From: Tom Rini >> > > > > > >> > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt >> > > > > > wrote: >> > > > > > > >> > > > > > > >> > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini >> > > > > > > : >> > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: >> > > > > > > >> Hi, >> > > > > > > >> >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass >> > > > > > > >> wrote: >> > > > > > > >> > >> > > > > > > >> > Hi, >> > > > > > > >> > >> > > > > > > >> > This records my thoughts after a discussion with Ilias & >> > > > > > > >> > Heinrich re >> > > > > > > >> > memory allocation in U-Boot. >> > > > > > > >> > >> > > > > > > >> > 1. malloc() >> > > > > > > >> > >> > > > > > > >> > malloc() is used for programmatic memory allocation. It >> > > > > > > >> > allows memory >> > > > > > > >> > to be freed. It is not designed for very large allocations >> > > > > > > >> > (e.g. a >> > > > > > > >> > 10MB kernel or 100MB ramdisk). >> > > > > > > >> > >> > > > > > > >> > 2. lmb >> > > > > > > >> > >> > > > > > > >> > lmb is used for large blocks of memory, such as those >> > > > > > > >> > needed for a >> > > > > > > >> > kernel or ramdisk. Allocation is only transitory, for the >> > > > > > > >> > purposes of >> > > > > > > >> > loading some images and booting. If the boot fails, then >> > > > > > > >> > all lmb >> > > > > > > >> > allocations go away. >> > > > > > > >> > >> > > > > > > >> > lmb is set up by getting all available memory and then >> > > > > > > >> > removing what >> > > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) >> > > > > > > >> > >> > > > > > > >> > lmb reservations have a few flags so that areas of memory >> > > > > > > >> > can be >> > > > > > > >> > provided with attributes >> > > > > > > >> > >> > > > > > > >> > There are some corner cases...e.g. loading a file does an >> > > > > > > >> > lmb >> > > > > > > >> > allocation but only for the purpose of avoiding a file >> > > > > > > >> > being loaded >> > > > > > > >> > over U-Boot code/data. The allocation is dropped >> > > > > > > >> > immediately after the >> > > > > > > >> > file is loaded. Within the bootm command, or when using >> > > > > > > >> > standard boot, >> > > > > > > >> > this would be fairly easy to solve. >> > > > > > > >> > >> > > > > > > >> > Linux has renamed lmb to memblock. We should consider doing >> > > > > > > >> > the same. >> > > > > > > >> > >> > > > > > > >> > 3. EFI >> > > > > > > >> > >> > > > > > > >> > EFI has its own memory-allocation tables. >> > > > > > > >> > >> > > > > > > >> > Like lmb, EFI is able to deal with large allocations. But >> > > > > > > >> > via a 'pool' >> > > > > > > >> > function it can also do smaller allocations similar to >> > > > > > > >> > malloc(), >> > > > > > > >> > although each one uses at least 4KB at present. >> > > > > > > >> > >> > > > > > > >> > EFI allocations do not go away when a boot fails. >> > > > > > > >> > >> > > > > > > >> > With EFI it is possible to add allocations post facto, in >> > > > > > > >> > which case >> > > > > > > >> > they are added to the allocation table just as if the >> > > > > > > >> > memory was >> > > > > > > >> > allocated with EFI to begin with. >> > > > > > > >> > >> > > > > > > >> > The EFI allocations and the lmb allocations use the same >> > > > > > > >> > memory, so in >> > > > > > > >> > principle could conflict. >> > > > > > > >> > >> > > > > > > >> > EFI allocations are sometimes used to allocate internal >> > > > > > > >> > U-Boot data as >> > > > > > > >> > well, if needed by the EFI app. For example, while >> > > > > > > >> > efi_image_parse() >> > > > > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the >> > > > > > > >> > code runs >> > > > > > > >> > in the app context and may need to access the memory after >> > > > > > > >> > U-Boot has >> > > > > > > >> > exited. Also efi_smbios.c uses allocate_pages() and then >> > > > > > > >> > adds a new >> > > > > > > >> > mapping as well. >> > > > > > > >> > >> > > > > > > >> > EFI memory has attributes, including what the memory is >> > > > > > > >> > used for (to >> > > > > > > >> > some degree of granularity). See enum efi_memory_type and >> > > > > > > >> > struct >> > > > > > > >> > efi_mem_desc. In the latter there are also attribute flags >> > > > > > > >> > - whether >> > > > > > > >> > memory is cacheable, etc. >> > > > > > > >> > >> > > > > > > >> > EFI also has the x86 idea of
Re: Proposal: U-Boot memory management
On Sun, Dec 31, 2023 at 07:22:10AM -0700, Simon Glass wrote: > Hi Tom, > > On Sun, Dec 31, 2023 at 6:54 AM Tom Rini wrote: > > > > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > > Hi, > > > > > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini wrote: > > > > > > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > > From: Tom Rini > > > > > > > > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > > > : > > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > > > > > > > >> Hi, > > > > > > > >> > > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > >> > Hi, > > > > > > > >> > > > > > > > > >> > This records my thoughts after a discussion with Ilias & > > > > > > > >> > Heinrich re > > > > > > > >> > memory allocation in U-Boot. > > > > > > > >> > > > > > > > > >> > 1. malloc() > > > > > > > >> > > > > > > > > >> > malloc() is used for programmatic memory allocation. It > > > > > > > >> > allows memory > > > > > > > >> > to be freed. It is not designed for very large allocations > > > > > > > >> > (e.g. a > > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > > > >> > > > > > > > > >> > 2. lmb > > > > > > > >> > > > > > > > > >> > lmb is used for large blocks of memory, such as those needed > > > > > > > >> > for a > > > > > > > >> > kernel or ramdisk. Allocation is only transitory, for the > > > > > > > >> > purposes of > > > > > > > >> > loading some images and booting. If the boot fails, then all > > > > > > > >> > lmb > > > > > > > >> > allocations go away. > > > > > > > >> > > > > > > > > >> > lmb is set up by getting all available memory and then > > > > > > > >> > removing what > > > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > > > > > > >> > > > > > > > > >> > lmb reservations have a few flags so that areas of memory > > > > > > > >> > can be > > > > > > > >> > provided with attributes > > > > > > > >> > > > > > > > > >> > There are some corner cases...e.g. loading a file does an lmb > > > > > > > >> > allocation but only for the purpose of avoiding a file being > > > > > > > >> > loaded > > > > > > > >> > over U-Boot code/data. The allocation is dropped immediately > > > > > > > >> > after the > > > > > > > >> > file is loaded. Within the bootm command, or when using > > > > > > > >> > standard boot, > > > > > > > >> > this would be fairly easy to solve. > > > > > > > >> > > > > > > > > >> > Linux has renamed lmb to memblock. We should consider doing > > > > > > > >> > the same. > > > > > > > >> > > > > > > > > >> > 3. EFI > > > > > > > >> > > > > > > > > >> > EFI has its own memory-allocation tables. > > > > > > > >> > > > > > > > > >> > Like lmb, EFI is able to deal with large allocations. But > > > > > > > >> > via a 'pool' > > > > > > > >> > function it can also do smaller allocations similar to > > > > > > > >> > malloc(), > > > > > > > >> > although each one uses at least 4KB at present. > > > > > > > >> > > > > > > > > >> > EFI allocations do not go away when a boot fails. > > > > > > > >> > > > > > > > > >> > With EFI it is possible to add allocations post facto, in > > > > > > > >> > which case > > > > > > > >> > they are added to the allocation table just as if the memory > > > > > > > >> > was > > > > > > > >> > allocated with EFI to begin with. > > > > > > > >> > > > > > > > > >> > The EFI allocations and the lmb allocations use the same > > > > > > > >> > memory, so in > > > > > > > >> > principle could conflict. > > > > > > > >> > > > > > > > > >> > EFI allocations are sometimes used to allocate internal > > > > > > > >> > U-Boot data as > > > > > > > >> > well, if needed by the EFI app. For example, while > > > > > > > >> > efi_image_parse() > > > > > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the > > > > > > > >> > code runs > > > > > > > >> > in the app context and may need to access the memory after > > > > > > > >> > U-Boot has > > > > > > > >> > exited. Also efi_smbios.c uses allocate_pages() and then > > > > > > > >> > adds a new > > > > > > > >> > mapping as well. > > > > > > > >> > > > > > > > > >> > EFI memory has attributes, including what the memory is used > > > > > > > >> > for (to > > > > > > > >> > some degree of granularity). See enum efi_memory_type and > > > > > > > >> > struct > > > > > > > >> > efi_mem_desc. In the latter there are also attribute flags - > > > > > > > >> > whether > > > > > > > >> > memory is cacheable, etc. > > > > > > > >> > > > > > > > > >> > EFI also has the x86 idea of 'conventional' memory, meaning > > > > > > > >> > (I > > > > > > > >> > believe) that below 4GB that isn't reserved for the > > > > > > > >> > hardware/system. > > > > > > > >> > This is meaningless,
Re: Proposal: U-Boot memory management
Hi Tom, On Sun, Dec 31, 2023 at 6:54 AM Tom Rini wrote: > > On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > > Hi, > > > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini wrote: > > > > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > > From: Tom Rini > > > > > > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > > : > > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > > > > > > >> Hi, > > > > > > >> > > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > Hi, > > > > > > >> > > > > > > > >> > This records my thoughts after a discussion with Ilias & > > > > > > >> > Heinrich re > > > > > > >> > memory allocation in U-Boot. > > > > > > >> > > > > > > > >> > 1. malloc() > > > > > > >> > > > > > > > >> > malloc() is used for programmatic memory allocation. It allows > > > > > > >> > memory > > > > > > >> > to be freed. It is not designed for very large allocations > > > > > > >> > (e.g. a > > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > > >> > > > > > > > >> > 2. lmb > > > > > > >> > > > > > > > >> > lmb is used for large blocks of memory, such as those needed > > > > > > >> > for a > > > > > > >> > kernel or ramdisk. Allocation is only transitory, for the > > > > > > >> > purposes of > > > > > > >> > loading some images and booting. If the boot fails, then all > > > > > > >> > lmb > > > > > > >> > allocations go away. > > > > > > >> > > > > > > > >> > lmb is set up by getting all available memory and then > > > > > > >> > removing what > > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > > > > > >> > > > > > > > >> > lmb reservations have a few flags so that areas of memory can > > > > > > >> > be > > > > > > >> > provided with attributes > > > > > > >> > > > > > > > >> > There are some corner cases...e.g. loading a file does an lmb > > > > > > >> > allocation but only for the purpose of avoiding a file being > > > > > > >> > loaded > > > > > > >> > over U-Boot code/data. The allocation is dropped immediately > > > > > > >> > after the > > > > > > >> > file is loaded. Within the bootm command, or when using > > > > > > >> > standard boot, > > > > > > >> > this would be fairly easy to solve. > > > > > > >> > > > > > > > >> > Linux has renamed lmb to memblock. We should consider doing > > > > > > >> > the same. > > > > > > >> > > > > > > > >> > 3. EFI > > > > > > >> > > > > > > > >> > EFI has its own memory-allocation tables. > > > > > > >> > > > > > > > >> > Like lmb, EFI is able to deal with large allocations. But via > > > > > > >> > a 'pool' > > > > > > >> > function it can also do smaller allocations similar to > > > > > > >> > malloc(), > > > > > > >> > although each one uses at least 4KB at present. > > > > > > >> > > > > > > > >> > EFI allocations do not go away when a boot fails. > > > > > > >> > > > > > > > >> > With EFI it is possible to add allocations post facto, in > > > > > > >> > which case > > > > > > >> > they are added to the allocation table just as if the memory > > > > > > >> > was > > > > > > >> > allocated with EFI to begin with. > > > > > > >> > > > > > > > >> > The EFI allocations and the lmb allocations use the same > > > > > > >> > memory, so in > > > > > > >> > principle could conflict. > > > > > > >> > > > > > > > >> > EFI allocations are sometimes used to allocate internal U-Boot > > > > > > >> > data as > > > > > > >> > well, if needed by the EFI app. For example, while > > > > > > >> > efi_image_parse() > > > > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the > > > > > > >> > code runs > > > > > > >> > in the app context and may need to access the memory after > > > > > > >> > U-Boot has > > > > > > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds > > > > > > >> > a new > > > > > > >> > mapping as well. > > > > > > >> > > > > > > > >> > EFI memory has attributes, including what the memory is used > > > > > > >> > for (to > > > > > > >> > some degree of granularity). See enum efi_memory_type and > > > > > > >> > struct > > > > > > >> > efi_mem_desc. In the latter there are also attribute flags - > > > > > > >> > whether > > > > > > >> > memory is cacheable, etc. > > > > > > >> > > > > > > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > > > > > > >> > believe) that below 4GB that isn't reserved for the > > > > > > >> > hardware/system. > > > > > > >> > This is meaningless, or at least confusing, on ARM systems. > > > > > > >> > > > > > > > >> > 4. reservations > > > > > > >> > > > > > > > >> > It is perhaps worth mentioning a fourth method of memory > > > > > > >> > management, > > > > > > >> > where U-Boot reserves chunks of memory before relocation (in > > > > > > >> > boar
Re: Proposal: U-Boot memory management
On Sun, Dec 31, 2023 at 05:48:23AM -0700, Simon Glass wrote: > Hi, > > On Fri, Dec 29, 2023 at 10:52 AM Tom Rini wrote: > > > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > > From: Tom Rini > > > > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini > > > > > : > > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > > > > > >> Hi, > > > > > >> > > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > > >> wrote: > > > > > >> > > > > > > >> > Hi, > > > > > >> > > > > > > >> > This records my thoughts after a discussion with Ilias & > > > > > >> > Heinrich re > > > > > >> > memory allocation in U-Boot. > > > > > >> > > > > > > >> > 1. malloc() > > > > > >> > > > > > > >> > malloc() is used for programmatic memory allocation. It allows > > > > > >> > memory > > > > > >> > to be freed. It is not designed for very large allocations (e.g. > > > > > >> > a > > > > > >> > 10MB kernel or 100MB ramdisk). > > > > > >> > > > > > > >> > 2. lmb > > > > > >> > > > > > > >> > lmb is used for large blocks of memory, such as those needed for > > > > > >> > a > > > > > >> > kernel or ramdisk. Allocation is only transitory, for the > > > > > >> > purposes of > > > > > >> > loading some images and booting. If the boot fails, then all lmb > > > > > >> > allocations go away. > > > > > >> > > > > > > >> > lmb is set up by getting all available memory and then removing > > > > > >> > what > > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > > > > >> > > > > > > >> > lmb reservations have a few flags so that areas of memory can be > > > > > >> > provided with attributes > > > > > >> > > > > > > >> > There are some corner cases...e.g. loading a file does an lmb > > > > > >> > allocation but only for the purpose of avoiding a file being > > > > > >> > loaded > > > > > >> > over U-Boot code/data. The allocation is dropped immediately > > > > > >> > after the > > > > > >> > file is loaded. Within the bootm command, or when using standard > > > > > >> > boot, > > > > > >> > this would be fairly easy to solve. > > > > > >> > > > > > > >> > Linux has renamed lmb to memblock. We should consider doing the > > > > > >> > same. > > > > > >> > > > > > > >> > 3. EFI > > > > > >> > > > > > > >> > EFI has its own memory-allocation tables. > > > > > >> > > > > > > >> > Like lmb, EFI is able to deal with large allocations. But via a > > > > > >> > 'pool' > > > > > >> > function it can also do smaller allocations similar to malloc(), > > > > > >> > although each one uses at least 4KB at present. > > > > > >> > > > > > > >> > EFI allocations do not go away when a boot fails. > > > > > >> > > > > > > >> > With EFI it is possible to add allocations post facto, in which > > > > > >> > case > > > > > >> > they are added to the allocation table just as if the memory was > > > > > >> > allocated with EFI to begin with. > > > > > >> > > > > > > >> > The EFI allocations and the lmb allocations use the same memory, > > > > > >> > so in > > > > > >> > principle could conflict. > > > > > >> > > > > > > >> > EFI allocations are sometimes used to allocate internal U-Boot > > > > > >> > data as > > > > > >> > well, if needed by the EFI app. For example, while > > > > > >> > efi_image_parse() > > > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code > > > > > >> > runs > > > > > >> > in the app context and may need to access the memory after > > > > > >> > U-Boot has > > > > > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a > > > > > >> > new > > > > > >> > mapping as well. > > > > > >> > > > > > > >> > EFI memory has attributes, including what the memory is used for > > > > > >> > (to > > > > > >> > some degree of granularity). See enum efi_memory_type and struct > > > > > >> > efi_mem_desc. In the latter there are also attribute flags - > > > > > >> > whether > > > > > >> > memory is cacheable, etc. > > > > > >> > > > > > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > > > > > >> > believe) that below 4GB that isn't reserved for the > > > > > >> > hardware/system. > > > > > >> > This is meaningless, or at least confusing, on ARM systems. > > > > > >> > > > > > > >> > 4. reservations > > > > > >> > > > > > > >> > It is perhaps worth mentioning a fourth method of memory > > > > > >> > management, > > > > > >> > where U-Boot reserves chunks of memory before relocation (in > > > > > >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the > > > > > >> > malloc() > > > > > >> > region, etc. > > > > > >> > > > > > > >> > > > > > > >> > Problems > > > > > >> > —--- > > > > > >> > > > > > > >> > There are no urgent problems, but here are some things that > > > > > >> > could be improved: > > > > > >> > > > > > > >> > 1. EFI should attach most of it
Re: Proposal: U-Boot memory management
Hi, On Fri, Dec 29, 2023 at 10:52 AM Tom Rini wrote: > > On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > > From: Tom Rini > > > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini : > > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > > > > >> Hi, > > > > >> > > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass > > > > >> wrote: > > > > >> > > > > > >> > Hi, > > > > >> > > > > > >> > This records my thoughts after a discussion with Ilias & Heinrich > > > > >> > re > > > > >> > memory allocation in U-Boot. > > > > >> > > > > > >> > 1. malloc() > > > > >> > > > > > >> > malloc() is used for programmatic memory allocation. It allows > > > > >> > memory > > > > >> > to be freed. It is not designed for very large allocations (e.g. a > > > > >> > 10MB kernel or 100MB ramdisk). > > > > >> > > > > > >> > 2. lmb > > > > >> > > > > > >> > lmb is used for large blocks of memory, such as those needed for a > > > > >> > kernel or ramdisk. Allocation is only transitory, for the purposes > > > > >> > of > > > > >> > loading some images and booting. If the boot fails, then all lmb > > > > >> > allocations go away. > > > > >> > > > > > >> > lmb is set up by getting all available memory and then removing > > > > >> > what > > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > > > >> > > > > > >> > lmb reservations have a few flags so that areas of memory can be > > > > >> > provided with attributes > > > > >> > > > > > >> > There are some corner cases...e.g. loading a file does an lmb > > > > >> > allocation but only for the purpose of avoiding a file being loaded > > > > >> > over U-Boot code/data. The allocation is dropped immediately after > > > > >> > the > > > > >> > file is loaded. Within the bootm command, or when using standard > > > > >> > boot, > > > > >> > this would be fairly easy to solve. > > > > >> > > > > > >> > Linux has renamed lmb to memblock. We should consider doing the > > > > >> > same. > > > > >> > > > > > >> > 3. EFI > > > > >> > > > > > >> > EFI has its own memory-allocation tables. > > > > >> > > > > > >> > Like lmb, EFI is able to deal with large allocations. But via a > > > > >> > 'pool' > > > > >> > function it can also do smaller allocations similar to malloc(), > > > > >> > although each one uses at least 4KB at present. > > > > >> > > > > > >> > EFI allocations do not go away when a boot fails. > > > > >> > > > > > >> > With EFI it is possible to add allocations post facto, in which > > > > >> > case > > > > >> > they are added to the allocation table just as if the memory was > > > > >> > allocated with EFI to begin with. > > > > >> > > > > > >> > The EFI allocations and the lmb allocations use the same memory, > > > > >> > so in > > > > >> > principle could conflict. > > > > >> > > > > > >> > EFI allocations are sometimes used to allocate internal U-Boot > > > > >> > data as > > > > >> > well, if needed by the EFI app. For example, while > > > > >> > efi_image_parse() > > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code > > > > >> > runs > > > > >> > in the app context and may need to access the memory after U-Boot > > > > >> > has > > > > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > > > > >> > mapping as well. > > > > >> > > > > > >> > EFI memory has attributes, including what the memory is used for > > > > >> > (to > > > > >> > some degree of granularity). See enum efi_memory_type and struct > > > > >> > efi_mem_desc. In the latter there are also attribute flags - > > > > >> > whether > > > > >> > memory is cacheable, etc. > > > > >> > > > > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > > > > >> > believe) that below 4GB that isn't reserved for the > > > > >> > hardware/system. > > > > >> > This is meaningless, or at least confusing, on ARM systems. > > > > >> > > > > > >> > 4. reservations > > > > >> > > > > > >> > It is perhaps worth mentioning a fourth method of memory > > > > >> > management, > > > > >> > where U-Boot reserves chunks of memory before relocation (in > > > > >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the > > > > >> > malloc() > > > > >> > region, etc. > > > > >> > > > > > >> > > > > > >> > Problems > > > > >> > —--- > > > > >> > > > > > >> > There are no urgent problems, but here are some things that could > > > > >> > be improved: > > > > >> > > > > > >> > 1. EFI should attach most of its data structures to driver model. > > > > >> > This > > > > >> > work has started, with the partition support, but more effort would > > > > >> > help. This would make it easier to see which memory is related to > > > > >> > devices and which is separate. > > > > >> > > > > > >> > 2. Some drivers do EFI reservations today, whether EFI is used for > > > > >> > bootin
Re: Proposal: U-Boot memory management
On Fri, Dec 29, 2023 at 06:44:15PM +0100, Mark Kettenis wrote: > > Date: Fri, 29 Dec 2023 11:17:44 -0500 > > From: Tom Rini > > > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt wrote: > > > > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini : > > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > > > >> Hi, > > > >> > > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass wrote: > > > >> > > > > >> > Hi, > > > >> > > > > >> > This records my thoughts after a discussion with Ilias & Heinrich re > > > >> > memory allocation in U-Boot. > > > >> > > > > >> > 1. malloc() > > > >> > > > > >> > malloc() is used for programmatic memory allocation. It allows memory > > > >> > to be freed. It is not designed for very large allocations (e.g. a > > > >> > 10MB kernel or 100MB ramdisk). > > > >> > > > > >> > 2. lmb > > > >> > > > > >> > lmb is used for large blocks of memory, such as those needed for a > > > >> > kernel or ramdisk. Allocation is only transitory, for the purposes of > > > >> > loading some images and booting. If the boot fails, then all lmb > > > >> > allocations go away. > > > >> > > > > >> > lmb is set up by getting all available memory and then removing what > > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > > >> > > > > >> > lmb reservations have a few flags so that areas of memory can be > > > >> > provided with attributes > > > >> > > > > >> > There are some corner cases...e.g. loading a file does an lmb > > > >> > allocation but only for the purpose of avoiding a file being loaded > > > >> > over U-Boot code/data. The allocation is dropped immediately after > > > >> > the > > > >> > file is loaded. Within the bootm command, or when using standard > > > >> > boot, > > > >> > this would be fairly easy to solve. > > > >> > > > > >> > Linux has renamed lmb to memblock. We should consider doing the same. > > > >> > > > > >> > 3. EFI > > > >> > > > > >> > EFI has its own memory-allocation tables. > > > >> > > > > >> > Like lmb, EFI is able to deal with large allocations. But via a > > > >> > 'pool' > > > >> > function it can also do smaller allocations similar to malloc(), > > > >> > although each one uses at least 4KB at present. > > > >> > > > > >> > EFI allocations do not go away when a boot fails. > > > >> > > > > >> > With EFI it is possible to add allocations post facto, in which case > > > >> > they are added to the allocation table just as if the memory was > > > >> > allocated with EFI to begin with. > > > >> > > > > >> > The EFI allocations and the lmb allocations use the same memory, so > > > >> > in > > > >> > principle could conflict. > > > >> > > > > >> > EFI allocations are sometimes used to allocate internal U-Boot data > > > >> > as > > > >> > well, if needed by the EFI app. For example, while efi_image_parse() > > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > > > >> > in the app context and may need to access the memory after U-Boot has > > > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > > > >> > mapping as well. > > > >> > > > > >> > EFI memory has attributes, including what the memory is used for (to > > > >> > some degree of granularity). See enum efi_memory_type and struct > > > >> > efi_mem_desc. In the latter there are also attribute flags - whether > > > >> > memory is cacheable, etc. > > > >> > > > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > > > >> > believe) that below 4GB that isn't reserved for the hardware/system. > > > >> > This is meaningless, or at least confusing, on ARM systems. > > > >> > > > > >> > 4. reservations > > > >> > > > > >> > It is perhaps worth mentioning a fourth method of memory management, > > > >> > where U-Boot reserves chunks of memory before relocation (in > > > >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > > > >> > region, etc. > > > >> > > > > >> > > > > >> > Problems > > > >> > —--- > > > >> > > > > >> > There are no urgent problems, but here are some things that could be > > > >> > improved: > > > >> > > > > >> > 1. EFI should attach most of its data structures to driver model. > > > >> > This > > > >> > work has started, with the partition support, but more effort would > > > >> > help. This would make it easier to see which memory is related to > > > >> > devices and which is separate. > > > >> > > > > >> > 2. Some drivers do EFI reservations today, whether EFI is used for > > > >> > booting or not (e.g. rockchip video rk_vop_probe()). > > > >> > > > > >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > > > >> > armv8's struct mm_region) to EFI ones. > > > >> > > > > >> > 4. EFI duplicates some code from bootm, some of which relates to > > > >> > memory allocation (e.g. FDT fixup). > > > >> > > > > >> > 5. EFI code is used even if EFI is never used to boot > > > >> > > > > >> > 6. EFI allocations can result in the same memory bein
Re: Proposal: U-Boot memory management
On Fri, Dec 29, 2023 at 06:30:43PM +0100, Heinrich Schuchardt wrote: > On 12/29/23 18:21, Tom Rini wrote: > > On Fri, Dec 29, 2023 at 06:09:44PM +0100, Heinrich Schuchardt wrote: > > > On 12/29/23 17:47, Tom Rini wrote: > > > > On Fri, Dec 29, 2023 at 05:42:17PM +0100, Heinrich Schuchardt wrote: > > > > > On 12/20/23 20:12, Tom Rini wrote: > > > > > > On Tue, Dec 19, 2023 at 09:15:21PM -0700, Simon Glass wrote: > > > > > > > Hi, > > > > > > > > > > > > > > On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: > > > > > > > > > > > > > > > > On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini > > > > > > > > > : > > > > > > > > > > On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini > > > > > > > > > > > : > > > > > > > > > > > > On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich > > > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini > > > > > > > > > > > > > : > > > > > > > > > > > > > > On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich > > > > > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom > > > > > > > > > > > > > > > Rini : > > > > > > > > > > > > > > > > On Mon, Dec 18, 2023 at 11:34:16PM +0100, > > > > > > > > > > > > > > > > Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [snip] > > > > > > > > > > > > > > > > > Or take: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > load host 0:1 $c kernel.efi > > > > > > > > > > > > > > > > > load host 0:1 $d initrd.img > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could we ensure that initrd.img is not > > > > > > > > > > > > > > > > > overwriting a part of kernel.efi without > > > > > > > > > > > > > > > > > memory allocation? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Today, invalid checksum as part of some part of > > > > > > > > > > > > > > > > the kernel fails. But > > > > > > > > > > > > > > > > how do we do this tomorrow, are you suggesting > > > > > > > > > > > > > > > > that "load" perform > > > > > > > > > > > > > > > > malloc() in some predefined size? If $c is > > > > > > > > > > > > > > > > below $d and $c + kernel.efi > > > > > > > > > > > > > > > > is now above $d we can throw an error before > > > > > > > > > > > > > > > > trying to load, yes. But > > > > > > > > > > > > > > > > what about: > > > > > > > > > > > > > > > > load host 0:1 $d initrd.img > > > > > > > > > > > > > > > > load host 0:1 $c kernel.efi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In that case (which is only marginally > > > > > > > > > > > > > > > > contrived, the more real case is > > > > > > > > > > > > > > > > loading device tree in to unexpectedly large > > > > > > > > > > > > > > > > ramdisk because someone > > > > > > > > > > > > > > > > didn't understand the general advice on why > > > > > > > > > > > > > > > > device tree is lower than > > > > > > > > > > > > > > > > ramdisk address) I'm fine with an error that > > > > > > > > > > > > > > > > amounts to "you just > > > > > > > > > > > > > > > > corrupted another allocation" and then "fail, > > > > > > > > > > > > > > > > reset the board" or so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Our current malloc library cannot manage the > > > > > > > > > > > > > > > complete memory. We need a library like lmb which > > > > > > > > > > > > > > > should also cover the memory management that we > > > > > > > > > > > > > > > currently have in lib/efi/efi_memory.c. This must > > > > > > > > > > > > > > > include a memory type attribute for usage in the > > > > > > > > > > > > > > > GetMemoryMap() service. A management on page > > > > > > > > > > > > > > > level seems sufficient. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The load command should permanently allocate > > > > > > > > > > > > > > > memory in that lmb+ library. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We need an unload command to free the memory if > > > > > > > > > > > > > > > we want to reuse the memory or we might let the > > > > > > > > > > > > > > > load comand free the memory if exactly the same > > > > > > > > > > > > > > > start address is reused. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Our current way of loading things in to memory does > > > > > > > > > > > > > > not handle the case > > > > > > > > > > > > > > I described, yes. How would what you're proposing >
Re: Proposal: U-Boot memory management
> Date: Fri, 29 Dec 2023 11:17:44 -0500 > From: Tom Rini > > On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt wrote: > > > > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini : > > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > > >> Hi, > > >> > > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass wrote: > > >> > > > >> > Hi, > > >> > > > >> > This records my thoughts after a discussion with Ilias & Heinrich re > > >> > memory allocation in U-Boot. > > >> > > > >> > 1. malloc() > > >> > > > >> > malloc() is used for programmatic memory allocation. It allows memory > > >> > to be freed. It is not designed for very large allocations (e.g. a > > >> > 10MB kernel or 100MB ramdisk). > > >> > > > >> > 2. lmb > > >> > > > >> > lmb is used for large blocks of memory, such as those needed for a > > >> > kernel or ramdisk. Allocation is only transitory, for the purposes of > > >> > loading some images and booting. If the boot fails, then all lmb > > >> > allocations go away. > > >> > > > >> > lmb is set up by getting all available memory and then removing what > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > >> > > > >> > lmb reservations have a few flags so that areas of memory can be > > >> > provided with attributes > > >> > > > >> > There are some corner cases...e.g. loading a file does an lmb > > >> > allocation but only for the purpose of avoiding a file being loaded > > >> > over U-Boot code/data. The allocation is dropped immediately after the > > >> > file is loaded. Within the bootm command, or when using standard boot, > > >> > this would be fairly easy to solve. > > >> > > > >> > Linux has renamed lmb to memblock. We should consider doing the same. > > >> > > > >> > 3. EFI > > >> > > > >> > EFI has its own memory-allocation tables. > > >> > > > >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > > >> > function it can also do smaller allocations similar to malloc(), > > >> > although each one uses at least 4KB at present. > > >> > > > >> > EFI allocations do not go away when a boot fails. > > >> > > > >> > With EFI it is possible to add allocations post facto, in which case > > >> > they are added to the allocation table just as if the memory was > > >> > allocated with EFI to begin with. > > >> > > > >> > The EFI allocations and the lmb allocations use the same memory, so in > > >> > principle could conflict. > > >> > > > >> > EFI allocations are sometimes used to allocate internal U-Boot data as > > >> > well, if needed by the EFI app. For example, while efi_image_parse() > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > > >> > in the app context and may need to access the memory after U-Boot has > > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > > >> > mapping as well. > > >> > > > >> > EFI memory has attributes, including what the memory is used for (to > > >> > some degree of granularity). See enum efi_memory_type and struct > > >> > efi_mem_desc. In the latter there are also attribute flags - whether > > >> > memory is cacheable, etc. > > >> > > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > > >> > believe) that below 4GB that isn't reserved for the hardware/system. > > >> > This is meaningless, or at least confusing, on ARM systems. > > >> > > > >> > 4. reservations > > >> > > > >> > It is perhaps worth mentioning a fourth method of memory management, > > >> > where U-Boot reserves chunks of memory before relocation (in > > >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > > >> > region, etc. > > >> > > > >> > > > >> > Problems > > >> > —--- > > >> > > > >> > There are no urgent problems, but here are some things that could be > > >> > improved: > > >> > > > >> > 1. EFI should attach most of its data structures to driver model. This > > >> > work has started, with the partition support, but more effort would > > >> > help. This would make it easier to see which memory is related to > > >> > devices and which is separate. > > >> > > > >> > 2. Some drivers do EFI reservations today, whether EFI is used for > > >> > booting or not (e.g. rockchip video rk_vop_probe()). > > >> > > > >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > > >> > armv8's struct mm_region) to EFI ones. > > >> > > > >> > 4. EFI duplicates some code from bootm, some of which relates to > > >> > memory allocation (e.g. FDT fixup). > > >> > > > >> > 5. EFI code is used even if EFI is never used to boot > > >> > > > >> > 6. EFI allocations can result in the same memory being used as has > > >> > already been allocated by lmb. Users may load files which overwrite > > >> > memory allocated by EFI. > > >> > > >> 7. We need to support doing an allocation when a file is loaded (to > > >> ensure files do not overlap), without making it too difficult to load > > >> multiple files to the same place, if desired. > > >> > > >
Re: Proposal: U-Boot memory management
On 12/29/23 18:21, Tom Rini wrote: On Fri, Dec 29, 2023 at 06:09:44PM +0100, Heinrich Schuchardt wrote: On 12/29/23 17:47, Tom Rini wrote: On Fri, Dec 29, 2023 at 05:42:17PM +0100, Heinrich Schuchardt wrote: On 12/20/23 20:12, Tom Rini wrote: On Tue, Dec 19, 2023 at 09:15:21PM -0700, Simon Glass wrote: Hi, On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: [snip] Or take: load host 0:1 $c kernel.efi load host 0:1 $d initrd.img How could we ensure that initrd.img is not overwriting a part of kernel.efi without memory allocation? Today, invalid checksum as part of some part of the kernel fails. But how do we do this tomorrow, are you suggesting that "load" perform malloc() in some predefined size? If $c is below $d and $c + kernel.efi is now above $d we can throw an error before trying to load, yes. But what about: load host 0:1 $d initrd.img load host 0:1 $c kernel.efi In that case (which is only marginally contrived, the more real case is loading device tree in to unexpectedly large ramdisk because someone didn't understand the general advice on why device tree is lower than ramdisk address) I'm fine with an error that amounts to "you just corrupted another allocation" and then "fail, reset the board" or so. Our current malloc library cannot manage the complete memory. We need a library like lmb which should also cover the memory management that we currently have in lib/efi/efi_memory.c. This must include a memory type attribute for usage in the GetMemoryMap() service. A management on page level seems sufficient. The load command should permanently allocate memory in that lmb+ library. We need an unload command to free the memory if we want to reuse the memory or we might let the load comand free the memory if exactly the same start address is reused. Our current way of loading things in to memory does not handle the case I described, yes. How would what you're proposing handle it? If the load command has to allocate memory for the image and that allocation is kept, any attempt to allocate overlapping memory would fail. So you're saying that the load command has to pre-allocate memory? Or as it goes? If the latter, in what size chunks? This starts to get at what Simon was talking about with respect to memory fragmentation. Which to be clear is a problem we have today, we just let things overlap and hope something later catches an incorrect checksum. I don't want to replace the malloc library which handles large numbets of allocations. I'm confused. The normal malloc library is not involved with current image loading, it's direct to memory (with some attempts at sanity checking by lmb). Are you proposing a different allocator with malloc/free like behavior? If so, please outline how it will determine pool size, and how we'll use it to load thing to memory. All memory below the stack needs to be managed. Malloc uses a small memory area (a few MiB) above the stack. That's a rather huge change for how U-Boot works. Closing the eyes when the user loads multiple files does not solve the fragmentation problem. Yes. I'm only noting that today we just ignore the problem and sometimes catch it via checksums. Fragmentation only happens if we have many concurrent allocations. In EFI we are allocating top down. The number of concurrent allocations is low. Typically a few dozen at most. After terminating an application these should be freed again. OK, so are you saying that we would no longer be loading _to_ a location in memory and instead just be saying "load this thing" and picking where dynamically? Both preassigned and allocator assigned adresses are compatible with memory management. Architectures and binaries have different requirements. On riscv64 you can load Linux kernel, initrd, fdt anywhere. We don't need predefined addresses there. Other architectures have restrictions. Yes, 64 bit architecture tend to only have alignment requirements while 32bit architectures have both alignment requirements and some memory window requirement. Whatever we implement here needs to handle both cases. When loading a file from a file system we know the filesize beforehand. So allocation is trivial. The loady command currently does not use the offered size information but could do so. We should be using that information to make sure we don't overwrite U-Boot itself, but I don't recall how exactly
Re: Proposal: U-Boot memory management
On Fri, Dec 29, 2023 at 06:09:44PM +0100, Heinrich Schuchardt wrote: > On 12/29/23 17:47, Tom Rini wrote: > > On Fri, Dec 29, 2023 at 05:42:17PM +0100, Heinrich Schuchardt wrote: > > > On 12/20/23 20:12, Tom Rini wrote: > > > > On Tue, Dec 19, 2023 at 09:15:21PM -0700, Simon Glass wrote: > > > > > Hi, > > > > > > > > > > On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: > > > > > > > > > > > > On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini > > > > > > > : > > > > > > > > On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini > > > > > > > > > : > > > > > > > > > > On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini > > > > > > > > > > > : > > > > > > > > > > > > On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich > > > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini > > > > > > > > > > > > > : > > > > > > > > > > > > > > On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich > > > > > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > [snip] > > > > > > > > > > > > > > > Or take: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > load host 0:1 $c kernel.efi > > > > > > > > > > > > > > > load host 0:1 $d initrd.img > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could we ensure that initrd.img is not > > > > > > > > > > > > > > > overwriting a part of kernel.efi without memory > > > > > > > > > > > > > > > allocation? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Today, invalid checksum as part of some part of the > > > > > > > > > > > > > > kernel fails. But > > > > > > > > > > > > > > how do we do this tomorrow, are you suggesting that > > > > > > > > > > > > > > "load" perform > > > > > > > > > > > > > > malloc() in some predefined size? If $c is below $d > > > > > > > > > > > > > > and $c + kernel.efi > > > > > > > > > > > > > > is now above $d we can throw an error before trying > > > > > > > > > > > > > > to load, yes. But > > > > > > > > > > > > > > what about: > > > > > > > > > > > > > > load host 0:1 $d initrd.img > > > > > > > > > > > > > > load host 0:1 $c kernel.efi > > > > > > > > > > > > > > > > > > > > > > > > > > > > In that case (which is only marginally contrived, > > > > > > > > > > > > > > the more real case is > > > > > > > > > > > > > > loading device tree in to unexpectedly large > > > > > > > > > > > > > > ramdisk because someone > > > > > > > > > > > > > > didn't understand the general advice on why device > > > > > > > > > > > > > > tree is lower than > > > > > > > > > > > > > > ramdisk address) I'm fine with an error that > > > > > > > > > > > > > > amounts to "you just > > > > > > > > > > > > > > corrupted another allocation" and then "fail, reset > > > > > > > > > > > > > > the board" or so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Our current malloc library cannot manage the complete > > > > > > > > > > > > > memory. We need a library like lmb which should also > > > > > > > > > > > > > cover the memory management that we currently have in > > > > > > > > > > > > > lib/efi/efi_memory.c. This must include a memory type > > > > > > > > > > > > > attribute for usage in the GetMemoryMap() service. A > > > > > > > > > > > > > management on page level seems sufficient. > > > > > > > > > > > > > > > > > > > > > > > > > > The load command should permanently allocate memory > > > > > > > > > > > > > in that lmb+ library. > > > > > > > > > > > > > > > > > > > > > > > > > > We need an unload command to free the memory if we > > > > > > > > > > > > > want to reuse the memory or we might let the load > > > > > > > > > > > > > comand free the memory if exactly the same start > > > > > > > > > > > > > address is reused. > > > > > > > > > > > > > > > > > > > > > > > > Our current way of loading things in to memory does not > > > > > > > > > > > > handle the case > > > > > > > > > > > > I described, yes. How would what you're proposing > > > > > > > > > > > > handle it? > > > > > > > > > > > > > > > > > > > > > > If the load command has to allocate memory for the image > > > > > > > > > > > and that allocation is kept, any attempt to allocate > > > > > > > > > > > overlapping memory would fail. > > > > > > > > > > > > > > > > > > > > So you're saying that the load command has to pre-allocate > > > > > > > > > > memory? Or as > > > > > > > > > > it goes? If the latter, in what size chunks? This starts to > > > > > > > > > >
Re: Proposal: U-Boot memory management
On 12/29/23 17:47, Tom Rini wrote: On Fri, Dec 29, 2023 at 05:42:17PM +0100, Heinrich Schuchardt wrote: On 12/20/23 20:12, Tom Rini wrote: On Tue, Dec 19, 2023 at 09:15:21PM -0700, Simon Glass wrote: Hi, On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: [snip] Or take: load host 0:1 $c kernel.efi load host 0:1 $d initrd.img How could we ensure that initrd.img is not overwriting a part of kernel.efi without memory allocation? Today, invalid checksum as part of some part of the kernel fails. But how do we do this tomorrow, are you suggesting that "load" perform malloc() in some predefined size? If $c is below $d and $c + kernel.efi is now above $d we can throw an error before trying to load, yes. But what about: load host 0:1 $d initrd.img load host 0:1 $c kernel.efi In that case (which is only marginally contrived, the more real case is loading device tree in to unexpectedly large ramdisk because someone didn't understand the general advice on why device tree is lower than ramdisk address) I'm fine with an error that amounts to "you just corrupted another allocation" and then "fail, reset the board" or so. Our current malloc library cannot manage the complete memory. We need a library like lmb which should also cover the memory management that we currently have in lib/efi/efi_memory.c. This must include a memory type attribute for usage in the GetMemoryMap() service. A management on page level seems sufficient. The load command should permanently allocate memory in that lmb+ library. We need an unload command to free the memory if we want to reuse the memory or we might let the load comand free the memory if exactly the same start address is reused. Our current way of loading things in to memory does not handle the case I described, yes. How would what you're proposing handle it? If the load command has to allocate memory for the image and that allocation is kept, any attempt to allocate overlapping memory would fail. So you're saying that the load command has to pre-allocate memory? Or as it goes? If the latter, in what size chunks? This starts to get at what Simon was talking about with respect to memory fragmentation. Which to be clear is a problem we have today, we just let things overlap and hope something later catches an incorrect checksum. I don't want to replace the malloc library which handles large numbets of allocations. I'm confused. The normal malloc library is not involved with current image loading, it's direct to memory (with some attempts at sanity checking by lmb). Are you proposing a different allocator with malloc/free like behavior? If so, please outline how it will determine pool size, and how we'll use it to load thing to memory. All memory below the stack needs to be managed. Malloc uses a small memory area (a few MiB) above the stack. That's a rather huge change for how U-Boot works. Closing the eyes when the user loads multiple files does not solve the fragmentation problem. Yes. I'm only noting that today we just ignore the problem and sometimes catch it via checksums. Fragmentation only happens if we have many concurrent allocations. In EFI we are allocating top down. The number of concurrent allocations is low. Typically a few dozen at most. After terminating an application these should be freed again. OK, so are you saying that we would no longer be loading _to_ a location in memory and instead just be saying "load this thing" and picking where dynamically? Both preassigned and allocator assigned adresses are compatible with memory management. Architectures and binaries have different requirements. On riscv64 you can load Linux kernel, initrd, fdt anywhere. We don't need predefined addresses there. Other architectures have restrictions. Yes, 64 bit architecture tend to only have alignment requirements while 32bit architectures have both alignment requirements and some memory window requirement. Whatever we implement here needs to handle both cases. When loading a file from a file system we know the filesize beforehand. So allocation is trivial. The loady command currently does not use the offered size information but could do so. We should be using that information to make sure we don't overwrite U-Boot itself, but I don't recall how exactly we handle it today off-hand. If the user issues multiple load commands, he can overwrite previous files
Re: Proposal: U-Boot memory management
On Fri, Dec 29, 2023 at 05:42:17PM +0100, Heinrich Schuchardt wrote: > On 12/20/23 20:12, Tom Rini wrote: > > On Tue, Dec 19, 2023 at 09:15:21PM -0700, Simon Glass wrote: > > > Hi, > > > > > > On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: > > > > > > > > On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini > > > > > : > > > > > > On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini > > > > > > > : > > > > > > > > On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini > > > > > > > > > : > > > > > > > > > > On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini > > > > > > > > > > > : > > > > > > > > > > > > On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich > > > > > > > > > > > > Schuchardt wrote: > > > > > > > > > > > > > > > > > > > > > > > > [snip] > > > > > > > > > > > > > Or take: > > > > > > > > > > > > > > > > > > > > > > > > > > load host 0:1 $c kernel.efi > > > > > > > > > > > > > load host 0:1 $d initrd.img > > > > > > > > > > > > > > > > > > > > > > > > > > How could we ensure that initrd.img is not > > > > > > > > > > > > > overwriting a part of kernel.efi without memory > > > > > > > > > > > > > allocation? > > > > > > > > > > > > > > > > > > > > > > > > Today, invalid checksum as part of some part of the > > > > > > > > > > > > kernel fails. But > > > > > > > > > > > > how do we do this tomorrow, are you suggesting that > > > > > > > > > > > > "load" perform > > > > > > > > > > > > malloc() in some predefined size? If $c is below $d and > > > > > > > > > > > > $c + kernel.efi > > > > > > > > > > > > is now above $d we can throw an error before trying to > > > > > > > > > > > > load, yes. But > > > > > > > > > > > > what about: > > > > > > > > > > > > load host 0:1 $d initrd.img > > > > > > > > > > > > load host 0:1 $c kernel.efi > > > > > > > > > > > > > > > > > > > > > > > > In that case (which is only marginally contrived, the > > > > > > > > > > > > more real case is > > > > > > > > > > > > loading device tree in to unexpectedly large ramdisk > > > > > > > > > > > > because someone > > > > > > > > > > > > didn't understand the general advice on why device tree > > > > > > > > > > > > is lower than > > > > > > > > > > > > ramdisk address) I'm fine with an error that amounts to > > > > > > > > > > > > "you just > > > > > > > > > > > > corrupted another allocation" and then "fail, reset the > > > > > > > > > > > > board" or so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Our current malloc library cannot manage the complete > > > > > > > > > > > memory. We need a library like lmb which should also > > > > > > > > > > > cover the memory management that we currently have in > > > > > > > > > > > lib/efi/efi_memory.c. This must include a memory type > > > > > > > > > > > attribute for usage in the GetMemoryMap() service. A > > > > > > > > > > > management on page level seems sufficient. > > > > > > > > > > > > > > > > > > > > > > The load command should permanently allocate memory in > > > > > > > > > > > that lmb+ library. > > > > > > > > > > > > > > > > > > > > > > We need an unload command to free the memory if we want > > > > > > > > > > > to reuse the memory or we might let the load comand free > > > > > > > > > > > the memory if exactly the same start address is reused. > > > > > > > > > > > > > > > > > > > > Our current way of loading things in to memory does not > > > > > > > > > > handle the case > > > > > > > > > > I described, yes. How would what you're proposing handle it? > > > > > > > > > > > > > > > > > > If the load command has to allocate memory for the image and > > > > > > > > > that allocation is kept, any attempt to allocate overlapping > > > > > > > > > memory would fail. > > > > > > > > > > > > > > > > So you're saying that the load command has to pre-allocate > > > > > > > > memory? Or as > > > > > > > > it goes? If the latter, in what size chunks? This starts to get > > > > > > > > at what > > > > > > > > Simon was talking about with respect to memory fragmentation. > > > > > > > > Which to > > > > > > > > be clear is a problem we have today, we just let things overlap > > > > > > > > and hope > > > > > > > > something later catches an incorrect checksum. > > > > > > > > > > > > > > > > > > > > > > I don't want to replace the malloc library which handles large > > > > > > > numbets of allocations. > > > > > > > > > > > > I'm confused. The normal malloc library is not involved with current > > > > > > imag
Re: Proposal: U-Boot memory management
On 12/20/23 20:12, Tom Rini wrote: On Tue, Dec 19, 2023 at 09:15:21PM -0700, Simon Glass wrote: Hi, On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: [snip] Or take: load host 0:1 $c kernel.efi load host 0:1 $d initrd.img How could we ensure that initrd.img is not overwriting a part of kernel.efi without memory allocation? Today, invalid checksum as part of some part of the kernel fails. But how do we do this tomorrow, are you suggesting that "load" perform malloc() in some predefined size? If $c is below $d and $c + kernel.efi is now above $d we can throw an error before trying to load, yes. But what about: load host 0:1 $d initrd.img load host 0:1 $c kernel.efi In that case (which is only marginally contrived, the more real case is loading device tree in to unexpectedly large ramdisk because someone didn't understand the general advice on why device tree is lower than ramdisk address) I'm fine with an error that amounts to "you just corrupted another allocation" and then "fail, reset the board" or so. Our current malloc library cannot manage the complete memory. We need a library like lmb which should also cover the memory management that we currently have in lib/efi/efi_memory.c. This must include a memory type attribute for usage in the GetMemoryMap() service. A management on page level seems sufficient. The load command should permanently allocate memory in that lmb+ library. We need an unload command to free the memory if we want to reuse the memory or we might let the load comand free the memory if exactly the same start address is reused. Our current way of loading things in to memory does not handle the case I described, yes. How would what you're proposing handle it? If the load command has to allocate memory for the image and that allocation is kept, any attempt to allocate overlapping memory would fail. So you're saying that the load command has to pre-allocate memory? Or as it goes? If the latter, in what size chunks? This starts to get at what Simon was talking about with respect to memory fragmentation. Which to be clear is a problem we have today, we just let things overlap and hope something later catches an incorrect checksum. I don't want to replace the malloc library which handles large numbets of allocations. I'm confused. The normal malloc library is not involved with current image loading, it's direct to memory (with some attempts at sanity checking by lmb). Are you proposing a different allocator with malloc/free like behavior? If so, please outline how it will determine pool size, and how we'll use it to load thing to memory. All memory below the stack needs to be managed. Malloc uses a small memory area (a few MiB) above the stack. That's a rather huge change for how U-Boot works. Closing the eyes when the user loads multiple files does not solve the fragmentation problem. Yes. I'm only noting that today we just ignore the problem and sometimes catch it via checksums. Fragmentation only happens if we have many concurrent allocations. In EFI we are allocating top down. The number of concurrent allocations is low. Typically a few dozen at most. After terminating an application these should be freed again. OK, so are you saying that we would no longer be loading _to_ a location in memory and instead just be saying "load this thing" and picking where dynamically? Both preassigned and allocator assigned adresses are compatible with memory management. Architectures and binaries have different requirements. On riscv64 you can load Linux kernel, initrd, fdt anywhere. We don't need predefined addresses there. Other architectures have restrictions. Yes, 64 bit architecture tend to only have alignment requirements while 32bit architectures have both alignment requirements and some memory window requirement. Whatever we implement here needs to handle both cases. When loading a file from a file system we know the filesize beforehand. So allocation is trivial. The loady command currently does not use the offered size information but could do so. We should be using that information to make sure we don't overwrite U-Boot itself, but I don't recall how exactly we handle it today off-hand. If the user issues multiple load commands, he can overwrite previous files. Then it sounds like we lost one benefit of all of this overhead. During boot command execution I gue
Re: Proposal: U-Boot memory management
On Fri, Dec 29, 2023 at 05:05:17PM +0100, Heinrich Schuchardt wrote: > > > Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini : > >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > >> Hi, > >> > >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass wrote: > >> > > >> > Hi, > >> > > >> > This records my thoughts after a discussion with Ilias & Heinrich re > >> > memory allocation in U-Boot. > >> > > >> > 1. malloc() > >> > > >> > malloc() is used for programmatic memory allocation. It allows memory > >> > to be freed. It is not designed for very large allocations (e.g. a > >> > 10MB kernel or 100MB ramdisk). > >> > > >> > 2. lmb > >> > > >> > lmb is used for large blocks of memory, such as those needed for a > >> > kernel or ramdisk. Allocation is only transitory, for the purposes of > >> > loading some images and booting. If the boot fails, then all lmb > >> > allocations go away. > >> > > >> > lmb is set up by getting all available memory and then removing what > >> > is used by U-Boot (code, data, malloc() space, etc.) > >> > > >> > lmb reservations have a few flags so that areas of memory can be > >> > provided with attributes > >> > > >> > There are some corner cases...e.g. loading a file does an lmb > >> > allocation but only for the purpose of avoiding a file being loaded > >> > over U-Boot code/data. The allocation is dropped immediately after the > >> > file is loaded. Within the bootm command, or when using standard boot, > >> > this would be fairly easy to solve. > >> > > >> > Linux has renamed lmb to memblock. We should consider doing the same. > >> > > >> > 3. EFI > >> > > >> > EFI has its own memory-allocation tables. > >> > > >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > >> > function it can also do smaller allocations similar to malloc(), > >> > although each one uses at least 4KB at present. > >> > > >> > EFI allocations do not go away when a boot fails. > >> > > >> > With EFI it is possible to add allocations post facto, in which case > >> > they are added to the allocation table just as if the memory was > >> > allocated with EFI to begin with. > >> > > >> > The EFI allocations and the lmb allocations use the same memory, so in > >> > principle could conflict. > >> > > >> > EFI allocations are sometimes used to allocate internal U-Boot data as > >> > well, if needed by the EFI app. For example, while efi_image_parse() > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > >> > in the app context and may need to access the memory after U-Boot has > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > >> > mapping as well. > >> > > >> > EFI memory has attributes, including what the memory is used for (to > >> > some degree of granularity). See enum efi_memory_type and struct > >> > efi_mem_desc. In the latter there are also attribute flags - whether > >> > memory is cacheable, etc. > >> > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > >> > believe) that below 4GB that isn't reserved for the hardware/system. > >> > This is meaningless, or at least confusing, on ARM systems. > >> > > >> > 4. reservations > >> > > >> > It is perhaps worth mentioning a fourth method of memory management, > >> > where U-Boot reserves chunks of memory before relocation (in > >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > >> > region, etc. > >> > > >> > > >> > Problems > >> > —--- > >> > > >> > There are no urgent problems, but here are some things that could be > >> > improved: > >> > > >> > 1. EFI should attach most of its data structures to driver model. This > >> > work has started, with the partition support, but more effort would > >> > help. This would make it easier to see which memory is related to > >> > devices and which is separate. > >> > > >> > 2. Some drivers do EFI reservations today, whether EFI is used for > >> > booting or not (e.g. rockchip video rk_vop_probe()). > >> > > >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > >> > armv8's struct mm_region) to EFI ones. > >> > > >> > 4. EFI duplicates some code from bootm, some of which relates to > >> > memory allocation (e.g. FDT fixup). > >> > > >> > 5. EFI code is used even if EFI is never used to boot > >> > > >> > 6. EFI allocations can result in the same memory being used as has > >> > already been allocated by lmb. Users may load files which overwrite > >> > memory allocated by EFI. > >> > >> 7. We need to support doing an allocation when a file is loaded (to > >> ensure files do not overlap), without making it too difficult to load > >> multiple files to the same place, if desired. > >> > >> > > >> > > >> > Lifetime > >> > > >> > > >> > We have three different memory allocators with different purposes. Can > >> > we unify them a little? > >> > > >> > Within U-Boot: > >> > - malloc() space lives forever > >> > - lmb lives while setting out images for booting > >>
Re: Proposal: U-Boot memory management
Am 29. Dezember 2023 16:43:07 MEZ schrieb Tom Rini : >On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: >> Hi, >> >> On Sat, Dec 16, 2023 at 6:01 PM Simon Glass wrote: >> > >> > Hi, >> > >> > This records my thoughts after a discussion with Ilias & Heinrich re >> > memory allocation in U-Boot. >> > >> > 1. malloc() >> > >> > malloc() is used for programmatic memory allocation. It allows memory >> > to be freed. It is not designed for very large allocations (e.g. a >> > 10MB kernel or 100MB ramdisk). >> > >> > 2. lmb >> > >> > lmb is used for large blocks of memory, such as those needed for a >> > kernel or ramdisk. Allocation is only transitory, for the purposes of >> > loading some images and booting. If the boot fails, then all lmb >> > allocations go away. >> > >> > lmb is set up by getting all available memory and then removing what >> > is used by U-Boot (code, data, malloc() space, etc.) >> > >> > lmb reservations have a few flags so that areas of memory can be >> > provided with attributes >> > >> > There are some corner cases...e.g. loading a file does an lmb >> > allocation but only for the purpose of avoiding a file being loaded >> > over U-Boot code/data. The allocation is dropped immediately after the >> > file is loaded. Within the bootm command, or when using standard boot, >> > this would be fairly easy to solve. >> > >> > Linux has renamed lmb to memblock. We should consider doing the same. >> > >> > 3. EFI >> > >> > EFI has its own memory-allocation tables. >> > >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' >> > function it can also do smaller allocations similar to malloc(), >> > although each one uses at least 4KB at present. >> > >> > EFI allocations do not go away when a boot fails. >> > >> > With EFI it is possible to add allocations post facto, in which case >> > they are added to the allocation table just as if the memory was >> > allocated with EFI to begin with. >> > >> > The EFI allocations and the lmb allocations use the same memory, so in >> > principle could conflict. >> > >> > EFI allocations are sometimes used to allocate internal U-Boot data as >> > well, if needed by the EFI app. For example, while efi_image_parse() >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs >> > in the app context and may need to access the memory after U-Boot has >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new >> > mapping as well. >> > >> > EFI memory has attributes, including what the memory is used for (to >> > some degree of granularity). See enum efi_memory_type and struct >> > efi_mem_desc. In the latter there are also attribute flags - whether >> > memory is cacheable, etc. >> > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I >> > believe) that below 4GB that isn't reserved for the hardware/system. >> > This is meaningless, or at least confusing, on ARM systems. >> > >> > 4. reservations >> > >> > It is perhaps worth mentioning a fourth method of memory management, >> > where U-Boot reserves chunks of memory before relocation (in >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() >> > region, etc. >> > >> > >> > Problems >> > —--- >> > >> > There are no urgent problems, but here are some things that could be >> > improved: >> > >> > 1. EFI should attach most of its data structures to driver model. This >> > work has started, with the partition support, but more effort would >> > help. This would make it easier to see which memory is related to >> > devices and which is separate. >> > >> > 2. Some drivers do EFI reservations today, whether EFI is used for >> > booting or not (e.g. rockchip video rk_vop_probe()). >> > >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. >> > armv8's struct mm_region) to EFI ones. >> > >> > 4. EFI duplicates some code from bootm, some of which relates to >> > memory allocation (e.g. FDT fixup). >> > >> > 5. EFI code is used even if EFI is never used to boot >> > >> > 6. EFI allocations can result in the same memory being used as has >> > already been allocated by lmb. Users may load files which overwrite >> > memory allocated by EFI. >> >> 7. We need to support doing an allocation when a file is loaded (to >> ensure files do not overlap), without making it too difficult to load >> multiple files to the same place, if desired. >> >> > >> > >> > Lifetime >> > >> > >> > We have three different memory allocators with different purposes. Can >> > we unify them a little? >> > >> > Within U-Boot: >> > - malloc() space lives forever >> > - lmb lives while setting out images for booting >> > - EFI (mostly) lives while booting an EFI app >> > >> > In practice, EFI is set up early in U-Boot. Some of this is necessary, >> > some not. EFI allocations stay around forever. This works OK since >> > large allocations are normally not done in EFI, so memory isn't really >> > consumed to any great degree by
Re: Proposal: U-Boot memory management
On Fri, Dec 29, 2023 at 05:36:09AM +, Simon Glass wrote: > Hi, > > On Sat, Dec 16, 2023 at 6:01 PM Simon Glass wrote: > > > > Hi, > > > > This records my thoughts after a discussion with Ilias & Heinrich re > > memory allocation in U-Boot. > > > > 1. malloc() > > > > malloc() is used for programmatic memory allocation. It allows memory > > to be freed. It is not designed for very large allocations (e.g. a > > 10MB kernel or 100MB ramdisk). > > > > 2. lmb > > > > lmb is used for large blocks of memory, such as those needed for a > > kernel or ramdisk. Allocation is only transitory, for the purposes of > > loading some images and booting. If the boot fails, then all lmb > > allocations go away. > > > > lmb is set up by getting all available memory and then removing what > > is used by U-Boot (code, data, malloc() space, etc.) > > > > lmb reservations have a few flags so that areas of memory can be > > provided with attributes > > > > There are some corner cases...e.g. loading a file does an lmb > > allocation but only for the purpose of avoiding a file being loaded > > over U-Boot code/data. The allocation is dropped immediately after the > > file is loaded. Within the bootm command, or when using standard boot, > > this would be fairly easy to solve. > > > > Linux has renamed lmb to memblock. We should consider doing the same. > > > > 3. EFI > > > > EFI has its own memory-allocation tables. > > > > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > > function it can also do smaller allocations similar to malloc(), > > although each one uses at least 4KB at present. > > > > EFI allocations do not go away when a boot fails. > > > > With EFI it is possible to add allocations post facto, in which case > > they are added to the allocation table just as if the memory was > > allocated with EFI to begin with. > > > > The EFI allocations and the lmb allocations use the same memory, so in > > principle could conflict. > > > > EFI allocations are sometimes used to allocate internal U-Boot data as > > well, if needed by the EFI app. For example, while efi_image_parse() > > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > > in the app context and may need to access the memory after U-Boot has > > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > > mapping as well. > > > > EFI memory has attributes, including what the memory is used for (to > > some degree of granularity). See enum efi_memory_type and struct > > efi_mem_desc. In the latter there are also attribute flags - whether > > memory is cacheable, etc. > > > > EFI also has the x86 idea of 'conventional' memory, meaning (I > > believe) that below 4GB that isn't reserved for the hardware/system. > > This is meaningless, or at least confusing, on ARM systems. > > > > 4. reservations > > > > It is perhaps worth mentioning a fourth method of memory management, > > where U-Boot reserves chunks of memory before relocation (in > > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > > region, etc. > > > > > > Problems > > —--- > > > > There are no urgent problems, but here are some things that could be > > improved: > > > > 1. EFI should attach most of its data structures to driver model. This > > work has started, with the partition support, but more effort would > > help. This would make it easier to see which memory is related to > > devices and which is separate. > > > > 2. Some drivers do EFI reservations today, whether EFI is used for > > booting or not (e.g. rockchip video rk_vop_probe()). > > > > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > > armv8's struct mm_region) to EFI ones. > > > > 4. EFI duplicates some code from bootm, some of which relates to > > memory allocation (e.g. FDT fixup). > > > > 5. EFI code is used even if EFI is never used to boot > > > > 6. EFI allocations can result in the same memory being used as has > > already been allocated by lmb. Users may load files which overwrite > > memory allocated by EFI. > > 7. We need to support doing an allocation when a file is loaded (to > ensure files do not overlap), without making it too difficult to load > multiple files to the same place, if desired. > > > > > > > Lifetime > > > > > > We have three different memory allocators with different purposes. Can > > we unify them a little? > > > > Within U-Boot: > > - malloc() space lives forever > > - lmb lives while setting out images for booting > > - EFI (mostly) lives while booting an EFI app > > > > In practice, EFI is set up early in U-Boot. Some of this is necessary, > > some not. EFI allocations stay around forever. This works OK since > > large allocations are normally not done in EFI, so memory isn't really > > consumed to any great degree by the boot process. > > > > What happens to EFI allocations if the app returns? They are still > > present, in case another app is run. This seems fine. > > > > API > > –-- > > Can we
Re: Proposal: U-Boot memory management
Am 29. Dezember 2023 06:36:09 MEZ schrieb Simon Glass : >Hi, > >On Sat, Dec 16, 2023 at 6:01 PM Simon Glass wrote: >> >> Hi, >> >> This records my thoughts after a discussion with Ilias & Heinrich re >> memory allocation in U-Boot. >> >> 1. malloc() >> >> malloc() is used for programmatic memory allocation. It allows memory >> to be freed. It is not designed for very large allocations (e.g. a >> 10MB kernel or 100MB ramdisk). >> >> 2. lmb >> >> lmb is used for large blocks of memory, such as those needed for a >> kernel or ramdisk. Allocation is only transitory, for the purposes of >> loading some images and booting. If the boot fails, then all lmb >> allocations go away. >> >> lmb is set up by getting all available memory and then removing what >> is used by U-Boot (code, data, malloc() space, etc.) >> >> lmb reservations have a few flags so that areas of memory can be >> provided with attributes >> >> There are some corner cases...e.g. loading a file does an lmb >> allocation but only for the purpose of avoiding a file being loaded >> over U-Boot code/data. The allocation is dropped immediately after the >> file is loaded. Within the bootm command, or when using standard boot, >> this would be fairly easy to solve. >> >> Linux has renamed lmb to memblock. We should consider doing the same. >> >> 3. EFI >> >> EFI has its own memory-allocation tables. >> >> Like lmb, EFI is able to deal with large allocations. But via a 'pool' >> function it can also do smaller allocations similar to malloc(), >> although each one uses at least 4KB at present. >> >> EFI allocations do not go away when a boot fails. >> >> With EFI it is possible to add allocations post facto, in which case >> they are added to the allocation table just as if the memory was >> allocated with EFI to begin with. >> >> The EFI allocations and the lmb allocations use the same memory, so in >> principle could conflict. >> >> EFI allocations are sometimes used to allocate internal U-Boot data as >> well, if needed by the EFI app. For example, while efi_image_parse() >> uses malloc(), efi_var_mem.c uses EFI allocations since the code runs >> in the app context and may need to access the memory after U-Boot has >> exited. Also efi_smbios.c uses allocate_pages() and then adds a new >> mapping as well. >> >> EFI memory has attributes, including what the memory is used for (to >> some degree of granularity). See enum efi_memory_type and struct >> efi_mem_desc. In the latter there are also attribute flags - whether >> memory is cacheable, etc. >> >> EFI also has the x86 idea of 'conventional' memory, meaning (I >> believe) that below 4GB that isn't reserved for the hardware/system. >> This is meaningless, or at least confusing, on ARM systems. >> >> 4. reservations >> >> It is perhaps worth mentioning a fourth method of memory management, >> where U-Boot reserves chunks of memory before relocation (in >> board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() >> region, etc. >> >> >> Problems >> —--- >> >> There are no urgent problems, but here are some things that could be >> improved: >> >> 1. EFI should attach most of its data structures to driver model. This >> work has started, with the partition support, but more effort would >> help. This would make it easier to see which memory is related to >> devices and which is separate. >> >> 2. Some drivers do EFI reservations today, whether EFI is used for >> booting or not (e.g. rockchip video rk_vop_probe()). >> >> 3. U-Boot doesn't really map arch-specific memory attributes (e.g. >> armv8's struct mm_region) to EFI ones. >> >> 4. EFI duplicates some code from bootm, some of which relates to >> memory allocation (e.g. FDT fixup). >> >> 5. EFI code is used even if EFI is never used to boot >> >> 6. EFI allocations can result in the same memory being used as has >> already been allocated by lmb. Users may load files which overwrite >> memory allocated by EFI. > >7. We need to support doing an allocation when a file is loaded (to >ensure files do not overlap), without making it too difficult to load >multiple files to the same place, if desired. > >> >> >> Lifetime >> >> >> We have three different memory allocators with different purposes. Can >> we unify them a little? >> >> Within U-Boot: >> - malloc() space lives forever >> - lmb lives while setting out images for booting >> - EFI (mostly) lives while booting an EFI app >> >> In practice, EFI is set up early in U-Boot. Some of this is necessary, >> some not. EFI allocations stay around forever. This works OK since >> large allocations are normally not done in EFI, so memory isn't really >> consumed to any great degree by the boot process. >> >> What happens to EFI allocations if the app returns? They are still >> present, in case another app is run. This seems fine. >> >> API >> –-- >> Can we unify some APIs? >> >> It should be possible to use lmb for large EFI memory allocations, so >> long as they are only needed for boo
Re: Proposal: U-Boot memory management
Hi Simon, On Fri, 29 Dec 2023 at 07:36, Simon Glass wrote: > > Hi Ilias, > > On Fri, Dec 22, 2023 at 12:02 PM Ilias Apalodimas > wrote: > > > > Hi Simon > > > > I'll respond to the rest more thoroughly but I since I caught this early, > > > > [...] > > > > > > > > 5. Avoid calling efi_allocate_pages() and efi_allocate_pool() outside > > > boot-time services. This solves the problem 6. If memory is needed by > > > an app, allocate it with malloc() and see 3. There are only two > > > efi_allocate_pages() (smbios and efi_runtime). There are more calls of > > > efi_allocate_pool(), but most of these seem easy to fix up. For > > > example, efi_init_event_log() allocates a buffer, but this can be > > > allocated in normal malloc() space or in a bloblist. > > > > The TCG event log is only valid in the EFI world and is described by > > the EFI spec extensions [0]. I prefer it to remain as is > > How does that relate to [1] ? We use that today. The tl;dr is that TF-A measures the early stages, but doesn't have drivers to extend TPM PCRs. Instead, it creates an EventLog which hands over to U-Boot. The TPM subsystem searches for that and if it finds one && PCR0 == 0 replays the EventLog in hardware and copies it over to EFI memory. That being said, I'd much prefer using a bloblist for that EventLog (which also contains information on replaying), over a DT entry that's only described in TF-A docs. But that's a future improvement. > > > > > > > > > 6. Don't worry too much about whether EFI will be used for booting. > > > The cost is likely not that great: use bootstage to measure it as is > > > done for driver model. Try to minmise the cost of its tables, > > > particularly for execution time, but otherwise just rely on the > > > ability to disable EFI_LOADER. > > > > > > – > > > > > > Regards, > > > Simon > > > > [0] > > https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf > > Thanks > > /Ilias > > [1] > https://trustedfirmware-a.readthedocs.io/en/latest/components/measured_boot/event_log.html Cheers /Ilias
Re: Proposal: U-Boot memory management
Hi, On Sat, Dec 16, 2023 at 6:01 PM Simon Glass wrote: > > Hi, > > This records my thoughts after a discussion with Ilias & Heinrich re > memory allocation in U-Boot. > > 1. malloc() > > malloc() is used for programmatic memory allocation. It allows memory > to be freed. It is not designed for very large allocations (e.g. a > 10MB kernel or 100MB ramdisk). > > 2. lmb > > lmb is used for large blocks of memory, such as those needed for a > kernel or ramdisk. Allocation is only transitory, for the purposes of > loading some images and booting. If the boot fails, then all lmb > allocations go away. > > lmb is set up by getting all available memory and then removing what > is used by U-Boot (code, data, malloc() space, etc.) > > lmb reservations have a few flags so that areas of memory can be > provided with attributes > > There are some corner cases...e.g. loading a file does an lmb > allocation but only for the purpose of avoiding a file being loaded > over U-Boot code/data. The allocation is dropped immediately after the > file is loaded. Within the bootm command, or when using standard boot, > this would be fairly easy to solve. > > Linux has renamed lmb to memblock. We should consider doing the same. > > 3. EFI > > EFI has its own memory-allocation tables. > > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > function it can also do smaller allocations similar to malloc(), > although each one uses at least 4KB at present. > > EFI allocations do not go away when a boot fails. > > With EFI it is possible to add allocations post facto, in which case > they are added to the allocation table just as if the memory was > allocated with EFI to begin with. > > The EFI allocations and the lmb allocations use the same memory, so in > principle could conflict. > > EFI allocations are sometimes used to allocate internal U-Boot data as > well, if needed by the EFI app. For example, while efi_image_parse() > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > in the app context and may need to access the memory after U-Boot has > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > mapping as well. > > EFI memory has attributes, including what the memory is used for (to > some degree of granularity). See enum efi_memory_type and struct > efi_mem_desc. In the latter there are also attribute flags - whether > memory is cacheable, etc. > > EFI also has the x86 idea of 'conventional' memory, meaning (I > believe) that below 4GB that isn't reserved for the hardware/system. > This is meaningless, or at least confusing, on ARM systems. > > 4. reservations > > It is perhaps worth mentioning a fourth method of memory management, > where U-Boot reserves chunks of memory before relocation (in > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > region, etc. > > > Problems > —--- > > There are no urgent problems, but here are some things that could be improved: > > 1. EFI should attach most of its data structures to driver model. This > work has started, with the partition support, but more effort would > help. This would make it easier to see which memory is related to > devices and which is separate. > > 2. Some drivers do EFI reservations today, whether EFI is used for > booting or not (e.g. rockchip video rk_vop_probe()). > > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > armv8's struct mm_region) to EFI ones. > > 4. EFI duplicates some code from bootm, some of which relates to > memory allocation (e.g. FDT fixup). > > 5. EFI code is used even if EFI is never used to boot > > 6. EFI allocations can result in the same memory being used as has > already been allocated by lmb. Users may load files which overwrite > memory allocated by EFI. 7. We need to support doing an allocation when a file is loaded (to ensure files do not overlap), without making it too difficult to load multiple files to the same place, if desired. > > > Lifetime > > > We have three different memory allocators with different purposes. Can > we unify them a little? > > Within U-Boot: > - malloc() space lives forever > - lmb lives while setting out images for booting > - EFI (mostly) lives while booting an EFI app > > In practice, EFI is set up early in U-Boot. Some of this is necessary, > some not. EFI allocations stay around forever. This works OK since > large allocations are normally not done in EFI, so memory isn't really > consumed to any great degree by the boot process. > > What happens to EFI allocations if the app returns? They are still > present, in case another app is run. This seems fine. > > API > –-- > Can we unify some APIs? > > It should be possible to use lmb for large EFI memory allocations, so > long as they are only needed for booting. We effectively do this > today, since EFI does not manage the arrangement of loaded images in > memory. for the most part. > > It would not make sense to use EFI allocation to replace lmb an
Re: Proposal: U-Boot memory management
Hi Ilias, On Fri, Dec 22, 2023 at 12:02 PM Ilias Apalodimas wrote: > > Hi Simon > > I'll respond to the rest more thoroughly but I since I caught this early, > > [...] > > > > > 5. Avoid calling efi_allocate_pages() and efi_allocate_pool() outside > > boot-time services. This solves the problem 6. If memory is needed by > > an app, allocate it with malloc() and see 3. There are only two > > efi_allocate_pages() (smbios and efi_runtime). There are more calls of > > efi_allocate_pool(), but most of these seem easy to fix up. For > > example, efi_init_event_log() allocates a buffer, but this can be > > allocated in normal malloc() space or in a bloblist. > > The TCG event log is only valid in the EFI world and is described by > the EFI spec extensions [0]. I prefer it to remain as is How does that relate to [1] ? > > > > > 6. Don't worry too much about whether EFI will be used for booting. > > The cost is likely not that great: use bootstage to measure it as is > > done for driver model. Try to minmise the cost of its tables, > > particularly for execution time, but otherwise just rely on the > > ability to disable EFI_LOADER. > > > > – > > > > Regards, > > Simon > > [0] > https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf > Thanks > /Ilias [1] https://trustedfirmware-a.readthedocs.io/en/latest/components/measured_boot/event_log.html
Re: Proposal: U-Boot memory management
Hi Simon I'll respond to the rest more thoroughly but I since I caught this early, [...] > > 5. Avoid calling efi_allocate_pages() and efi_allocate_pool() outside > boot-time services. This solves the problem 6. If memory is needed by > an app, allocate it with malloc() and see 3. There are only two > efi_allocate_pages() (smbios and efi_runtime). There are more calls of > efi_allocate_pool(), but most of these seem easy to fix up. For > example, efi_init_event_log() allocates a buffer, but this can be > allocated in normal malloc() space or in a bloblist. The TCG event log is only valid in the EFI world and is described by the EFI spec extensions [0]. I prefer it to remain as is > > 6. Don't worry too much about whether EFI will be used for booting. > The cost is likely not that great: use bootstage to measure it as is > done for driver model. Try to minmise the cost of its tables, > particularly for execution time, but otherwise just rely on the > ability to disable EFI_LOADER. > > – > > Regards, > Simon [0] https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf Thanks /Ilias
Re: Proposal: U-Boot memory management
On Tue, Dec 19, 2023 at 09:15:21PM -0700, Simon Glass wrote: > Hi, > > On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: > > > > On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: > > > > > > > > > Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini : > > > >On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: > > > >> > > > >> > > > >> Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini > > > >> : > > > >> >On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: > > > >> >> > > > >> >> > > > >> >> Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini > > > >> >> : > > > >> >> >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt > > > >> >> >wrote: > > > >> >> >> > > > >> >> >> > > > >> >> >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini > > > >> >> >> : > > > >> >> >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt > > > >> >> >> >wrote: > > > >> >> >> > > > > >> >> >> >[snip] > > > >> >> >> >> Or take: > > > >> >> >> >> > > > >> >> >> >> load host 0:1 $c kernel.efi > > > >> >> >> >> load host 0:1 $d initrd.img > > > >> >> >> >> > > > >> >> >> >> How could we ensure that initrd.img is not overwriting a part > > > >> >> >> >> of kernel.efi without memory allocation? > > > >> >> >> > > > > >> >> >> >Today, invalid checksum as part of some part of the kernel > > > >> >> >> >fails. But > > > >> >> >> >how do we do this tomorrow, are you suggesting that "load" > > > >> >> >> >perform > > > >> >> >> >malloc() in some predefined size? If $c is below $d and $c + > > > >> >> >> >kernel.efi > > > >> >> >> >is now above $d we can throw an error before trying to load, > > > >> >> >> >yes. But > > > >> >> >> >what about: > > > >> >> >> >load host 0:1 $d initrd.img > > > >> >> >> >load host 0:1 $c kernel.efi > > > >> >> >> > > > > >> >> >> >In that case (which is only marginally contrived, the more real > > > >> >> >> >case is > > > >> >> >> >loading device tree in to unexpectedly large ramdisk because > > > >> >> >> >someone > > > >> >> >> >didn't understand the general advice on why device tree is > > > >> >> >> >lower than > > > >> >> >> >ramdisk address) I'm fine with an error that amounts to "you > > > >> >> >> >just > > > >> >> >> >corrupted another allocation" and then "fail, reset the board" > > > >> >> >> >or so. > > > >> >> >> > > > > >> >> >> > > > >> >> >> Our current malloc library cannot manage the complete memory. We > > > >> >> >> need a library like lmb which should also cover the memory > > > >> >> >> management that we currently have in lib/efi/efi_memory.c. This > > > >> >> >> must include a memory type attribute for usage in the > > > >> >> >> GetMemoryMap() service. A management on page level seems > > > >> >> >> sufficient. > > > >> >> >> > > > >> >> >> The load command should permanently allocate memory in that lmb+ > > > >> >> >> library. > > > >> >> >> > > > >> >> >> We need an unload command to free the memory if we want to reuse > > > >> >> >> the memory or we might let the load comand free the memory if > > > >> >> >> exactly the same start address is reused. > > > >> >> > > > > >> >> >Our current way of loading things in to memory does not handle the > > > >> >> >case > > > >> >> >I described, yes. How would what you're proposing handle it? > > > >> >> > > > >> >> If the load command has to allocate memory for the image and that > > > >> >> allocation is kept, any attempt to allocate overlapping memory > > > >> >> would fail. > > > >> > > > > >> >So you're saying that the load command has to pre-allocate memory? Or > > > >> >as > > > >> >it goes? If the latter, in what size chunks? This starts to get at > > > >> >what > > > >> >Simon was talking about with respect to memory fragmentation. Which to > > > >> >be clear is a problem we have today, we just let things overlap and > > > >> >hope > > > >> >something later catches an incorrect checksum. > > > >> > > > > >> > > > >> I don't want to replace the malloc library which handles large numbets > > > >> of allocations. > > > > > > > >I'm confused. The normal malloc library is not involved with current > > > >image loading, it's direct to memory (with some attempts at sanity > > > >checking by lmb). Are you proposing a different allocator with > > > >malloc/free like behavior? If so, please outline how it will determine > > > >pool size, and how we'll use it to load thing to memory. > > > > > > All memory below the stack needs to be managed. Malloc uses a small > > > memory area (a few MiB) above the stack. > > > > That's a rather huge change for how U-Boot works. > > > > > >> Closing the eyes when the user loads multiple files does not solve the > > > >> fragmentation problem. > > > > > > > >Yes. I'm only noting that today we just ignore the problem and sometimes > > > >catch it via checksums. > > > > > > > >> Fragmentation only happens if we have many concurrent allocations. In > > > >> EFI we are allocating top down. The number of concu
Re: Proposal: U-Boot memory management
Hi, On Tue, 19 Dec 2023 at 05:46, Tom Rini wrote: > > On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: > > > > > > Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini : > > >On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: > > >> > > >> > > >> Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : > > >> >On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: > > >> >> > > >> >> > > >> >> Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini > > >> >> : > > >> >> >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: > > >> >> >> > > >> >> >> > > >> >> >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini > > >> >> >> : > > >> >> >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt > > >> >> >> >wrote: > > >> >> >> > > > >> >> >> >[snip] > > >> >> >> >> Or take: > > >> >> >> >> > > >> >> >> >> load host 0:1 $c kernel.efi > > >> >> >> >> load host 0:1 $d initrd.img > > >> >> >> >> > > >> >> >> >> How could we ensure that initrd.img is not overwriting a part > > >> >> >> >> of kernel.efi without memory allocation? > > >> >> >> > > > >> >> >> >Today, invalid checksum as part of some part of the kernel fails. > > >> >> >> >But > > >> >> >> >how do we do this tomorrow, are you suggesting that "load" perform > > >> >> >> >malloc() in some predefined size? If $c is below $d and $c + > > >> >> >> >kernel.efi > > >> >> >> >is now above $d we can throw an error before trying to load, yes. > > >> >> >> >But > > >> >> >> >what about: > > >> >> >> >load host 0:1 $d initrd.img > > >> >> >> >load host 0:1 $c kernel.efi > > >> >> >> > > > >> >> >> >In that case (which is only marginally contrived, the more real > > >> >> >> >case is > > >> >> >> >loading device tree in to unexpectedly large ramdisk because > > >> >> >> >someone > > >> >> >> >didn't understand the general advice on why device tree is lower > > >> >> >> >than > > >> >> >> >ramdisk address) I'm fine with an error that amounts to "you just > > >> >> >> >corrupted another allocation" and then "fail, reset the board" or > > >> >> >> >so. > > >> >> >> > > > >> >> >> > > >> >> >> Our current malloc library cannot manage the complete memory. We > > >> >> >> need a library like lmb which should also cover the memory > > >> >> >> management that we currently have in lib/efi/efi_memory.c. This > > >> >> >> must include a memory type attribute for usage in the > > >> >> >> GetMemoryMap() service. A management on page level seems > > >> >> >> sufficient. > > >> >> >> > > >> >> >> The load command should permanently allocate memory in that lmb+ > > >> >> >> library. > > >> >> >> > > >> >> >> We need an unload command to free the memory if we want to reuse > > >> >> >> the memory or we might let the load comand free the memory if > > >> >> >> exactly the same start address is reused. > > >> >> > > > >> >> >Our current way of loading things in to memory does not handle the > > >> >> >case > > >> >> >I described, yes. How would what you're proposing handle it? > > >> >> > > >> >> If the load command has to allocate memory for the image and that > > >> >> allocation is kept, any attempt to allocate overlapping memory would > > >> >> fail. > > >> > > > >> >So you're saying that the load command has to pre-allocate memory? Or as > > >> >it goes? If the latter, in what size chunks? This starts to get at what > > >> >Simon was talking about with respect to memory fragmentation. Which to > > >> >be clear is a problem we have today, we just let things overlap and hope > > >> >something later catches an incorrect checksum. > > >> > > > >> > > >> I don't want to replace the malloc library which handles large numbets > > >> of allocations. > > > > > >I'm confused. The normal malloc library is not involved with current > > >image loading, it's direct to memory (with some attempts at sanity > > >checking by lmb). Are you proposing a different allocator with > > >malloc/free like behavior? If so, please outline how it will determine > > >pool size, and how we'll use it to load thing to memory. > > > > All memory below the stack needs to be managed. Malloc uses a small memory > > area (a few MiB) above the stack. > > That's a rather huge change for how U-Boot works. > > > >> Closing the eyes when the user loads multiple files does not solve the > > >> fragmentation problem. > > > > > >Yes. I'm only noting that today we just ignore the problem and sometimes > > >catch it via checksums. > > > > > >> Fragmentation only happens if we have many concurrent allocations. In > > >> EFI we are allocating top down. The number of concurrent allocations is > > >> low. Typically a few dozen at most. After terminating an application > > >> these should be freed again. > > > > > >OK, so are you saying that we would no longer be loading _to_ a location > > >in memory and instead just be saying "load this thing" and picking where > > >dynamically? > > > > Both preassigned and allocator assigned adr
Re: Proposal: U-Boot memory management
On Tue, Dec 19, 2023 at 03:15:38AM +0100, Heinrich Schuchardt wrote: > > > Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini : > >On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: > >> > >> > >> Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : > >> >On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: > >> >> > >> >> > >> >> Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : > >> >> >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: > >> >> >> > >> >> >> > >> >> >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini > >> >> >> : > >> >> >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: > >> >> >> > > >> >> >> >[snip] > >> >> >> >> Or take: > >> >> >> >> > >> >> >> >> load host 0:1 $c kernel.efi > >> >> >> >> load host 0:1 $d initrd.img > >> >> >> >> > >> >> >> >> How could we ensure that initrd.img is not overwriting a part of > >> >> >> >> kernel.efi without memory allocation? > >> >> >> > > >> >> >> >Today, invalid checksum as part of some part of the kernel fails. > >> >> >> >But > >> >> >> >how do we do this tomorrow, are you suggesting that "load" perform > >> >> >> >malloc() in some predefined size? If $c is below $d and $c + > >> >> >> >kernel.efi > >> >> >> >is now above $d we can throw an error before trying to load, yes. > >> >> >> >But > >> >> >> >what about: > >> >> >> >load host 0:1 $d initrd.img > >> >> >> >load host 0:1 $c kernel.efi > >> >> >> > > >> >> >> >In that case (which is only marginally contrived, the more real > >> >> >> >case is > >> >> >> >loading device tree in to unexpectedly large ramdisk because someone > >> >> >> >didn't understand the general advice on why device tree is lower > >> >> >> >than > >> >> >> >ramdisk address) I'm fine with an error that amounts to "you just > >> >> >> >corrupted another allocation" and then "fail, reset the board" or > >> >> >> >so. > >> >> >> > > >> >> >> > >> >> >> Our current malloc library cannot manage the complete memory. We > >> >> >> need a library like lmb which should also cover the memory > >> >> >> management that we currently have in lib/efi/efi_memory.c. This must > >> >> >> include a memory type attribute for usage in the GetMemoryMap() > >> >> >> service. A management on page level seems sufficient. > >> >> >> > >> >> >> The load command should permanently allocate memory in that lmb+ > >> >> >> library. > >> >> >> > >> >> >> We need an unload command to free the memory if we want to reuse the > >> >> >> memory or we might let the load comand free the memory if exactly > >> >> >> the same start address is reused. > >> >> > > >> >> >Our current way of loading things in to memory does not handle the case > >> >> >I described, yes. How would what you're proposing handle it? > >> >> > >> >> If the load command has to allocate memory for the image and that > >> >> allocation is kept, any attempt to allocate overlapping memory would > >> >> fail. > >> > > >> >So you're saying that the load command has to pre-allocate memory? Or as > >> >it goes? If the latter, in what size chunks? This starts to get at what > >> >Simon was talking about with respect to memory fragmentation. Which to > >> >be clear is a problem we have today, we just let things overlap and hope > >> >something later catches an incorrect checksum. > >> > > >> > >> I don't want to replace the malloc library which handles large numbets of > >> allocations. > > > >I'm confused. The normal malloc library is not involved with current > >image loading, it's direct to memory (with some attempts at sanity > >checking by lmb). Are you proposing a different allocator with > >malloc/free like behavior? If so, please outline how it will determine > >pool size, and how we'll use it to load thing to memory. > > All memory below the stack needs to be managed. Malloc uses a small memory > area (a few MiB) above the stack. That's a rather huge change for how U-Boot works. > >> Closing the eyes when the user loads multiple files does not solve the > >> fragmentation problem. > > > >Yes. I'm only noting that today we just ignore the problem and sometimes > >catch it via checksums. > > > >> Fragmentation only happens if we have many concurrent allocations. In EFI > >> we are allocating top down. The number of concurrent allocations is low. > >> Typically a few dozen at most. After terminating an application these > >> should be freed again. > > > >OK, so are you saying that we would no longer be loading _to_ a location > >in memory and instead just be saying "load this thing" and picking where > >dynamically? > > Both preassigned and allocator assigned adresses are compatible with memory > management. > > Architectures and binaries have different requirements. On riscv64 you can > load Linux kernel, initrd, fdt anywhere. We don't need predefined addresses > there. Other architectures have restrictions. Yes, 64 bit architecture tend to only have alignment requ
Re: Proposal: U-Boot memory management
Am 19. Dezember 2023 02:26:00 MEZ schrieb Tom Rini : >On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: >> >> >> Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : >> >On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: >> >> >> >> >> >> Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : >> >> >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: >> >> >> >> >> >> >> >> >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini >> >> >> : >> >> >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: >> >> >> > >> >> >> >[snip] >> >> >> >> Or take: >> >> >> >> >> >> >> >> load host 0:1 $c kernel.efi >> >> >> >> load host 0:1 $d initrd.img >> >> >> >> >> >> >> >> How could we ensure that initrd.img is not overwriting a part of >> >> >> >> kernel.efi without memory allocation? >> >> >> > >> >> >> >Today, invalid checksum as part of some part of the kernel fails. But >> >> >> >how do we do this tomorrow, are you suggesting that "load" perform >> >> >> >malloc() in some predefined size? If $c is below $d and $c + >> >> >> >kernel.efi >> >> >> >is now above $d we can throw an error before trying to load, yes. But >> >> >> >what about: >> >> >> >load host 0:1 $d initrd.img >> >> >> >load host 0:1 $c kernel.efi >> >> >> > >> >> >> >In that case (which is only marginally contrived, the more real case >> >> >> >is >> >> >> >loading device tree in to unexpectedly large ramdisk because someone >> >> >> >didn't understand the general advice on why device tree is lower than >> >> >> >ramdisk address) I'm fine with an error that amounts to "you just >> >> >> >corrupted another allocation" and then "fail, reset the board" or so. >> >> >> > >> >> >> >> >> >> Our current malloc library cannot manage the complete memory. We need >> >> >> a library like lmb which should also cover the memory management that >> >> >> we currently have in lib/efi/efi_memory.c. This must include a memory >> >> >> type attribute for usage in the GetMemoryMap() service. A management >> >> >> on page level seems sufficient. >> >> >> >> >> >> The load command should permanently allocate memory in that lmb+ >> >> >> library. >> >> >> >> >> >> We need an unload command to free the memory if we want to reuse the >> >> >> memory or we might let the load comand free the memory if exactly the >> >> >> same start address is reused. >> >> > >> >> >Our current way of loading things in to memory does not handle the case >> >> >I described, yes. How would what you're proposing handle it? >> >> >> >> If the load command has to allocate memory for the image and that >> >> allocation is kept, any attempt to allocate overlapping memory would fail. >> > >> >So you're saying that the load command has to pre-allocate memory? Or as >> >it goes? If the latter, in what size chunks? This starts to get at what >> >Simon was talking about with respect to memory fragmentation. Which to >> >be clear is a problem we have today, we just let things overlap and hope >> >something later catches an incorrect checksum. >> > >> >> I don't want to replace the malloc library which handles large numbets of >> allocations. > >I'm confused. The normal malloc library is not involved with current >image loading, it's direct to memory (with some attempts at sanity >checking by lmb). Are you proposing a different allocator with >malloc/free like behavior? If so, please outline how it will determine >pool size, and how we'll use it to load thing to memory. All memory below the stack needs to be managed. Malloc uses a small memory area (a few MiB) above the stack. > >> Closing the eyes when the user loads multiple files does not solve the >> fragmentation problem. > >Yes. I'm only noting that today we just ignore the problem and sometimes >catch it via checksums. > >> Fragmentation only happens if we have many concurrent allocations. In EFI >> we are allocating top down. The number of concurrent allocations is low. >> Typically a few dozen at most. After terminating an application these should >> be freed again. > >OK, so are you saying that we would no longer be loading _to_ a location >in memory and instead just be saying "load this thing" and picking where >dynamically? Both preassigned and allocator assigned adresses are compatible with memory management. Architectures and binaries have different requirements. On riscv64 you can load Linux kernel, initrd, fdt anywhere. We don't need predefined addresses there. Other architectures have restrictions. > >> When loading a file from a file system we know the filesize beforehand. So >> allocation is trivial. >> >> The loady command currently does not use the offered size information but >> could do so. > >We should be using that information to make sure we don't overwrite >U-Boot itself, but I don't recall how exactly we handle it today >off-hand. If the user issues multiple load commands, he can overwrite previous files. During
Re: Proposal: U-Boot memory management
On Tue, Dec 19, 2023 at 01:01:51AM +0100, Heinrich Schuchardt wrote: > > > Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : > >On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: > >> > >> > >> Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : > >> >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: > >> >> > >> >> > >> >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : > >> >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: > >> >> > > >> >> >[snip] > >> >> >> Or take: > >> >> >> > >> >> >> load host 0:1 $c kernel.efi > >> >> >> load host 0:1 $d initrd.img > >> >> >> > >> >> >> How could we ensure that initrd.img is not overwriting a part of > >> >> >> kernel.efi without memory allocation? > >> >> > > >> >> >Today, invalid checksum as part of some part of the kernel fails. But > >> >> >how do we do this tomorrow, are you suggesting that "load" perform > >> >> >malloc() in some predefined size? If $c is below $d and $c + kernel.efi > >> >> >is now above $d we can throw an error before trying to load, yes. But > >> >> >what about: > >> >> >load host 0:1 $d initrd.img > >> >> >load host 0:1 $c kernel.efi > >> >> > > >> >> >In that case (which is only marginally contrived, the more real case is > >> >> >loading device tree in to unexpectedly large ramdisk because someone > >> >> >didn't understand the general advice on why device tree is lower than > >> >> >ramdisk address) I'm fine with an error that amounts to "you just > >> >> >corrupted another allocation" and then "fail, reset the board" or so. > >> >> > > >> >> > >> >> Our current malloc library cannot manage the complete memory. We need a > >> >> library like lmb which should also cover the memory management that we > >> >> currently have in lib/efi/efi_memory.c. This must include a memory type > >> >> attribute for usage in the GetMemoryMap() service. A management on page > >> >> level seems sufficient. > >> >> > >> >> The load command should permanently allocate memory in that lmb+ > >> >> library. > >> >> > >> >> We need an unload command to free the memory if we want to reuse the > >> >> memory or we might let the load comand free the memory if exactly the > >> >> same start address is reused. > >> > > >> >Our current way of loading things in to memory does not handle the case > >> >I described, yes. How would what you're proposing handle it? > >> > >> If the load command has to allocate memory for the image and that > >> allocation is kept, any attempt to allocate overlapping memory would fail. > > > >So you're saying that the load command has to pre-allocate memory? Or as > >it goes? If the latter, in what size chunks? This starts to get at what > >Simon was talking about with respect to memory fragmentation. Which to > >be clear is a problem we have today, we just let things overlap and hope > >something later catches an incorrect checksum. > > > > I don't want to replace the malloc library which handles large numbets of > allocations. I'm confused. The normal malloc library is not involved with current image loading, it's direct to memory (with some attempts at sanity checking by lmb). Are you proposing a different allocator with malloc/free like behavior? If so, please outline how it will determine pool size, and how we'll use it to load thing to memory. > Closing the eyes when the user loads multiple files does not solve the > fragmentation problem. Yes. I'm only noting that today we just ignore the problem and sometimes catch it via checksums. > Fragmentation only happens if we have many concurrent allocations. In EFI we > are allocating top down. The number of concurrent allocations is low. > Typically a few dozen at most. After terminating an application these should > be freed again. OK, so are you saying that we would no longer be loading _to_ a location in memory and instead just be saying "load this thing" and picking where dynamically? > When loading a file from a file system we know the filesize beforehand. So > allocation is trivial. > > The loady command currently does not use the offered size information but > could do so. We should be using that information to make sure we don't overwrite U-Boot itself, but I don't recall how exactly we handle it today off-hand. > TFTP is problematic because it does not transfer the filesize. We would > probably try to allocate a large chunk of memory and then downsize the > allocation after reading the whole file. Reading from non-filesystem flash also has this problem, but we at least specify the amount to read too. But yes, it gets back to what I was asking about on how you're proposing to handle network load cases. -- Tom signature.asc Description: PGP signature
Re: Proposal: U-Boot memory management
Am 19. Dezember 2023 00:31:30 MEZ schrieb Tom Rini : >On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: >> >> >> Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : >> >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: >> >> >> >> >> >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : >> >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: >> >> > >> >> >[snip] >> >> >> Or take: >> >> >> >> >> >> load host 0:1 $c kernel.efi >> >> >> load host 0:1 $d initrd.img >> >> >> >> >> >> How could we ensure that initrd.img is not overwriting a part of >> >> >> kernel.efi without memory allocation? >> >> > >> >> >Today, invalid checksum as part of some part of the kernel fails. But >> >> >how do we do this tomorrow, are you suggesting that "load" perform >> >> >malloc() in some predefined size? If $c is below $d and $c + kernel.efi >> >> >is now above $d we can throw an error before trying to load, yes. But >> >> >what about: >> >> >load host 0:1 $d initrd.img >> >> >load host 0:1 $c kernel.efi >> >> > >> >> >In that case (which is only marginally contrived, the more real case is >> >> >loading device tree in to unexpectedly large ramdisk because someone >> >> >didn't understand the general advice on why device tree is lower than >> >> >ramdisk address) I'm fine with an error that amounts to "you just >> >> >corrupted another allocation" and then "fail, reset the board" or so. >> >> > >> >> >> >> Our current malloc library cannot manage the complete memory. We need a >> >> library like lmb which should also cover the memory management that we >> >> currently have in lib/efi/efi_memory.c. This must include a memory type >> >> attribute for usage in the GetMemoryMap() service. A management on page >> >> level seems sufficient. >> >> >> >> The load command should permanently allocate memory in that lmb+ library. >> >> >> >> We need an unload command to free the memory if we want to reuse the >> >> memory or we might let the load comand free the memory if exactly the >> >> same start address is reused. >> > >> >Our current way of loading things in to memory does not handle the case >> >I described, yes. How would what you're proposing handle it? >> >> If the load command has to allocate memory for the image and that allocation >> is kept, any attempt to allocate overlapping memory would fail. > >So you're saying that the load command has to pre-allocate memory? Or as >it goes? If the latter, in what size chunks? This starts to get at what >Simon was talking about with respect to memory fragmentation. Which to >be clear is a problem we have today, we just let things overlap and hope >something later catches an incorrect checksum. > I don't want to replace the malloc library which handles large numbets of allocations. Closing the eyes when the user loads multiple files does not solve the fragmentation problem. Fragmentation only happens if we have many concurrent allocations. In EFI we are allocating top down. The number of concurrent allocations is low. Typically a few dozen at most. After terminating an application these should be freed again. When loading a file from a file system we know the filesize beforehand. So allocation is trivial. The loady command currently does not use the offered size information but could do so. TFTP is problematic because it does not transfer the filesize. We would probably try to allocate a large chunk of memory and then downsize the allocation after reading the whole file. We will have to review each file load method individually. Best regards Heinrich
Re: Proposal: U-Boot memory management
On Tue, Dec 19, 2023 at 12:29:19AM +0100, Heinrich Schuchardt wrote: > > > Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : > >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: > >> > >> > >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : > >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: > >> > > >> >[snip] > >> >> Or take: > >> >> > >> >> load host 0:1 $c kernel.efi > >> >> load host 0:1 $d initrd.img > >> >> > >> >> How could we ensure that initrd.img is not overwriting a part of > >> >> kernel.efi without memory allocation? > >> > > >> >Today, invalid checksum as part of some part of the kernel fails. But > >> >how do we do this tomorrow, are you suggesting that "load" perform > >> >malloc() in some predefined size? If $c is below $d and $c + kernel.efi > >> >is now above $d we can throw an error before trying to load, yes. But > >> >what about: > >> >load host 0:1 $d initrd.img > >> >load host 0:1 $c kernel.efi > >> > > >> >In that case (which is only marginally contrived, the more real case is > >> >loading device tree in to unexpectedly large ramdisk because someone > >> >didn't understand the general advice on why device tree is lower than > >> >ramdisk address) I'm fine with an error that amounts to "you just > >> >corrupted another allocation" and then "fail, reset the board" or so. > >> > > >> > >> Our current malloc library cannot manage the complete memory. We need a > >> library like lmb which should also cover the memory management that we > >> currently have in lib/efi/efi_memory.c. This must include a memory type > >> attribute for usage in the GetMemoryMap() service. A management on page > >> level seems sufficient. > >> > >> The load command should permanently allocate memory in that lmb+ library. > >> > >> We need an unload command to free the memory if we want to reuse the > >> memory or we might let the load comand free the memory if exactly the same > >> start address is reused. > > > >Our current way of loading things in to memory does not handle the case > >I described, yes. How would what you're proposing handle it? > > If the load command has to allocate memory for the image and that allocation > is kept, any attempt to allocate overlapping memory would fail. So you're saying that the load command has to pre-allocate memory? Or as it goes? If the latter, in what size chunks? This starts to get at what Simon was talking about with respect to memory fragmentation. Which to be clear is a problem we have today, we just let things overlap and hope something later catches an incorrect checksum. -- Tom signature.asc Description: PGP signature
Re: Proposal: U-Boot memory management
Am 19. Dezember 2023 00:16:40 MEZ schrieb Tom Rini : >On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: >> >> >> Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : >> >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: >> > >> >[snip] >> >> Or take: >> >> >> >> load host 0:1 $c kernel.efi >> >> load host 0:1 $d initrd.img >> >> >> >> How could we ensure that initrd.img is not overwriting a part of >> >> kernel.efi without memory allocation? >> > >> >Today, invalid checksum as part of some part of the kernel fails. But >> >how do we do this tomorrow, are you suggesting that "load" perform >> >malloc() in some predefined size? If $c is below $d and $c + kernel.efi >> >is now above $d we can throw an error before trying to load, yes. But >> >what about: >> >load host 0:1 $d initrd.img >> >load host 0:1 $c kernel.efi >> > >> >In that case (which is only marginally contrived, the more real case is >> >loading device tree in to unexpectedly large ramdisk because someone >> >didn't understand the general advice on why device tree is lower than >> >ramdisk address) I'm fine with an error that amounts to "you just >> >corrupted another allocation" and then "fail, reset the board" or so. >> > >> >> Our current malloc library cannot manage the complete memory. We need a >> library like lmb which should also cover the memory management that we >> currently have in lib/efi/efi_memory.c. This must include a memory type >> attribute for usage in the GetMemoryMap() service. A management on page >> level seems sufficient. >> >> The load command should permanently allocate memory in that lmb+ library. >> >> We need an unload command to free the memory if we want to reuse the memory >> or we might let the load comand free the memory if exactly the same start >> address is reused. > >Our current way of loading things in to memory does not handle the case >I described, yes. How would what you're proposing handle it? If the load command has to allocate memory for the image and that allocation is kept, any attempt to allocate overlapping memory would fail. Furthermore the EFI sub-system would be aware of such memory allocations and respect them. This would of course also hold true for allocations for SMBIOS or ACPI tables. Best regards Heinrich
Re: Proposal: U-Boot memory management
On Tue, Dec 19, 2023 at 12:08:31AM +0100, Heinrich Schuchardt wrote: > > > Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : > >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: > > > >[snip] > >> Or take: > >> > >> load host 0:1 $c kernel.efi > >> load host 0:1 $d initrd.img > >> > >> How could we ensure that initrd.img is not overwriting a part of > >> kernel.efi without memory allocation? > > > >Today, invalid checksum as part of some part of the kernel fails. But > >how do we do this tomorrow, are you suggesting that "load" perform > >malloc() in some predefined size? If $c is below $d and $c + kernel.efi > >is now above $d we can throw an error before trying to load, yes. But > >what about: > >load host 0:1 $d initrd.img > >load host 0:1 $c kernel.efi > > > >In that case (which is only marginally contrived, the more real case is > >loading device tree in to unexpectedly large ramdisk because someone > >didn't understand the general advice on why device tree is lower than > >ramdisk address) I'm fine with an error that amounts to "you just > >corrupted another allocation" and then "fail, reset the board" or so. > > > > Our current malloc library cannot manage the complete memory. We need a > library like lmb which should also cover the memory management that we > currently have in lib/efi/efi_memory.c. This must include a memory type > attribute for usage in the GetMemoryMap() service. A management on page level > seems sufficient. > > The load command should permanently allocate memory in that lmb+ library. > > We need an unload command to free the memory if we want to reuse the memory > or we might let the load comand free the memory if exactly the same start > address is reused. Our current way of loading things in to memory does not handle the case I described, yes. How would what you're proposing handle it? -- Tom signature.asc Description: PGP signature
Re: Proposal: U-Boot memory management
Am 18. Dezember 2023 23:41:08 MEZ schrieb Tom Rini : >On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: > >[snip] >> Or take: >> >> load host 0:1 $c kernel.efi >> load host 0:1 $d initrd.img >> >> How could we ensure that initrd.img is not overwriting a part of kernel.efi >> without memory allocation? > >Today, invalid checksum as part of some part of the kernel fails. But >how do we do this tomorrow, are you suggesting that "load" perform >malloc() in some predefined size? If $c is below $d and $c + kernel.efi >is now above $d we can throw an error before trying to load, yes. But >what about: >load host 0:1 $d initrd.img >load host 0:1 $c kernel.efi > >In that case (which is only marginally contrived, the more real case is >loading device tree in to unexpectedly large ramdisk because someone >didn't understand the general advice on why device tree is lower than >ramdisk address) I'm fine with an error that amounts to "you just >corrupted another allocation" and then "fail, reset the board" or so. > Our current malloc library cannot manage the complete memory. We need a library like lmb which should also cover the memory management that we currently have in lib/efi/efi_memory.c. This must include a memory type attribute for usage in the GetMemoryMap() service. A management on page level seems sufficient. The load command should permanently allocate memory in that lmb+ library. We need an unload command to free the memory if we want to reuse the memory or we might let the load comand free the memory if exactly the same start address is reused. Best regards Heinrich
Re: Proposal: U-Boot memory management
On Mon, Dec 18, 2023 at 11:34:16PM +0100, Heinrich Schuchardt wrote: [snip] > Or take: > > load host 0:1 $c kernel.efi > load host 0:1 $d initrd.img > > How could we ensure that initrd.img is not overwriting a part of kernel.efi > without memory allocation? Today, invalid checksum as part of some part of the kernel fails. But how do we do this tomorrow, are you suggesting that "load" perform malloc() in some predefined size? If $c is below $d and $c + kernel.efi is now above $d we can throw an error before trying to load, yes. But what about: load host 0:1 $d initrd.img load host 0:1 $c kernel.efi In that case (which is only marginally contrived, the more real case is loading device tree in to unexpectedly large ramdisk because someone didn't understand the general advice on why device tree is lower than ramdisk address) I'm fine with an error that amounts to "you just corrupted another allocation" and then "fail, reset the board" or so. -- Tom signature.asc Description: PGP signature
Re: Proposal: U-Boot memory management
Am 18. Dezember 2023 22:48:43 MEZ schrieb Simon Glass : >Hi Heinrich, > >On Mon, 18 Dec 2023 at 14:37, Heinrich Schuchardt wrote: >> >> >> >> Am 18. Dezember 2023 22:03:41 MEZ schrieb Simon Glass : >> >Hi Heinrich, >> > >> >On Mon, 18 Dec 2023 at 13:00, Heinrich Schuchardt >> >wrote: >> >> >> >> >> >> >> >> Am 18. Dezember 2023 19:12:11 MEZ schrieb Simon Glass : >> >> >Hi Heinrich, >> >> > >> >> >On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt >> >> >wrote: >> >> >> >> >> >> On 12/16/23 19:01, Simon Glass wrote: >> >> >> > Hi, >> >> >> > >> >> >> > This records my thoughts after a discussion with Ilias & Heinrich re >> >> >> > memory allocation in U-Boot. >> >> >> > >> >> >> > 1. malloc() >> >> >> > >> >> >> > malloc() is used for programmatic memory allocation. It allows memory >> >> >> > to be freed. It is not designed for very large allocations (e.g. a >> >> >> > 10MB kernel or 100MB ramdisk). >> >> >> > >> >> >> > 2. lmb >> >> >> > >> >> >> > lmb is used for large blocks of memory, such as those needed for a >> >> >> > kernel or ramdisk. Allocation is only transitory, for the purposes of >> >> >> > loading some images and booting. If the boot fails, then all lmb >> >> >> > allocations go away. >> >> >> > >> >> >> > lmb is set up by getting all available memory and then removing what >> >> >> > is used by U-Boot (code, data, malloc() space, etc.) >> >> >> > >> >> >> > lmb reservations have a few flags so that areas of memory can be >> >> >> > provided with attributes >> >> >> > >> >> >> > There are some corner cases...e.g. loading a file does an lmb >> >> >> > allocation but only for the purpose of avoiding a file being loaded >> >> >> > over U-Boot code/data. The allocation is dropped immediately after >> >> >> > the >> >> >> > file is loaded. Within the bootm command, or when using standard >> >> >> > boot, >> >> >> > this would be fairly easy to solve. >> >> >> > >> >> >> > Linux has renamed lmb to memblock. We should consider doing the same. >> >> >> > >> >> >> > 3. EFI >> >> >> > >> >> >> > EFI has its own memory-allocation tables. >> >> >> > >> >> >> > Like lmb, EFI is able to deal with large allocations. But via a >> >> >> > 'pool' >> >> >> > function it can also do smaller allocations similar to malloc(), >> >> >> > although each one uses at least 4KB at present. >> >> >> > >> >> >> > EFI allocations do not go away when a boot fails. >> >> >> > >> >> >> > With EFI it is possible to add allocations post facto, in which case >> >> >> > they are added to the allocation table just as if the memory was >> >> >> > allocated with EFI to begin with. >> >> >> > >> >> >> > The EFI allocations and the lmb allocations use the same memory, so >> >> >> > in >> >> >> > principle could conflict. >> >> >> > >> >> >> > EFI allocations are sometimes used to allocate internal U-Boot data >> >> >> > as >> >> >> > well, if needed by the EFI app. For example, while efi_image_parse() >> >> >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs >> >> >> > in the app context and may need to access the memory after U-Boot has >> >> >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new >> >> >> > mapping as well. >> >> >> > >> >> >> > EFI memory has attributes, including what the memory is used for (to >> >> >> > some degree of granularity). See enum efi_memory_type and struct >> >> >> > efi_mem_desc. In the latter there are also attribute flags - whether >> >> >> > memory is cacheable, etc. >> >> >> > >> >> >> > EFI also has the x86 idea of 'conventional' memory, meaning (I >> >> >> > believe) that below 4GB that isn't reserved for the hardware/system. >> >> >> > This is meaningless, or at least confusing, on ARM systems. >> >> >> > >> >> >> > 4. reservations >> >> >> > >> >> >> > It is perhaps worth mentioning a fourth method of memory management, >> >> >> > where U-Boot reserves chunks of memory before relocation (in >> >> >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() >> >> >> > region, etc. >> >> >> > >> >> >> > >> >> >> > Problems >> >> >> > —--- >> >> >> > >> >> >> > There are no urgent problems, but here are some things that could be >> >> >> > improved: >> >> >> > >> >> >> > 1. EFI should attach most of its data structures to driver model. >> >> >> > This >> >> >> > work has started, with the partition support, but more effort would >> >> >> > help. This would make it easier to see which memory is related to >> >> >> > devices and which is separate. >> >> >> > >> >> >> > 2. Some drivers do EFI reservations today, whether EFI is used for >> >> >> > booting or not (e.g. rockchip video rk_vop_probe()). >> >> >> >> >> >> Hello Simon, >> >> >> >> >> >> thank you for summarizing our discussion. >> >> >> >> >> >> Some U-Boot drivers including rockchip video inform the EFI sub-system >> >> >> that memory is reserved. >> >> >> >> >> >> Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are >> >> >> still used after ExitBoo
Re: Proposal: U-Boot memory management
On Mon, Dec 18, 2023 at 02:48:43PM -0700, Simon Glass wrote: > Hi Heinrich, > > On Mon, 18 Dec 2023 at 14:37, Heinrich Schuchardt wrote: > > > > > > > > Am 18. Dezember 2023 22:03:41 MEZ schrieb Simon Glass : > > >Hi Heinrich, > > > > > >On Mon, 18 Dec 2023 at 13:00, Heinrich Schuchardt > > >wrote: > > >> > > >> > > >> > > >> Am 18. Dezember 2023 19:12:11 MEZ schrieb Simon Glass > > >> : > > >> >Hi Heinrich, > > >> > > > >> >On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt > > >> >wrote: > > >> >> > > >> >> On 12/16/23 19:01, Simon Glass wrote: > > >> >> > Hi, > > >> >> > > > >> >> > This records my thoughts after a discussion with Ilias & Heinrich re > > >> >> > memory allocation in U-Boot. > > >> >> > > > >> >> > 1. malloc() > > >> >> > > > >> >> > malloc() is used for programmatic memory allocation. It allows > > >> >> > memory > > >> >> > to be freed. It is not designed for very large allocations (e.g. a > > >> >> > 10MB kernel or 100MB ramdisk). > > >> >> > > > >> >> > 2. lmb > > >> >> > > > >> >> > lmb is used for large blocks of memory, such as those needed for a > > >> >> > kernel or ramdisk. Allocation is only transitory, for the purposes > > >> >> > of > > >> >> > loading some images and booting. If the boot fails, then all lmb > > >> >> > allocations go away. > > >> >> > > > >> >> > lmb is set up by getting all available memory and then removing what > > >> >> > is used by U-Boot (code, data, malloc() space, etc.) > > >> >> > > > >> >> > lmb reservations have a few flags so that areas of memory can be > > >> >> > provided with attributes > > >> >> > > > >> >> > There are some corner cases...e.g. loading a file does an lmb > > >> >> > allocation but only for the purpose of avoiding a file being loaded > > >> >> > over U-Boot code/data. The allocation is dropped immediately after > > >> >> > the > > >> >> > file is loaded. Within the bootm command, or when using standard > > >> >> > boot, > > >> >> > this would be fairly easy to solve. > > >> >> > > > >> >> > Linux has renamed lmb to memblock. We should consider doing the > > >> >> > same. > > >> >> > > > >> >> > 3. EFI > > >> >> > > > >> >> > EFI has its own memory-allocation tables. > > >> >> > > > >> >> > Like lmb, EFI is able to deal with large allocations. But via a > > >> >> > 'pool' > > >> >> > function it can also do smaller allocations similar to malloc(), > > >> >> > although each one uses at least 4KB at present. > > >> >> > > > >> >> > EFI allocations do not go away when a boot fails. > > >> >> > > > >> >> > With EFI it is possible to add allocations post facto, in which case > > >> >> > they are added to the allocation table just as if the memory was > > >> >> > allocated with EFI to begin with. > > >> >> > > > >> >> > The EFI allocations and the lmb allocations use the same memory, so > > >> >> > in > > >> >> > principle could conflict. > > >> >> > > > >> >> > EFI allocations are sometimes used to allocate internal U-Boot data > > >> >> > as > > >> >> > well, if needed by the EFI app. For example, while efi_image_parse() > > >> >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code > > >> >> > runs > > >> >> > in the app context and may need to access the memory after U-Boot > > >> >> > has > > >> >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > > >> >> > mapping as well. > > >> >> > > > >> >> > EFI memory has attributes, including what the memory is used for (to > > >> >> > some degree of granularity). See enum efi_memory_type and struct > > >> >> > efi_mem_desc. In the latter there are also attribute flags - whether > > >> >> > memory is cacheable, etc. > > >> >> > > > >> >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > > >> >> > believe) that below 4GB that isn't reserved for the hardware/system. > > >> >> > This is meaningless, or at least confusing, on ARM systems. > > >> >> > > > >> >> > 4. reservations > > >> >> > > > >> >> > It is perhaps worth mentioning a fourth method of memory management, > > >> >> > where U-Boot reserves chunks of memory before relocation (in > > >> >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > > >> >> > region, etc. > > >> >> > > > >> >> > > > >> >> > Problems > > >> >> > —--- > > >> >> > > > >> >> > There are no urgent problems, but here are some things that could > > >> >> > be improved: > > >> >> > > > >> >> > 1. EFI should attach most of its data structures to driver model. > > >> >> > This > > >> >> > work has started, with the partition support, but more effort would > > >> >> > help. This would make it easier to see which memory is related to > > >> >> > devices and which is separate. > > >> >> > > > >> >> > 2. Some drivers do EFI reservations today, whether EFI is used for > > >> >> > booting or not (e.g. rockchip video rk_vop_probe()). > > >> >> > > >> >> Hello Simon, > > >> >> > > >> >> thank you for summarizing our discussion. > > >> >> > > >> >> Some U-Boot drivers including ro
Re: Proposal: U-Boot memory management
Hi Heinrich, On Mon, 18 Dec 2023 at 14:37, Heinrich Schuchardt wrote: > > > > Am 18. Dezember 2023 22:03:41 MEZ schrieb Simon Glass : > >Hi Heinrich, > > > >On Mon, 18 Dec 2023 at 13:00, Heinrich Schuchardt wrote: > >> > >> > >> > >> Am 18. Dezember 2023 19:12:11 MEZ schrieb Simon Glass : > >> >Hi Heinrich, > >> > > >> >On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt > >> >wrote: > >> >> > >> >> On 12/16/23 19:01, Simon Glass wrote: > >> >> > Hi, > >> >> > > >> >> > This records my thoughts after a discussion with Ilias & Heinrich re > >> >> > memory allocation in U-Boot. > >> >> > > >> >> > 1. malloc() > >> >> > > >> >> > malloc() is used for programmatic memory allocation. It allows memory > >> >> > to be freed. It is not designed for very large allocations (e.g. a > >> >> > 10MB kernel or 100MB ramdisk). > >> >> > > >> >> > 2. lmb > >> >> > > >> >> > lmb is used for large blocks of memory, such as those needed for a > >> >> > kernel or ramdisk. Allocation is only transitory, for the purposes of > >> >> > loading some images and booting. If the boot fails, then all lmb > >> >> > allocations go away. > >> >> > > >> >> > lmb is set up by getting all available memory and then removing what > >> >> > is used by U-Boot (code, data, malloc() space, etc.) > >> >> > > >> >> > lmb reservations have a few flags so that areas of memory can be > >> >> > provided with attributes > >> >> > > >> >> > There are some corner cases...e.g. loading a file does an lmb > >> >> > allocation but only for the purpose of avoiding a file being loaded > >> >> > over U-Boot code/data. The allocation is dropped immediately after the > >> >> > file is loaded. Within the bootm command, or when using standard boot, > >> >> > this would be fairly easy to solve. > >> >> > > >> >> > Linux has renamed lmb to memblock. We should consider doing the same. > >> >> > > >> >> > 3. EFI > >> >> > > >> >> > EFI has its own memory-allocation tables. > >> >> > > >> >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > >> >> > function it can also do smaller allocations similar to malloc(), > >> >> > although each one uses at least 4KB at present. > >> >> > > >> >> > EFI allocations do not go away when a boot fails. > >> >> > > >> >> > With EFI it is possible to add allocations post facto, in which case > >> >> > they are added to the allocation table just as if the memory was > >> >> > allocated with EFI to begin with. > >> >> > > >> >> > The EFI allocations and the lmb allocations use the same memory, so in > >> >> > principle could conflict. > >> >> > > >> >> > EFI allocations are sometimes used to allocate internal U-Boot data as > >> >> > well, if needed by the EFI app. For example, while efi_image_parse() > >> >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > >> >> > in the app context and may need to access the memory after U-Boot has > >> >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > >> >> > mapping as well. > >> >> > > >> >> > EFI memory has attributes, including what the memory is used for (to > >> >> > some degree of granularity). See enum efi_memory_type and struct > >> >> > efi_mem_desc. In the latter there are also attribute flags - whether > >> >> > memory is cacheable, etc. > >> >> > > >> >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > >> >> > believe) that below 4GB that isn't reserved for the hardware/system. > >> >> > This is meaningless, or at least confusing, on ARM systems. > >> >> > > >> >> > 4. reservations > >> >> > > >> >> > It is perhaps worth mentioning a fourth method of memory management, > >> >> > where U-Boot reserves chunks of memory before relocation (in > >> >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > >> >> > region, etc. > >> >> > > >> >> > > >> >> > Problems > >> >> > —--- > >> >> > > >> >> > There are no urgent problems, but here are some things that could be > >> >> > improved: > >> >> > > >> >> > 1. EFI should attach most of its data structures to driver model. This > >> >> > work has started, with the partition support, but more effort would > >> >> > help. This would make it easier to see which memory is related to > >> >> > devices and which is separate. > >> >> > > >> >> > 2. Some drivers do EFI reservations today, whether EFI is used for > >> >> > booting or not (e.g. rockchip video rk_vop_probe()). > >> >> > >> >> Hello Simon, > >> >> > >> >> thank you for summarizing our discussion. > >> >> > >> >> Some U-Boot drivers including rockchip video inform the EFI sub-system > >> >> that memory is reserved. > >> >> > >> >> Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are > >> >> still used after ExitBootServices. mmio addresses have to be updated > >> >> when Linux creates its virtual memory map. Currently this is done via > >> >> efi_add_runtime_mmio(). A more UEFI style method would be to register an > >> >> event handler for ExitBootServices() and
Re: Proposal: U-Boot memory management
On Mon, Dec 18, 2023 at 02:03:41PM -0700, Simon Glass wrote: > Hi Heinrich, > > On Mon, 18 Dec 2023 at 13:00, Heinrich Schuchardt wrote: > > > > > > > > Am 18. Dezember 2023 19:12:11 MEZ schrieb Simon Glass : > > >Hi Heinrich, > > > > > >On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt > > >wrote: > > >> > > >> On 12/16/23 19:01, Simon Glass wrote: > > >> > Hi, > > >> > > > >> > This records my thoughts after a discussion with Ilias & Heinrich re > > >> > memory allocation in U-Boot. > > >> > > > >> > 1. malloc() > > >> > > > >> > malloc() is used for programmatic memory allocation. It allows memory > > >> > to be freed. It is not designed for very large allocations (e.g. a > > >> > 10MB kernel or 100MB ramdisk). > > >> > > > >> > 2. lmb > > >> > > > >> > lmb is used for large blocks of memory, such as those needed for a > > >> > kernel or ramdisk. Allocation is only transitory, for the purposes of > > >> > loading some images and booting. If the boot fails, then all lmb > > >> > allocations go away. > > >> > > > >> > lmb is set up by getting all available memory and then removing what > > >> > is used by U-Boot (code, data, malloc() space, etc.) > > >> > > > >> > lmb reservations have a few flags so that areas of memory can be > > >> > provided with attributes > > >> > > > >> > There are some corner cases...e.g. loading a file does an lmb > > >> > allocation but only for the purpose of avoiding a file being loaded > > >> > over U-Boot code/data. The allocation is dropped immediately after the > > >> > file is loaded. Within the bootm command, or when using standard boot, > > >> > this would be fairly easy to solve. > > >> > > > >> > Linux has renamed lmb to memblock. We should consider doing the same. > > >> > > > >> > 3. EFI > > >> > > > >> > EFI has its own memory-allocation tables. > > >> > > > >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > > >> > function it can also do smaller allocations similar to malloc(), > > >> > although each one uses at least 4KB at present. > > >> > > > >> > EFI allocations do not go away when a boot fails. > > >> > > > >> > With EFI it is possible to add allocations post facto, in which case > > >> > they are added to the allocation table just as if the memory was > > >> > allocated with EFI to begin with. > > >> > > > >> > The EFI allocations and the lmb allocations use the same memory, so in > > >> > principle could conflict. > > >> > > > >> > EFI allocations are sometimes used to allocate internal U-Boot data as > > >> > well, if needed by the EFI app. For example, while efi_image_parse() > > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > > >> > in the app context and may need to access the memory after U-Boot has > > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > > >> > mapping as well. > > >> > > > >> > EFI memory has attributes, including what the memory is used for (to > > >> > some degree of granularity). See enum efi_memory_type and struct > > >> > efi_mem_desc. In the latter there are also attribute flags - whether > > >> > memory is cacheable, etc. > > >> > > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > > >> > believe) that below 4GB that isn't reserved for the hardware/system. > > >> > This is meaningless, or at least confusing, on ARM systems. > > >> > > > >> > 4. reservations > > >> > > > >> > It is perhaps worth mentioning a fourth method of memory management, > > >> > where U-Boot reserves chunks of memory before relocation (in > > >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > > >> > region, etc. > > >> > > > >> > > > >> > Problems > > >> > —--- > > >> > > > >> > There are no urgent problems, but here are some things that could be > > >> > improved: > > >> > > > >> > 1. EFI should attach most of its data structures to driver model. This > > >> > work has started, with the partition support, but more effort would > > >> > help. This would make it easier to see which memory is related to > > >> > devices and which is separate. > > >> > > > >> > 2. Some drivers do EFI reservations today, whether EFI is used for > > >> > booting or not (e.g. rockchip video rk_vop_probe()). > > >> > > >> Hello Simon, > > >> > > >> thank you for summarizing our discussion. > > >> > > >> Some U-Boot drivers including rockchip video inform the EFI sub-system > > >> that memory is reserved. > > >> > > >> Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are > > >> still used after ExitBootServices. mmio addresses have to be updated > > >> when Linux creates its virtual memory map. Currently this is done via > > >> efi_add_runtime_mmio(). A more UEFI style method would be to register an > > >> event handler for ExitBootServices() and use ConvertPointer() in the > > >> event handler. > > >> > > >> > > > >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > > >> > armv8's struct mm_region) to EFI on
Re: Proposal: U-Boot memory management
Am 18. Dezember 2023 22:03:41 MEZ schrieb Simon Glass : >Hi Heinrich, > >On Mon, 18 Dec 2023 at 13:00, Heinrich Schuchardt wrote: >> >> >> >> Am 18. Dezember 2023 19:12:11 MEZ schrieb Simon Glass : >> >Hi Heinrich, >> > >> >On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt >> >wrote: >> >> >> >> On 12/16/23 19:01, Simon Glass wrote: >> >> > Hi, >> >> > >> >> > This records my thoughts after a discussion with Ilias & Heinrich re >> >> > memory allocation in U-Boot. >> >> > >> >> > 1. malloc() >> >> > >> >> > malloc() is used for programmatic memory allocation. It allows memory >> >> > to be freed. It is not designed for very large allocations (e.g. a >> >> > 10MB kernel or 100MB ramdisk). >> >> > >> >> > 2. lmb >> >> > >> >> > lmb is used for large blocks of memory, such as those needed for a >> >> > kernel or ramdisk. Allocation is only transitory, for the purposes of >> >> > loading some images and booting. If the boot fails, then all lmb >> >> > allocations go away. >> >> > >> >> > lmb is set up by getting all available memory and then removing what >> >> > is used by U-Boot (code, data, malloc() space, etc.) >> >> > >> >> > lmb reservations have a few flags so that areas of memory can be >> >> > provided with attributes >> >> > >> >> > There are some corner cases...e.g. loading a file does an lmb >> >> > allocation but only for the purpose of avoiding a file being loaded >> >> > over U-Boot code/data. The allocation is dropped immediately after the >> >> > file is loaded. Within the bootm command, or when using standard boot, >> >> > this would be fairly easy to solve. >> >> > >> >> > Linux has renamed lmb to memblock. We should consider doing the same. >> >> > >> >> > 3. EFI >> >> > >> >> > EFI has its own memory-allocation tables. >> >> > >> >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' >> >> > function it can also do smaller allocations similar to malloc(), >> >> > although each one uses at least 4KB at present. >> >> > >> >> > EFI allocations do not go away when a boot fails. >> >> > >> >> > With EFI it is possible to add allocations post facto, in which case >> >> > they are added to the allocation table just as if the memory was >> >> > allocated with EFI to begin with. >> >> > >> >> > The EFI allocations and the lmb allocations use the same memory, so in >> >> > principle could conflict. >> >> > >> >> > EFI allocations are sometimes used to allocate internal U-Boot data as >> >> > well, if needed by the EFI app. For example, while efi_image_parse() >> >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs >> >> > in the app context and may need to access the memory after U-Boot has >> >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new >> >> > mapping as well. >> >> > >> >> > EFI memory has attributes, including what the memory is used for (to >> >> > some degree of granularity). See enum efi_memory_type and struct >> >> > efi_mem_desc. In the latter there are also attribute flags - whether >> >> > memory is cacheable, etc. >> >> > >> >> > EFI also has the x86 idea of 'conventional' memory, meaning (I >> >> > believe) that below 4GB that isn't reserved for the hardware/system. >> >> > This is meaningless, or at least confusing, on ARM systems. >> >> > >> >> > 4. reservations >> >> > >> >> > It is perhaps worth mentioning a fourth method of memory management, >> >> > where U-Boot reserves chunks of memory before relocation (in >> >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() >> >> > region, etc. >> >> > >> >> > >> >> > Problems >> >> > —--- >> >> > >> >> > There are no urgent problems, but here are some things that could be >> >> > improved: >> >> > >> >> > 1. EFI should attach most of its data structures to driver model. This >> >> > work has started, with the partition support, but more effort would >> >> > help. This would make it easier to see which memory is related to >> >> > devices and which is separate. >> >> > >> >> > 2. Some drivers do EFI reservations today, whether EFI is used for >> >> > booting or not (e.g. rockchip video rk_vop_probe()). >> >> >> >> Hello Simon, >> >> >> >> thank you for summarizing our discussion. >> >> >> >> Some U-Boot drivers including rockchip video inform the EFI sub-system >> >> that memory is reserved. >> >> >> >> Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are >> >> still used after ExitBootServices. mmio addresses have to be updated >> >> when Linux creates its virtual memory map. Currently this is done via >> >> efi_add_runtime_mmio(). A more UEFI style method would be to register an >> >> event handler for ExitBootServices() and use ConvertPointer() in the >> >> event handler. >> >> >> >> > >> >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. >> >> > armv8's struct mm_region) to EFI ones. >> >> >> >> U-Boot fails to set up RWX properties. E.g. the region where a FIT image >> >> is loaded should not be
Re: Proposal: U-Boot memory management
Hi Heinrich, On Mon, 18 Dec 2023 at 13:00, Heinrich Schuchardt wrote: > > > > Am 18. Dezember 2023 19:12:11 MEZ schrieb Simon Glass : > >Hi Heinrich, > > > >On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt wrote: > >> > >> On 12/16/23 19:01, Simon Glass wrote: > >> > Hi, > >> > > >> > This records my thoughts after a discussion with Ilias & Heinrich re > >> > memory allocation in U-Boot. > >> > > >> > 1. malloc() > >> > > >> > malloc() is used for programmatic memory allocation. It allows memory > >> > to be freed. It is not designed for very large allocations (e.g. a > >> > 10MB kernel or 100MB ramdisk). > >> > > >> > 2. lmb > >> > > >> > lmb is used for large blocks of memory, such as those needed for a > >> > kernel or ramdisk. Allocation is only transitory, for the purposes of > >> > loading some images and booting. If the boot fails, then all lmb > >> > allocations go away. > >> > > >> > lmb is set up by getting all available memory and then removing what > >> > is used by U-Boot (code, data, malloc() space, etc.) > >> > > >> > lmb reservations have a few flags so that areas of memory can be > >> > provided with attributes > >> > > >> > There are some corner cases...e.g. loading a file does an lmb > >> > allocation but only for the purpose of avoiding a file being loaded > >> > over U-Boot code/data. The allocation is dropped immediately after the > >> > file is loaded. Within the bootm command, or when using standard boot, > >> > this would be fairly easy to solve. > >> > > >> > Linux has renamed lmb to memblock. We should consider doing the same. > >> > > >> > 3. EFI > >> > > >> > EFI has its own memory-allocation tables. > >> > > >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > >> > function it can also do smaller allocations similar to malloc(), > >> > although each one uses at least 4KB at present. > >> > > >> > EFI allocations do not go away when a boot fails. > >> > > >> > With EFI it is possible to add allocations post facto, in which case > >> > they are added to the allocation table just as if the memory was > >> > allocated with EFI to begin with. > >> > > >> > The EFI allocations and the lmb allocations use the same memory, so in > >> > principle could conflict. > >> > > >> > EFI allocations are sometimes used to allocate internal U-Boot data as > >> > well, if needed by the EFI app. For example, while efi_image_parse() > >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > >> > in the app context and may need to access the memory after U-Boot has > >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > >> > mapping as well. > >> > > >> > EFI memory has attributes, including what the memory is used for (to > >> > some degree of granularity). See enum efi_memory_type and struct > >> > efi_mem_desc. In the latter there are also attribute flags - whether > >> > memory is cacheable, etc. > >> > > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I > >> > believe) that below 4GB that isn't reserved for the hardware/system. > >> > This is meaningless, or at least confusing, on ARM systems. > >> > > >> > 4. reservations > >> > > >> > It is perhaps worth mentioning a fourth method of memory management, > >> > where U-Boot reserves chunks of memory before relocation (in > >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > >> > region, etc. > >> > > >> > > >> > Problems > >> > —--- > >> > > >> > There are no urgent problems, but here are some things that could be > >> > improved: > >> > > >> > 1. EFI should attach most of its data structures to driver model. This > >> > work has started, with the partition support, but more effort would > >> > help. This would make it easier to see which memory is related to > >> > devices and which is separate. > >> > > >> > 2. Some drivers do EFI reservations today, whether EFI is used for > >> > booting or not (e.g. rockchip video rk_vop_probe()). > >> > >> Hello Simon, > >> > >> thank you for summarizing our discussion. > >> > >> Some U-Boot drivers including rockchip video inform the EFI sub-system > >> that memory is reserved. > >> > >> Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are > >> still used after ExitBootServices. mmio addresses have to be updated > >> when Linux creates its virtual memory map. Currently this is done via > >> efi_add_runtime_mmio(). A more UEFI style method would be to register an > >> event handler for ExitBootServices() and use ConvertPointer() in the > >> event handler. > >> > >> > > >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > >> > armv8's struct mm_region) to EFI ones. > >> > >> U-Boot fails to set up RWX properties. E.g. the region where a FIT image > >> is loaded should not be executable. > >> > >> > > >> > 4. EFI duplicates some code from bootm, some of which relates to > >> > memory allocation (e.g. FDT fixup). > >> > >> Fixup code is not duplicated but i
Re: Proposal: U-Boot memory management
Am 18. Dezember 2023 19:12:11 MEZ schrieb Simon Glass : >Hi Heinrich, > >On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt wrote: >> >> On 12/16/23 19:01, Simon Glass wrote: >> > Hi, >> > >> > This records my thoughts after a discussion with Ilias & Heinrich re >> > memory allocation in U-Boot. >> > >> > 1. malloc() >> > >> > malloc() is used for programmatic memory allocation. It allows memory >> > to be freed. It is not designed for very large allocations (e.g. a >> > 10MB kernel or 100MB ramdisk). >> > >> > 2. lmb >> > >> > lmb is used for large blocks of memory, such as those needed for a >> > kernel or ramdisk. Allocation is only transitory, for the purposes of >> > loading some images and booting. If the boot fails, then all lmb >> > allocations go away. >> > >> > lmb is set up by getting all available memory and then removing what >> > is used by U-Boot (code, data, malloc() space, etc.) >> > >> > lmb reservations have a few flags so that areas of memory can be >> > provided with attributes >> > >> > There are some corner cases...e.g. loading a file does an lmb >> > allocation but only for the purpose of avoiding a file being loaded >> > over U-Boot code/data. The allocation is dropped immediately after the >> > file is loaded. Within the bootm command, or when using standard boot, >> > this would be fairly easy to solve. >> > >> > Linux has renamed lmb to memblock. We should consider doing the same. >> > >> > 3. EFI >> > >> > EFI has its own memory-allocation tables. >> > >> > Like lmb, EFI is able to deal with large allocations. But via a 'pool' >> > function it can also do smaller allocations similar to malloc(), >> > although each one uses at least 4KB at present. >> > >> > EFI allocations do not go away when a boot fails. >> > >> > With EFI it is possible to add allocations post facto, in which case >> > they are added to the allocation table just as if the memory was >> > allocated with EFI to begin with. >> > >> > The EFI allocations and the lmb allocations use the same memory, so in >> > principle could conflict. >> > >> > EFI allocations are sometimes used to allocate internal U-Boot data as >> > well, if needed by the EFI app. For example, while efi_image_parse() >> > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs >> > in the app context and may need to access the memory after U-Boot has >> > exited. Also efi_smbios.c uses allocate_pages() and then adds a new >> > mapping as well. >> > >> > EFI memory has attributes, including what the memory is used for (to >> > some degree of granularity). See enum efi_memory_type and struct >> > efi_mem_desc. In the latter there are also attribute flags - whether >> > memory is cacheable, etc. >> > >> > EFI also has the x86 idea of 'conventional' memory, meaning (I >> > believe) that below 4GB that isn't reserved for the hardware/system. >> > This is meaningless, or at least confusing, on ARM systems. >> > >> > 4. reservations >> > >> > It is perhaps worth mentioning a fourth method of memory management, >> > where U-Boot reserves chunks of memory before relocation (in >> > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() >> > region, etc. >> > >> > >> > Problems >> > —--- >> > >> > There are no urgent problems, but here are some things that could be >> > improved: >> > >> > 1. EFI should attach most of its data structures to driver model. This >> > work has started, with the partition support, but more effort would >> > help. This would make it easier to see which memory is related to >> > devices and which is separate. >> > >> > 2. Some drivers do EFI reservations today, whether EFI is used for >> > booting or not (e.g. rockchip video rk_vop_probe()). >> >> Hello Simon, >> >> thank you for summarizing our discussion. >> >> Some U-Boot drivers including rockchip video inform the EFI sub-system >> that memory is reserved. >> >> Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are >> still used after ExitBootServices. mmio addresses have to be updated >> when Linux creates its virtual memory map. Currently this is done via >> efi_add_runtime_mmio(). A more UEFI style method would be to register an >> event handler for ExitBootServices() and use ConvertPointer() in the >> event handler. >> >> > >> > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. >> > armv8's struct mm_region) to EFI ones. >> >> U-Boot fails to set up RWX properties. E.g. the region where a FIT image >> is loaded should not be executable. >> >> > >> > 4. EFI duplicates some code from bootm, some of which relates to >> > memory allocation (e.g. FDT fixup). >> >> Fixup code is not duplicated but invoked via image_setup_libfdt(). >> >> > >> > 5. EFI code is used even if EFI is never used to boot >> >> >> * Only a minimum initialization of the EFI sub-system happens in >> efi_init_early(). >> * Some EFI code is called when probing block devices because we wanted >> the EFI and the dm part to be integrated.
Re: Proposal: U-Boot memory management
Hi Heinrich, On Sat, 16 Dec 2023 at 12:04, Heinrich Schuchardt wrote: > > On 12/16/23 19:01, Simon Glass wrote: > > Hi, > > > > This records my thoughts after a discussion with Ilias & Heinrich re > > memory allocation in U-Boot. > > > > 1. malloc() > > > > malloc() is used for programmatic memory allocation. It allows memory > > to be freed. It is not designed for very large allocations (e.g. a > > 10MB kernel or 100MB ramdisk). > > > > 2. lmb > > > > lmb is used for large blocks of memory, such as those needed for a > > kernel or ramdisk. Allocation is only transitory, for the purposes of > > loading some images and booting. If the boot fails, then all lmb > > allocations go away. > > > > lmb is set up by getting all available memory and then removing what > > is used by U-Boot (code, data, malloc() space, etc.) > > > > lmb reservations have a few flags so that areas of memory can be > > provided with attributes > > > > There are some corner cases...e.g. loading a file does an lmb > > allocation but only for the purpose of avoiding a file being loaded > > over U-Boot code/data. The allocation is dropped immediately after the > > file is loaded. Within the bootm command, or when using standard boot, > > this would be fairly easy to solve. > > > > Linux has renamed lmb to memblock. We should consider doing the same. > > > > 3. EFI > > > > EFI has its own memory-allocation tables. > > > > Like lmb, EFI is able to deal with large allocations. But via a 'pool' > > function it can also do smaller allocations similar to malloc(), > > although each one uses at least 4KB at present. > > > > EFI allocations do not go away when a boot fails. > > > > With EFI it is possible to add allocations post facto, in which case > > they are added to the allocation table just as if the memory was > > allocated with EFI to begin with. > > > > The EFI allocations and the lmb allocations use the same memory, so in > > principle could conflict. > > > > EFI allocations are sometimes used to allocate internal U-Boot data as > > well, if needed by the EFI app. For example, while efi_image_parse() > > uses malloc(), efi_var_mem.c uses EFI allocations since the code runs > > in the app context and may need to access the memory after U-Boot has > > exited. Also efi_smbios.c uses allocate_pages() and then adds a new > > mapping as well. > > > > EFI memory has attributes, including what the memory is used for (to > > some degree of granularity). See enum efi_memory_type and struct > > efi_mem_desc. In the latter there are also attribute flags - whether > > memory is cacheable, etc. > > > > EFI also has the x86 idea of 'conventional' memory, meaning (I > > believe) that below 4GB that isn't reserved for the hardware/system. > > This is meaningless, or at least confusing, on ARM systems. > > > > 4. reservations > > > > It is perhaps worth mentioning a fourth method of memory management, > > where U-Boot reserves chunks of memory before relocation (in > > board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() > > region, etc. > > > > > > Problems > > —--- > > > > There are no urgent problems, but here are some things that could be > > improved: > > > > 1. EFI should attach most of its data structures to driver model. This > > work has started, with the partition support, but more effort would > > help. This would make it easier to see which memory is related to > > devices and which is separate. > > > > 2. Some drivers do EFI reservations today, whether EFI is used for > > booting or not (e.g. rockchip video rk_vop_probe()). > > Hello Simon, > > thank you for summarizing our discussion. > > Some U-Boot drivers including rockchip video inform the EFI sub-system > that memory is reserved. > > Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are > still used after ExitBootServices. mmio addresses have to be updated > when Linux creates its virtual memory map. Currently this is done via > efi_add_runtime_mmio(). A more UEFI style method would be to register an > event handler for ExitBootServices() and use ConvertPointer() in the > event handler. > > > > > 3. U-Boot doesn't really map arch-specific memory attributes (e.g. > > armv8's struct mm_region) to EFI ones. > > U-Boot fails to set up RWX properties. E.g. the region where a FIT image > is loaded should not be executable. > > > > > 4. EFI duplicates some code from bootm, some of which relates to > > memory allocation (e.g. FDT fixup). > > Fixup code is not duplicated but invoked via image_setup_libfdt(). > > > > > 5. EFI code is used even if EFI is never used to boot > > > * Only a minimum initialization of the EFI sub-system happens in > efi_init_early(). > * Some EFI code is called when probing block devices because we wanted > the EFI and the dm part to be integrated. > * The rest of the initialization in efi_init_obj_list() is only invoked > if an EFI command is invoked. > > > > > 6. EFI allocations can result in the same memory being used as has >
Re: Proposal: U-Boot memory management
On 12/16/23 19:01, Simon Glass wrote: Hi, This records my thoughts after a discussion with Ilias & Heinrich re memory allocation in U-Boot. 1. malloc() malloc() is used for programmatic memory allocation. It allows memory to be freed. It is not designed for very large allocations (e.g. a 10MB kernel or 100MB ramdisk). 2. lmb lmb is used for large blocks of memory, such as those needed for a kernel or ramdisk. Allocation is only transitory, for the purposes of loading some images and booting. If the boot fails, then all lmb allocations go away. lmb is set up by getting all available memory and then removing what is used by U-Boot (code, data, malloc() space, etc.) lmb reservations have a few flags so that areas of memory can be provided with attributes There are some corner cases...e.g. loading a file does an lmb allocation but only for the purpose of avoiding a file being loaded over U-Boot code/data. The allocation is dropped immediately after the file is loaded. Within the bootm command, or when using standard boot, this would be fairly easy to solve. Linux has renamed lmb to memblock. We should consider doing the same. 3. EFI EFI has its own memory-allocation tables. Like lmb, EFI is able to deal with large allocations. But via a 'pool' function it can also do smaller allocations similar to malloc(), although each one uses at least 4KB at present. EFI allocations do not go away when a boot fails. With EFI it is possible to add allocations post facto, in which case they are added to the allocation table just as if the memory was allocated with EFI to begin with. The EFI allocations and the lmb allocations use the same memory, so in principle could conflict. EFI allocations are sometimes used to allocate internal U-Boot data as well, if needed by the EFI app. For example, while efi_image_parse() uses malloc(), efi_var_mem.c uses EFI allocations since the code runs in the app context and may need to access the memory after U-Boot has exited. Also efi_smbios.c uses allocate_pages() and then adds a new mapping as well. EFI memory has attributes, including what the memory is used for (to some degree of granularity). See enum efi_memory_type and struct efi_mem_desc. In the latter there are also attribute flags - whether memory is cacheable, etc. EFI also has the x86 idea of 'conventional' memory, meaning (I believe) that below 4GB that isn't reserved for the hardware/system. This is meaningless, or at least confusing, on ARM systems. 4. reservations It is perhaps worth mentioning a fourth method of memory management, where U-Boot reserves chunks of memory before relocation (in board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() region, etc. Problems —--- There are no urgent problems, but here are some things that could be improved: 1. EFI should attach most of its data structures to driver model. This work has started, with the partition support, but more effort would help. This would make it easier to see which memory is related to devices and which is separate. 2. Some drivers do EFI reservations today, whether EFI is used for booting or not (e.g. rockchip video rk_vop_probe()). Hello Simon, thank you for summarizing our discussion. Some U-Boot drivers including rockchip video inform the EFI sub-system that memory is reserved. Furthermore drivers like arch/arm/mach-bcm283x/reset.c exist that are still used after ExitBootServices. mmio addresses have to be updated when Linux creates its virtual memory map. Currently this is done via efi_add_runtime_mmio(). A more UEFI style method would be to register an event handler for ExitBootServices() and use ConvertPointer() in the event handler. 3. U-Boot doesn't really map arch-specific memory attributes (e.g. armv8's struct mm_region) to EFI ones. U-Boot fails to set up RWX properties. E.g. the region where a FIT image is loaded should not be executable. 4. EFI duplicates some code from bootm, some of which relates to memory allocation (e.g. FDT fixup). Fixup code is not duplicated but invoked via image_setup_libfdt(). 5. EFI code is used even if EFI is never used to boot * Only a minimum initialization of the EFI sub-system happens in efi_init_early(). * Some EFI code is called when probing block devices because we wanted the EFI and the dm part to be integrated. * The rest of the initialization in efi_init_obj_list() is only invoked if an EFI command is invoked. 6. EFI allocations can result in the same memory being used as has already been allocated by lmb. Users may load files which overwrite memory allocated by EFI. The most worrisome issue is that EFI may allocate memory where U-Boot has loaded files like initrd as the EFI sub-system is never informed which memory is used for files. Loading files should not be possible without creating a memory reservation that becomes visible to the EFI sub-system. Lifetime We have three different memory allocators with different purposes
Proposal: U-Boot memory management
Hi, This records my thoughts after a discussion with Ilias & Heinrich re memory allocation in U-Boot. 1. malloc() malloc() is used for programmatic memory allocation. It allows memory to be freed. It is not designed for very large allocations (e.g. a 10MB kernel or 100MB ramdisk). 2. lmb lmb is used for large blocks of memory, such as those needed for a kernel or ramdisk. Allocation is only transitory, for the purposes of loading some images and booting. If the boot fails, then all lmb allocations go away. lmb is set up by getting all available memory and then removing what is used by U-Boot (code, data, malloc() space, etc.) lmb reservations have a few flags so that areas of memory can be provided with attributes There are some corner cases...e.g. loading a file does an lmb allocation but only for the purpose of avoiding a file being loaded over U-Boot code/data. The allocation is dropped immediately after the file is loaded. Within the bootm command, or when using standard boot, this would be fairly easy to solve. Linux has renamed lmb to memblock. We should consider doing the same. 3. EFI EFI has its own memory-allocation tables. Like lmb, EFI is able to deal with large allocations. But via a 'pool' function it can also do smaller allocations similar to malloc(), although each one uses at least 4KB at present. EFI allocations do not go away when a boot fails. With EFI it is possible to add allocations post facto, in which case they are added to the allocation table just as if the memory was allocated with EFI to begin with. The EFI allocations and the lmb allocations use the same memory, so in principle could conflict. EFI allocations are sometimes used to allocate internal U-Boot data as well, if needed by the EFI app. For example, while efi_image_parse() uses malloc(), efi_var_mem.c uses EFI allocations since the code runs in the app context and may need to access the memory after U-Boot has exited. Also efi_smbios.c uses allocate_pages() and then adds a new mapping as well. EFI memory has attributes, including what the memory is used for (to some degree of granularity). See enum efi_memory_type and struct efi_mem_desc. In the latter there are also attribute flags - whether memory is cacheable, etc. EFI also has the x86 idea of 'conventional' memory, meaning (I believe) that below 4GB that isn't reserved for the hardware/system. This is meaningless, or at least confusing, on ARM systems. 4. reservations It is perhaps worth mentioning a fourth method of memory management, where U-Boot reserves chunks of memory before relocation (in board_init_f.c), e.g. for the framebuffer, U-Boot code, the malloc() region, etc. Problems —--- There are no urgent problems, but here are some things that could be improved: 1. EFI should attach most of its data structures to driver model. This work has started, with the partition support, but more effort would help. This would make it easier to see which memory is related to devices and which is separate. 2. Some drivers do EFI reservations today, whether EFI is used for booting or not (e.g. rockchip video rk_vop_probe()). 3. U-Boot doesn't really map arch-specific memory attributes (e.g. armv8's struct mm_region) to EFI ones. 4. EFI duplicates some code from bootm, some of which relates to memory allocation (e.g. FDT fixup). 5. EFI code is used even if EFI is never used to boot 6. EFI allocations can result in the same memory being used as has already been allocated by lmb. Users may load files which overwrite memory allocated by EFI. Lifetime We have three different memory allocators with different purposes. Can we unify them a little? Within U-Boot: - malloc() space lives forever - lmb lives while setting out images for booting - EFI (mostly) lives while booting an EFI app In practice, EFI is set up early in U-Boot. Some of this is necessary, some not. EFI allocations stay around forever. This works OK since large allocations are normally not done in EFI, so memory isn't really consumed to any great degree by the boot process. What happens to EFI allocations if the app returns? They are still present, in case another app is run. This seems fine. API –-- Can we unify some APIs? It should be possible to use lmb for large EFI memory allocations, so long as they are only needed for booting. We effectively do this today, since EFI does not manage the arrangement of loaded images in memory. for the most part. It would not make sense to use EFI allocation to replace lmb and malloc(), of course. Could we use a common (lower-level) API for allocation, used by both lmb and EFI? They do have some similarities. However they have different lifetime constraints (EFI allocations are never dropped, unlikely lmb). ** Overall, it seems that the existence of memory allocation in boot-time services has created confusion. Memory allocation is muddled, with both U-Boot code and boot-time services calling the same memory allocator. This jus
