Re: [coreboot] Rebuilding coreboot image generation

[Peter Stuge]

Julius Werner wrote:
> > When designing CBFS we definitely did have fixed locations in mind.
> 
> Sure, I'm definitely not trying to argue that... I'm just saying that
> I think this should change if we have a cleaner option. The old CBFS
> has done a good job for a long time, but it was designed for a very
> specific purpose (x86 ROMs, I believe even without IFDs?) and we are
> now trying to do many more things (ARM boards with different layout
> requirements, RO/RW split ROMs, individually updateable A/B copies of
> the same file sets) that it really doesn't support well.

ARM, flash sector alignment and individually updateable files is
already in CBFS.


> Let's face it, problems like "we need to control the order in which
> we add files because otherwise the fixed location placement breaks"
> are really ugly

They aren't caused by CBFS, but by cbfstool.


> > Can anyone see some concrete problems with Kevin's suggestion?
> 
> I actually think Kevin's suggestion was pretty much the same thing
> we're proposing anyway.

Fair enough. I would like this to first be implemented as he
suggested, then adding support to read all of them at once from a
file in cbfstool should be a very simple change, while cbfstool and
CBFS are still the main tools. I think it is very important to keep
existing tooling and existing data formats to stay compatibile and
not force people to learn more tooling than neccessary.


> it makes the layout very easy to read...

cbfstool print output is also very easy to read, and it's important
to me that this continues to be all that is neccessary to grok a
coreboot.rom file.


> The "little additional metadata" that Kevin said we needed *is* the
> FMAP. Think about it, how else would you store a list of regions with
> hardcoded bounds?

It seems like a very simple data structure (name, base, length) that
would be easy to store as a file within CBFS, so that a CBFS carries
its own metadata.


//Peter

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Julius Werner]

On Fri, Nov 20, 2015 at 4:18 PM, Patrick Georgi  wrote:
> So you think we're better off with one place to store the partition
> layout, a separate one to store what files belong where, and yet
> another one for the properties of those files, each with their own
> syntax and tooling?

Yes. I find it strange that you seem to imply this would make it more
complex or otherwise undesirable. Your proposal is to use a common
language, but spread the files that control all of this (including the
layout, which I think is the thing that really needs to be together in
one place to be readable) all over the place. I think we agree that
declarations of individual files should go in the different source
directories that relate to them, I just think that the layout should
stay centralized to be easily comprehensible and modifyable.

> When designing CBFS we definitely did have fixed locations in mind.

Sure, I'm definitely not trying to argue that... I'm just saying that
I think this should change if we have a cleaner option. The old CBFS
has done a good job for a long time, but it was designed for a very
specific purpose (x86 ROMs, I believe even without IFDs?) and we are
now trying to do many more things (ARM boards with different layout
requirements, RO/RW split ROMs, individually updateable A/B copies of
the same file sets) that it really doesn't support well. Let's face
it, problems like "we need to control the order in which we add files
because otherwise the fixed location placement breaks" are really ugly
and we should try to find a design that removes them completely.

> Can anyone see some concrete problems with Kevin's suggestion?

I actually think Kevin's suggestion was pretty much the same thing
we're proposing anyway. It's just that he suggested adding regions one
at a time per cbfstool invocation, while the .fmd format is designed
to read the complete region layout from a file at image creation
(which I think is nicer because it makes the layout very easy to
read... if you did the thing with commands, you'd effectively still
end up having a list of those region-defining commands in some
Makefile).

The "little additional metadata" that Kevin said we needed *is* the
FMAP. Think about it, how else would you store a list of regions with
hardcoded bounds? This whole thing really started as an idea for a
"CBFS partition layout"
(http://www.coreboot.org/User_talk:MrNuke/CBFS_Ideas), until we
realized that the difference between any "partition table" format we
could come up with and an FMAP was so negligible that we might as well
keep the established format with pre-existing tooling (most
importantly flashrom support).

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Patrick Georgi via coreboot]

2015-11-21 5:08 GMT+01:00 Peter Stuge :
> And it's certainly possible to implement fixed placement.

The main drawback of CBFS, pretty much from the start, was that it
creates a single chain of files, which makes it hard to implement
certain designs safely:

You need to be extra careful when adding files that are supposed to be
written to (eg memory training caches) to make sure that they start at
a flash block, and keep CBFS data structures out completely.

You can't enforce through its structures that some files aren't
"overwritten" by others by just reusing the file name, which is
relevant even for the fallback/normal scenario: On an x86 system you
have the bootblock at the top and the CBFS file chain moving bottom
up, so to keep "fallback" safe, you need to ensure that any "normal"
updates are written somewhere in the middle of the flash (and in a way
that the CBFS file chain is never broken, despite flash block
alignments).

With fmap, you simply have two (or more) separate chains that can span
whatever region is practical.

> We do not need to add *another* layer of complexity here, do we?
> We need to *eliminate* some of the complexity that has crept in.

fmap replaces the master header, which has its own issues, for example
its very x86-centric design that led to different interpretation of
its fields between coreboot, cbfstool and libpayload while adapting
things for non-x86.
The data structures are also reasonably simple: It's more or less a
list of offset/size/name triplets (plus some flag bits that we mostly
ignore). Any hierarchy we show in the fmap region descriptions merely
exists because fmap allows overlaps.

> Can anyone see some concrete problems with Kevin's suggestion?
It doesn't deal with the problem that each command has no visibility
into what's going to be added after it.

Some of the constraints that we need to support (fixed placement,
alignment, ...) can make the build break for a configuration that
would work fine when reordering command invocation. That's an issue we
ran into already. This is why I made the proposed flow work from a
complete view of what needs to happen, and I tried to keep it out of
the build system because people complained already that it's too
complex.

Given the opposition on the build description side, I'll punt that and
have make(1) deal with things (and ignore some duplication in
metadata). Both can still be improved later (or turn out not to be an
issue after all).


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Patrick Georgi via coreboot]

Let's extend your example so it becomes somewhat comparable to my proposal:

 HOST_FIRMWARE@0xff80 8M {
 SI_ALL 2M {
 SI_DESC 4K
 SI_ME 0x1ff000
 }
 SI_BIOS {
 RW_A 0xf {
 VBLOCK_A 64K
 RW_MAIN_A CBFS {
add-payload -n fallback/payload -f
/path/to/whereever/the/payload/resides.elf
add-stage -n fallback/ramstage -f $(obj)/cbfs/fallback/ramstage.elf
add-stage -n fallback/romstage -f
$(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000
}
 RW_FWID_A 0x40
 }
 RW_B 0xf {
 VBLOCK_B 64K
 RW_MAIN_B CBFS {
add-payload -n fallback/payload -f
/path/to/whereever/the/payload/resides.elf
add-stage -n fallback/ramstage -f $(obj)/cbfs/fallback/ramstage.elf
add-stage -n fallback/romstage -f
$(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000
}
RW_FWID_B 0x40
 }
 RO@4M {
 RO_VPD 16K
 FMAP@0x1 2K
 RO_FRID 0x40
 GBB@0x11000
 RO_MAIN CBFS {
add-payload -n fallback/payload -f
/path/to/whereever/the/payload/resides.elf
add-stage -n fallback/ramstage -f $(obj)/cbfs/fallback/ramstage.elf
add-stage -n fallback/romstage -f
$(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000
add-stage -n fallback/verstage -f
$(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000
add -n bootblock -f $(obj)/cbfs/fallback/bootblock.bin --base
-$(filesize $(obj)/cbfs/fallback/bootblock.bin)
}
}
 }
 }

And that for every mainboard, and not transferable to other flash
layout schemes.
I don't think this is looking better (or even just as good) as my
proposal, simple because of maintenance reasons.


Patrick

2015-11-06 6:55 GMT+01:00 Julius Werner :
>> What I'd like to see come out of this is a dts-like set of flash
>> descriptors, with the rules left in the makefiles. At the very least,
>> dts is not a new language.
>
> Just wanted to point out (without trying to usurp any of the ongoing
> discussion in that doc) that we already have an FMAP descriptor
> language that was implemented in cbfstool half a year ago, because I
> don't think that got much visibility upstream when it was merged.
> That's the FMD (flash map descriptor) format some of us are talking
> about in those comments, which was developed in anticipation of a
> build system redesign like this. It's aiming to describe FMAPs in a
> clean, flexible way without the cruft of a format (dts) that was never
> really meant for that purpose, and has some nice features (like
> variable-sized sections) built-in. The parser can be seen in
> util/cbfstool/fmd.c, and a sample file would look something like this
> (in general, '@offset' is the base offset to the parent section and
> the number after the space is the size):
>
> HOST_FIRMWARE@0xff80 8M {
> SI_ALL 2M {
> SI_DESC 4K
> SI_ME 0x1ff000
> }
> SI_BIOS {
> RW_A 0xf {
> VBLOCK_A 64K
> RW_MAIN_A
> RW_FWID_A 0x40
> }
> RW_B 0xf {
> VBLOCK_B 64K
> RW_MAIN_B
> RW_FWID_B 0x40
> }
> RO@4M {
> RO_VPD 16K
> FMAP@0x1 2K
> RO_FRID 0x40
> GBB@0x11000
> RO_MAIN 1M
> }
> }
> }



-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Julius Werner]

> Let's extend your example so it becomes somewhat comparable to my proposal:
> [...]
> And that for every mainboard, and not transferable to other flash
> layout schemes.
> I don't think this is looking better (or even just as good) as my
> proposal, simple because of maintenance reasons.

Sorry, I'm not quite sure how to read that... did you literally mean
that the cbfstool commands (or something equivalent) would be *in* the
.fmd file? That is certainly not what I meant.

Instead, I was suggesting to use well-known CBFS "categories" and have
another part of the system (e.g. the Makefiles) map files to those.
For example when something (a mainboard, an SoC, etc.) declares a
file, it would declare that this file belongs to the "pre-boot-split
CBFS" category or the "post-boot-split CBFS" category (or for all I
care you could also bind them directly to romstage/ramstage/etc. which
is I think closer to what you were proposing). Then the part deciding
the boot scheme ("vboot" and "traditional coreboot" for now) would
know that all "pre-boot-split" files would belong into the well-known
CBFS FMAP sections "FALLBACK" and "NORMAL" for traditional coreboot,
whereas "post-boot-split" would only go into "NORMAL" (and the vboot
version would instead know how to bind those categories to its
RO/RW_A/RW_B sections). The .fmd file should *only* define the layout,
and all the individual platform pieces that need to add files should
*only* declare something abstract about in which context those files
are used. Then something else (which is deciding the general boot
method) should define how to bind those two together.

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Patrick Georgi via coreboot]

So you think we're better off with one place to store the partition
layout, a separate one to store what files belong where, and yet
another one for the properties of those files, each with their own
syntax and tooling?

2015-11-21 1:08 GMT+01:00 Julius Werner :
>> Let's extend your example so it becomes somewhat comparable to my proposal:
>> [...]
>> And that for every mainboard, and not transferable to other flash
>> layout schemes.
>> I don't think this is looking better (or even just as good) as my
>> proposal, simple because of maintenance reasons.
>
> Sorry, I'm not quite sure how to read that... did you literally mean
> that the cbfstool commands (or something equivalent) would be *in* the
> .fmd file? That is certainly not what I meant.
>
> Instead, I was suggesting to use well-known CBFS "categories" and have
> another part of the system (e.g. the Makefiles) map files to those.
> For example when something (a mainboard, an SoC, etc.) declares a
> file, it would declare that this file belongs to the "pre-boot-split
> CBFS" category or the "post-boot-split CBFS" category (or for all I
> care you could also bind them directly to romstage/ramstage/etc. which
> is I think closer to what you were proposing). Then the part deciding
> the boot scheme ("vboot" and "traditional coreboot" for now) would
> know that all "pre-boot-split" files would belong into the well-known
> CBFS FMAP sections "FALLBACK" and "NORMAL" for traditional coreboot,
> whereas "post-boot-split" would only go into "NORMAL" (and the vboot
> version would instead know how to bind those categories to its
> RO/RW_A/RW_B sections). The .fmd file should *only* define the layout,
> and all the individual platform pieces that need to add files should
> *only* declare something abstract about in which context those files
> are used. Then something else (which is deciding the general boot
> method) should define how to bind those two together.



-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Peter Stuge]

Patrick Georgi wrote:
> I'm looking for an approach to make building Chrome OS style coreboot
> images easier to do with regular coreboot tools, instead of the rather
> large post-processing pipeline we have in the Chrome OS build system.

When designing CBFS we definitely did have fixed locations in mind.

Despite its name, CBFS is not intended to function as an opaque file system.

And it's certainly possible to implement fixed placement.

We do not need to add *another* layer of complexity here, do we?

We need to *eliminate* some of the complexity that has crept in.

FMAP merely re-invents a block layer, on top of which is supposed to
sit what seems to be perceived as a file system layer.

Come on - you guys can do better than that. And I think CBFS can do
better than that.


Can anyone see some concrete problems with Kevin's suggestion?

//Peter

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Patrick Georgi via coreboot]

2015-11-09 21:47 GMT+01:00 Julius Werner :
> If Intel's architectural requirements force us to have a component
> that must be linked at an exact spot in the image, I think it would
> only be consistent to use the tool that we have which is meant for
> these sorts of things... FMAP.
It must be linked to the location it was relocated to. That could even
change over time. Just like x86s execute-in-place romstage.

Updates may end up at a different location, and they will likely have
a different size. For all we know, updates may grow beyond the space
we reserve for FSP in FMAP (which is read-only on Chromebooks). It
would be unfortunate to have empty space, just in the wrong region,
and run into problems doing an update because of that.

I don't think the layout constraints we can impose on files in CBFS
are much of a problem, and there's not much overlap: As I understand
things, FMAP exists for higher-order partitioning; for things that the
current instance of coreboot need not to care about, like a separate
copy of coreboot in flash, or chipset data that is none of the CPU
firmware's business (ME and the like). FMAP is also suitable for
writable fixed-size sections (like MRC cache) that we want to keep far
away from the CBFS chain.

The other reason (besides file constraints) for this proposal is to
enable the payload's build system to describe the files they need to
add to CBFS themselves (and without knowing which FMAP regions the
payload needs to end up in). depthcharge definitely needs something
along those lines so we can get rid of that spaghetti code that is
bundle_firmware.py. I think grub2 (and to some degree, seabios) could
also benefit from dealing with their configuration files themselves in
a structured way.


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Julius Werner]

> Again then this becomes an inconsistency on Chrome OS. FSP lives in
> RO, RW-A, and RW-B regions. RW-A and RW-B will be CBFS. So do make it
> it partly CBFS and partly FMAP such that the code is consistent to use
> FMAP APIs throughout? In my opinion that isn't appropriate. From a
> "change this little bit of code here" thing it's doable. But from
> looking at source code I standpoint such it's inconsistent. Sure FMAP
> would be used in certain places, but I'd argue those are in the
> infrastructure parts that many people porting things don't deal with.
> I don't want people to head scratch where something should live and
> then figure out 1. how to access it and 2. how to add it properly. The
> latter is still not plumbed in anywhere in the build system.

Okay, I guess we disagree on that one, then. In my opinion, the
biggest problem and inconsistency here is that we have a CBFS file
that needs to be placed at an exact offset (breaking the whole concept
of CBFS as a "file system" that can transparently manage its space
allocation). I think this is something that was done in the past for
lack of a better option, but that can and should be fixed when a
better tool (FMAP) is available. In my recollection/perspective, this
was the biggest reason we decided to combine CBFS and FMAP in the same
image in the first place (as opposed to porting all the Chrome OS /
vboot tools over to access CBFS and have the whole Chrome OS image be
a single (enhanced) CBFS)... because we acknowledged that FMAP's
strengths (global lookup table, rigid regions that can be exactly
aligned to custom offsets or erase blocks) are completely different
and complementary to CBFS's strengths (organizing many small files in
a compact, easy to manage format with automatic, opaque layout).

If Intel's architectural requirements force us to have a component
that must be linked at an exact spot in the image, I think it would
only be consistent to use the tool that we have which is meant for
these sorts of things... FMAP. It makes the intention (of placing it
*exactly there*) very explicit, and it frees up CBFS from weird
requirements that you wouldn't expect (e.g. the RO CBFS must encompass
that region, you need to pay attention to the order you add files in,
etc.). If that makes the code that handles it more complicated that's
unfortunate, and we should try to mitigate it where we can (maybe hide
differences with an extra FSP access API, or have all chipsets keep it
in its own FMAP section (maybe even in RW?) even though only some
really need that, or find a way / ask Intel to separate the
location-dependent part out into a separate blob)... but it still
seems like a smaller price to me than keeping the separation of duties
between FMAP and CBFS muddied and imposing all of those weird
restrictions on the latter for this one special case.

I should probably stop arguing this because I don't even know the
Intel code this is for... if you and all other x86 developers think
the FSP really must go in CBFS I'll shut up. It's also really only
tangentially related to the original discussion about the build system
architecture... but I brought it up because some of the benefits
Patrick's approach is trying to achieve (being able to better control
CBFS file ordering) should be irrelevant in my opinion, and only even
exist because of this problem.

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Kevin O'Connor]

On Thu, Nov 05, 2015 at 12:42:02PM +0100, Patrick Georgi wrote:
> Hey coreboot folks,
> 
> I'm looking for an approach to make building Chrome OS style coreboot
> images easier to do with regular coreboot tools, instead of the rather
> large post-processing pipeline we have in the Chrome OS build system.
> 
> The rationale is that we also push the Chrome OS capabilities (eg.
> verified boot) upstream, and actually using them shouldn't depend on
> checking out yet another custom build environment.
> 
> I wrote a proposal on how to do that, which can be found and commented
> at 
> https://docs.google.com/document/d/1o2bFl5HCHDFPccQsOwa-75A8TWojjFiGK3r0yeIc7Vo/edit

It seems to me that similar functionality could be obtained by
enhancing (or replacing) cbfstool.  If possible, that seems simpler
than adding a new tool, a new custom language, and a new layer to the
build.

For example, instead of a "Chipset manifest" file, I think one could
run commands like:

$ cbfs2tool coreboot.rom add-region "IFD" --start=0 --end=4K
$ cbfs2tool coreboot.rom add-raw build/ifd.bin --region="IFD" --align=bottom 
--empty=0xff
$ cbfs2tool coreboot.rom add-region "ME" --start=4K --end=4M
...

The above would require cbfstool (or its replacement) to track some
additional metadata (either in the rom itself or in a similarly named
file).  However, that seems simpler than introducing a new tool.

Just my $.02
-Kevin

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Julius Werner]

Whoops... sorry, I meant to keep the discussion on the list.

On Fri, Nov 6, 2015 at 3:17 PM, Julius Werner  wrote:
>> That suggestion leads to different software using fmap vs CBFS for
>> finding and locating specifics assets that are required for booting.
>> In fact that would fundamentally break x86 w.r.t. common stage loading
>> because of the xip requirement. I don't think that's a good thing.
>
> Hmm... I don't really understand where you're seeing problems
> regarding XIP there. I've got to admit I don't understand that area
> very well though, especially since a lot of that changed recently.
>
> Just to be clear, I didn't mean to say that all XIP stages should go
> into separate FMAP sections... those can stay in CBFS as they
> currently are. I didn't intend to include everything XIP with "needs
> to be placed exactly"... because they theoretically can be placed
> anywhere (right? or are there limits?), they just need to be
> explicitly linked to that offset once it has been decided. I know that
> we used to link the romstage to offset 0x0, take the resulting binary
> size of that to help layout CBFS, then link it again with the now
> known offset... seems like all of that moved into cbfstool now, but in
> theory it should still work just as well (regardless of the order the
> files get added in and where the stage ends up being placed in the
> end), right?
>
> I think the case of that microcode blob is different because it seems
> to need an exact offset (at least that's what the whole hardcoded
> CONFIG_CPU_MICROCODE_CBFS_LOC thing looks like to me). Same for the
> bootblock, which always needs to be placed into the reset vector (and
> hasn't previously been a CBFS "file" anyway). Does that make it
> clearer or did I misunderstand your concerns?
>
>> Those requirements are not necessarily known until link time.
>
> Which ones are you thinking of in particular?
> CONFIG_CPU_MICROCODE_CBFS_LOC seems to be hardcoded right now (unless
> I misread how that works). The bootblock size is determined at link
> time, but I don't think it needs to be... i.e. I think it's fine to
> just reserve a certain bootblock FMAP section with the maximum size we
> expect to need, and waste a few K of space if necessary. We could also
> make it configurable with a Kconfig to allow more fine tuning by
> preprocessing FMD files. I think that would be much more reasonable
> and easier to work with than the current model of mushing the
> bootblock somewhere in the CBFS area in a way that even cbfstool
> cannot modify/extract again after the fact. It would also allow us to
> further simplify the CBFS format (removing the need of the master
> header and the concept that only a certain part of the full CBFS image
> is space available for files... which was another goal of the CBFS
> redesign earlier this year that had unfortunately not been finished).
>
>> But some files need special flags. One of the harder things is
>> identifying all the assets to fit into those 2 buckets. I personally
>> think one stop shopping in a single file is way better than digging
>> through Makefiles.
>
> Well, but I don't think Patrick's proposal is one-stop-shopping
> either... rather, it splits both layout and file information across
> multiple manifests in the same way that CBFS file rules are already
> split across our Makefiles.
>
> Personally, I'm more concerned with keeping the layout in one place.
> For the decision of which files are included I think the current split
> is reasonable... especially since you need to add extra files based on
> chipset or board much more often than you need to add extra sections,
> and in an "ideal" CBFS (where no file has magic placement
> requirements) it doesn't really matter that much how many files get
> included in the end (as long as the total size is sufficient which
> usually doesn't seem to be a problem).
>
> But I'd be open to discuss how to handle the files more centrally if
> you have a good idea for that... I just think that it should be
> separated from the (central) FMAP layout.

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Aaron Durbin]

On Fri, Nov 6, 2015 at 4:09 PM, Julius Werner  wrote:
> Okay, looks like we're continuing the discussion over here after all.
>
>> Rationale: The build system is rather stupid about the order in which
>> to add files, and while adding a proper scheduler that prevents
>> unsuitable ordering (eg adding
>> flexibly laid out files first, which may block files with stricter
>> requirements later), people are rightfully complaining about too much
>> magic in the build system already.
>> (something to work on, but not as part of this effort)
>
> I assume you're talking about things like crosreview.com/289491 as an
> example for strict requirements. I'm not really familiar with the
> details there so please correct me if I'm wrong, but if this is a blob
> that needs to be placed in one exact offset (which it looks like?),
> then I think we're approaching it from the completely wrong angle
> trying to stuff it into CBFS anyway. Maybe it was originally
> shoehorned into CBFS because that was the only thing we had at the
> time, but if we're rearchitecting things from the ground up to make
> FMAP a first class citizen this seems to be the right time to
> reconsider that.
>
> The whole point of FMAP is to allow exact placement in the ROM,
> whereas the whole point of CBFS is to conveniently group files with
> variable lengths and no strict placement requirements together in an
> easily manageable way. I think the only placement restriction CBFS
> should support in that new design is a (mild) minimum alignment, and
> anything that needs to be placed more exactly than that (including the
> bootblock and magic Intel blobs) should instead get its own FMAP
> section.
>

That suggestion leads to different software using fmap vs CBFS for
finding and locating specifics assets that are required for booting.
In fact that would fundamentally break x86 w.r.t. common stage loading
because of the xip requirement. I don't think that's a good thing.


>>> HOST_FIRMWARE@0xff80 8M {
>>> SI_ALL 2M {
>>> SI_DESC 4K
>>> SI_ME 0x1ff000
>>> }
>>> SI_BIOS {
>>> RW_A 0xf {
>>> VBLOCK_A 64K
>>> RW_MAIN_A
>>> RW_FWID_A 0x40
>>> }
>>> RW_B 0xf {
>>> VBLOCK_B 64K
>>> RW_MAIN_B
>>> RW_FWID_B 0x40
>>> }
>>> RO@4M {
>>> RO_VPD 16K
>>> FMAP@0x1 2K
>>> RO_FRID 0x40
>>> GBB@0x11000
>>> RO_MAIN 1M
>>> }
>>> }
>>> }
>> Just wanted to spell out the issues with this approach:
>> First, it doesn't tell you where to put various build collateral. Is
>> ramstage.elf to be added to SI_ME?
>
> This was just an example of an existing (and I think oversimplified)
> Chrome OS ROM that I pulled from an old discussion, to demonstrate the
> format. We totally can (and should) redesign and rename some existing
> parts for the new layouts, and we could also add new features to the
> existing FMD parser if we deem them useful (e.g. some sort of
> inclusion or macro mechanism to help deduplicate more). Stuff like
> RO_MAIN and RW_MAIN_B are probably better named RO_CBFS and RW_CBFS_B,
> and I think the bootblock should be split out into its own section
> (since it is not a proper CBFS file, and it has strict placement
> requirements).

Those requirements are not necessarily known until link time.

>
> In my current vision, you would then group all of coreboot's output
> files (both stages and separate files) into "pre-boot-path-split" and
> "post-boot-path-split" groups. I think the existing Makefile system
> would be fine for that, but we could discuss that part separately.
> Then, whatever controls the Chrome OS specific parts of the build
> process (I think a Makefile would be enough, but again, that's a
> separate issue from the ROM layout itself) will 'cbfstool add -r
> RO_CBFS' the pre-boot-path-split files, 'cbfstool add -r
> RO_CBFS,RW_CBFS_A,RW_CBFS_B' the post-boot-path-split files, and
> finally 'cbfstool add -r RW_CBFS_A,RW_CBFS_B' the EC-RW images (which
> are Chrome OS specific and should therefore probably only come into
> play at that time).

But some files need special flags. One of the harder things is
identifying all the assets to fit into those 2 buckets. I personally
think one stop shopping in a single file is way better than digging
through Makefiles.

>
> The advantage is that the layout (which may need to be very flexible
> due to all the weird corner cases that we keep encountering, and may
> be duplicated a little bit in practice) only tells you *where* the
> well-known sections are in a single, very easily readable and
> modifiable place that can be parsed at a glance, while another (more
> 

Re: [coreboot] Rebuilding coreboot image generation

[Julius Werner]

>> 1. Some platforms want it in a fixed placed. I'd like that to change,
>> but there is a need currently. Also, if we put microcode in a
>> different place then we'd have to change the code which reads cbfs to
>> get the microcode file later when loading microcode for other cpus,
>> etc.

Sure, we might have to change code that looks for it... but I'd
presume (without knowing that code) that it should be a minor and
pretty straight-forward change compared to this big rearchitecture of
our build system, right?

Also, I'm not saying we should do this on all Intel platforms
(although we might, depending on what makes it easier). We could only
do it for the ones that need it (Skylake?) and make those use a
different FMD file.

(About the FMD files, I think there are many small differences that
could be solved by either making separate FMD files (and living with a
bit of code duplication), or using Kconfig variables and #ifdefs to
make one file fit multiple cases. I think taking either of those
options to the extreme would be bad, and we'd probably want a middle
road of both depending on what works best for a certain situation. I
still think that's a good system... the FMD format is dead simple to
read/understand, and every developer already knows how #ifdefs and
Kconfigs work without having to become familiar with some completely
custom system.)

>> 2. bootlbock is a little different too. While the reset vector is at
>> at the end of the region mapped just below 4GiB we don't have it be a
>> fixed size currently. Because of that you effectively have downward
>> growing stack behavior w.r.t. where things eventually land. That's
>> currently handled by the linker proper.

Right. That's why I'm saying I would change that... make the bootblock
area a pre-defined size (maybe based on a Kconfig). Sure, we'll waste
some space... but that's at most a few kilobytes of ROM, and in return
I think it makes the whole thing much easier to understand and work
with than the current CBFS-style bootblocks.

>> 3. FSP on these newer intel platforms is linked XIP. However, we did
>> add relocation to that as well so that we no longer had to fix the
>> location. But that's only effective for the FSP requirements post
>> cache-as-ram. Currently cache-as-ram on these FSP platforms is brought
>> up in FSP. And identifying the location of FSP w/o a stack is leads to
>> the current fixed location. I'm hoping to rectify this situation going
>> forward because these requirements are unnecessarily binding us,
>> however this is the current situation as of today.

Is this runtime or compile-time relocation we're talking about?
Because I would presume compile-time relocation (which would be good
enough for cbfstool) shouldn't make a difference about pre-/post-CAR?

Anyway... from a purist's perspective (without knowing any details),
this also sounds to me like if the FSP (or those parts of it)
absolutely *must* be placed at an exact offset, it doesn't belong in
CBFS and should have its own FMAP section. I mean, I think this is the
whole reason why we (want to) have an FMAP... we could also keep a
whole Chrome OS image inside CBFS if we just carefully adjust the
offsets enough, but that would be terribly unmanageable, because CBFS
just isn't a good tool for exact placement.

>> I think the coreboot manifest just decided which cbfs region to add a
>> file. The other stuff was flash layout. Sadly, I haven't seen fmd
>> fully rolled out or this so I can't tell with certainty what either
>> looks like. fmd proper is just flash layout and doesn't fix the "what
>> cbfs region and what flags" for each file nor the scheduling for
>> optimal placement.

Sure, I see FMD as a building block of a whole solution, not the
solution itself. (And I'd say that building block is pretty rolled out
already... all the code for it is checked in and you can already run
'fmaptool input.fmd output_fmap.bin' today.

>> I think Patrick's concern w/ the layout having a chipset portion was
>> so that one didn't have to touch every mainboard when changing
>> something in the chipset that required such a layout change. I think
>> you suggested #include files and he was opposed to it. I think the
>> disagreement using the c preprocessor is where the design changes stem
>> from (If i'm reading things correctly).

I'm not married the preprocessor, I certainly agree that there are a
lot of ugly things about it. I was suggesting #include as a concept
for FMD, not necessarily implemented through CPP. We also don't need
to make one FMD file per board (which would then usually just #include
a generic one, like memlayout)... we could instead think about
symlinks, or using some kind of CONFIG_MAINBOARD_HAS_CUSTOM_FMD... I'm
more arguing for general concept than details right now. (The one nice
thing that has to be said about CPP, though, is that every developer
already knows how it works. I think it has a much lower barrier to
entry than any custom mechanism for this we could whip 

Re: [coreboot] Rebuilding coreboot image generation

[Aaron Durbin]

And I didn't notice it was not on the list when it 'a' in gmail...

On Fri, Nov 6, 2015 at 5:32 PM, Aaron Durbin  wrote:
> On Fri, Nov 6, 2015 at 5:17 PM, Julius Werner  wrote:
>>> That suggestion leads to different software using fmap vs CBFS for
>>> finding and locating specifics assets that are required for booting.
>>> In fact that would fundamentally break x86 w.r.t. common stage loading
>>> because of the xip requirement. I don't think that's a good thing.
>>
>> Hmm... I don't really understand where you're seeing problems
>> regarding XIP there. I've got to admit I don't understand that area
>> very well though, especially since a lot of that changed recently.
>>
>> Just to be clear, I didn't mean to say that all XIP stages should go
>> into separate FMAP sections... those can stay in CBFS as they
>> currently are. I didn't intend to include everything XIP with "needs
>> to be placed exactly"... because they theoretically can be placed
>> anywhere (right? or are there limits?), they just need to be
>> explicitly linked to that offset once it has been decided. I know that
>> we used to link the romstage to offset 0x0, take the resulting binary
>> size of that to help layout CBFS, then link it again with the now
>> known offset... seems like all of that moved into cbfstool now, but in
>> theory it should still work just as well (regardless of the order the
>> files get added in and where the stage ends up being placed in the
>> end), right?
>
> Yes. That will work. However, we need to pass flags to cbfstool when
> we do add-stage to make that work.
>
>>
>> I think the case of that microcode blob is different because it seems
>> to need an exact offset (at least that's what the whole hardcoded
>> CONFIG_CPU_MICROCODE_CBFS_LOC thing looks like to me). Same for the
>> bootblock, which always needs to be placed into the reset vector (and
>> hasn't previously been a CBFS "file" anyway). Does that make it
>> clearer or did I misunderstand your concerns?
>
> Well there are few subtleties:
>
> 1. Some platforms want it in a fixed placed. I'd like that to change,
> but there is a need currently. Also, if we put microcode in a
> different place then we'd have to change the code which reads cbfs to
> get the microcode file later when loading microcode for other cpus,
> etc.
> 2. bootlbock is a little different too. While the reset vector is at
> at the end of the region mapped just below 4GiB we don't have it be a
> fixed size currently. Because of that you effectively have downward
> growing stack behavior w.r.t. where things eventually land. That's
> currently handled by the linker proper.
> 3. FSP on these newer intel platforms is linked XIP. However, we did
> add relocation to that as well so that we no longer had to fix the
> location. But that's only effective for the FSP requirements post
> cache-as-ram. Currently cache-as-ram on these FSP platforms is brought
> up in FSP. And identifying the location of FSP w/o a stack is leads to
> the current fixed location. I'm hoping to rectify this situation going
> forward because these requirements are unnecessarily binding us,
> however this is the current situation as of today.
>
>>
>>> Those requirements are not necessarily known until link time.
>>
>> Which ones are you thinking of in particular?
>> CONFIG_CPU_MICROCODE_CBFS_LOC seems to be hardcoded right now (unless
>> I misread how that works). The bootblock size is determined at link
>> time, but I don't think it needs to be... i.e. I think it's fine to
>> just reserve a certain bootblock FMAP section with the maximum size we
>> expect to need, and waste a few K of space if necessary. We could also
>> make it configurable with a Kconfig to allow more fine tuning by
>> preprocessing FMD files. I think that would be much more reasonable
>> and easier to work with than the current model of mushing the
>> bootblock somewhere in the CBFS area in a way that even cbfstool
>> cannot modify/extract again after the fact. It would also allow us to
>> further simplify the CBFS format (removing the need of the master
>> header and the concept that only a certain part of the full CBFS image
>> is space available for files... which was another goal of the CBFS
>> redesign earlier this year that had unfortunately not been finished).
>
> It can be changed, yes. However, I was just noting that's not how
> things work. As noted above bootblock is not the only thing.
>
>>
>>> But some files need special flags. One of the harder things is
>>> identifying all the assets to fit into those 2 buckets. I personally
>>> think one stop shopping in a single file is way better than digging
>>> through Makefiles.
>>
>> Well, but I don't think Patrick's proposal is one-stop-shopping
>> either... rather, it splits both layout and file information across
>> multiple manifests in the same way that CBFS file rules are already
>> split across our Makefiles.
>
> I think the coreboot manifest just 

Re: [coreboot] Rebuilding coreboot image generation

[Aaron Durbin]

On Fri, Nov 6, 2015 at 5:17 PM, Julius Werner  wrote:
>>> 1. Some platforms want it in a fixed placed. I'd like that to change,
>>> but there is a need currently. Also, if we put microcode in a
>>> different place then we'd have to change the code which reads cbfs to
>>> get the microcode file later when loading microcode for other cpus,
>>> etc.
>
> Sure, we might have to change code that looks for it... but I'd
> presume (without knowing that code) that it should be a minor and
> pretty straight-forward change compared to this big rearchitecture of
> our build system, right?

Well, some of those code paths are common to many x86 systems. And the
biggest thing I don't particularly like about that approach (ignoring
how one would need to handle things different at compile time) is the
lack of consistency. I don't mind the mix of cbfs and fmap, but I
think using both for the granularity of a single "file" is odd.

w.r.t. to a "big rearchitecture" I don't think it's that. It was my
understanding that the cbfs-file-* entries in the Makefiles would be
used to generate the manifest. As it currently stands fmd hasn't been
rolled out yet. Both approaches need to parse the file to know fmap
location such that we don't *require* a Kconfig and the fmap
description to make things line up. The second pass is adding the
files (and where they go).

>
> Also, I'm not saying we should do this on all Intel platforms
> (although we might, depending on what makes it easier). We could only
> do it for the ones that need it (Skylake?) and make those use a
> different FMD file.
>
> (About the FMD files, I think there are many small differences that
> could be solved by either making separate FMD files (and living with a
> bit of code duplication), or using Kconfig variables and #ifdefs to
> make one file fit multiple cases. I think taking either of those
> options to the extreme would be bad, and we'd probably want a middle
> road of both depending on what works best for a certain situation. I
> still think that's a good system... the FMD format is dead simple to
> read/understand, and every developer already knows how #ifdefs and
> Kconfigs work without having to become familiar with some completely
> custom system.)

I don't think Patrick's proposal is too complicated either. I think
Patrick was attempting to solve both sets of problems (the one FMD
solves currently and the cbfs file scheduler). The scheduler problem
leverages the description of the fmap.

>
>>> 2. bootlbock is a little different too. While the reset vector is at
>>> at the end of the region mapped just below 4GiB we don't have it be a
>>> fixed size currently. Because of that you effectively have downward
>>> growing stack behavior w.r.t. where things eventually land. That's
>>> currently handled by the linker proper.
>
> Right. That's why I'm saying I would change that... make the bootblock
> area a pre-defined size (maybe based on a Kconfig). Sure, we'll waste
> some space... but that's at most a few kilobytes of ROM, and in return
> I think it makes the whole thing much easier to understand and work
> with than the current CBFS-style bootblocks.

I understand your proposal. It sounds straight forward -- not sure how
big of a pain it is. Definitely doable though. Just solves one issue.

>
>>> 3. FSP on these newer intel platforms is linked XIP. However, we did
>>> add relocation to that as well so that we no longer had to fix the
>>> location. But that's only effective for the FSP requirements post
>>> cache-as-ram. Currently cache-as-ram on these FSP platforms is brought
>>> up in FSP. And identifying the location of FSP w/o a stack is leads to
>>> the current fixed location. I'm hoping to rectify this situation going
>>> forward because these requirements are unnecessarily binding us,
>>> however this is the current situation as of today.
>
> Is this runtime or compile-time relocation we're talking about?
> Because I would presume compile-time relocation (which would be good
> enough for cbfstool) shouldn't make a difference about pre-/post-CAR?

The current situation is that relocation *does not* work now.  The
assembly code expects it to be at a specific place. There is both
compile time and runtime relocation doing for Chrome OS because of the
RO, RW-A, and RW-B region. However, the RO one is a hard coded
location.

>
> Anyway... from a purist's perspective (without knowing any details),
> this also sounds to me like if the FSP (or those parts of it)
> absolutely *must* be placed at an exact offset, it doesn't belong in
> CBFS and should have its own FMAP section. I mean, I think this is the
> whole reason why we (want to) have an FMAP... we could also keep a
> whole Chrome OS image inside CBFS if we just carefully adjust the
> offsets enough, but that would be terribly unmanageable, because CBFS
> just isn't a good tool for exact placement.

Again then this becomes an inconsistency on Chrome OS. FSP lives in
RO, RW-A, and RW-B 

Re: [coreboot] Rebuilding coreboot image generation

[Patrick Georgi]

2015-11-06 3:57 GMT+01:00 Alex G. :
> My main concern with this is that we're introducing another language,
> with something that may just as well end up being cmos.layout 2.0.
>
> I see where the manifest language is coming from. It's sort of like a
> FMAP dts, but then it really isn't.  I understand that we don't want to
> use a per-board fmap.dts in order to prevent the duplication we ended up
> having last time we did this (cmos.layout).
>
> From reading the manifest examples, these are the few things that jumped
> out at as being unwanted:
> (1) mixing region descriptors with rules. As soon as we agree to move
> some rules to manifests and some to makefiles, we risk ending up with a
> mess that's inconsistent between platforms, and hard to track down/debug.
All rules end up in manifests. The build system wouldn't add a file to
a cbfs _ever_, cbfs-files-y would create the "coreboot" manifest, to
be consumed by the later tool.

Rationale: The build system is rather stupid about the order in which
to add files, and while adding a proper scheduler that prevents
unsuitable ordering (eg adding
flexibly laid out files first, which may block files with stricter
requirements later), people are rightfully complaining about too much
magic in the build system already.
(something to work on, but not as part of this effort)

> (2) Undermining some of the existing infrastructure. We can extend the
> concept of cbfs-files-y, to fmap-regions-y, whereas the FMAP regions are
> dynamically declared. Of course, this is in stark contrast to the
> per-module descriptors (which I also like, assuming they are pure
> fmap.dts format). However, this inconsistency worries me.
I deliberately didn't go for using dts, because that's a gross hack:
When trying to define files to add to CBFS in dts (which I do as a
temporary measure on the CrOS build system side), I now use dts
properties (key/value pairs) with arbitrary unique keys. Bleh.

And as stated above, gnu make is the wrong tool for the job, even
though it _would_ be possible to force it to do the right thing with
lots of effort. Readability will suffer, as will execution speed, and
in the end, like with certain other parts of the build system the only
one touching that stuff without making a mess would be me.

_That_ is something I don't want to see a repeat of, no matter of nice
it is to see that I can coerce gnu make to do things it wasn't
designed for on an ego-stroking level.

> (3) It changes everything we've learnt so far about building a firmware
> image. Some platforms will end up using the new manifests, others will
> use the existing build infrastructure. I'm worried we'll end up with an
> inconsistent build system. Who is going to convert all the old hardware
> to manifests?
I'll take care of upstream and CrOS. If only because I want to cut
back on the mess we have in the tree now.
Most coreboot images right now are pretty simple to convert:
They're either CBFS_SIZE == ROM_SIZE, an IFD layout, or one or two
boards with some special handling where neither of the other two hold
true.

That I'd not have to build an fmap-regions-y for each and every of the
boards individually is a plus.

> Now to the good parts. I mentioned briefly the possibility of a
> fmap-regions-y macro. Quite honestly, I much better like seeing the
> fmap.dts in one place in order to get an idea of the flash layout.
Consider we're going for an fmap description like that, and use that
and cbfs-files-y for the layout.
We still need a way to allocate files going into the various CBFS
regions (where current configurations have one or three, and there's a
pretty obvious two-CBFS configuration out there). That's another
variable to take care of, and there's nothing around really that can
make obvious any inconsistencies between those.

The proposal will, in effect, have three manifests on the coreboot
side, plus payload and additional projects that may contribute files
(eg Chrome EC in the Chrome OS use case). Ignoring the external
manifests for a moment, that's
- the chipset/board specific descriptor for flash layouts, chosen at
build time (and there will be a bunch of common ones, with the option
to override for special cases);
- the boot method manifest. Chosen by the same family of Kconfig
options that choose between simple or fallback/normal bootblocks.
These two are static and reside somewhere in the repo.
The third one is (what I called) the coreboot manifest, which lists
the files. That one is dependent on the build system, and is pretty
much a serialization of cbfs-files-y (just that cbfs-files-y now
includes all files, instead of some files in cbfs-files-y and some
manually added through $(CBFSTOOL) add* invocations in
src/arch/*/Makefile.inc).

That's the coreboot "build" side. Other projects can contribute their
own manifests for the final "link" step.

With all that stuff around, the final step is about reading the files
and generating the image.
Right now, we don't 

Re: [coreboot] Rebuilding coreboot image generation

[Patrick Georgi]

2015-11-06 6:55 GMT+01:00 Julius Werner :
> HOST_FIRMWARE@0xff80 8M {
> SI_ALL 2M {
> SI_DESC 4K
> SI_ME 0x1ff000
> }
> SI_BIOS {
> RW_A 0xf {
> VBLOCK_A 64K
> RW_MAIN_A
> RW_FWID_A 0x40
> }
> RW_B 0xf {
> VBLOCK_B 64K
> RW_MAIN_B
> RW_FWID_B 0x40
> }
> RO@4M {
> RO_VPD 16K
> FMAP@0x1 2K
> RO_FRID 0x40
> GBB@0x11000
> RO_MAIN 1M
> }
> }
> }
Just wanted to spell out the issues with this approach:
First, it doesn't tell you where to put various build collateral. Is
ramstage.elf to be added to SI_ME?

Second, this requires more or less one such description file per board
and boot method configuration, which is even worse than Alexandru's
concern about having a reenactment of horribly maintained cmos.layout
files (they're "merely" per board, but otherwise stable).


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Julius Werner]

Okay, looks like we're continuing the discussion over here after all.

> Rationale: The build system is rather stupid about the order in which
> to add files, and while adding a proper scheduler that prevents
> unsuitable ordering (eg adding
> flexibly laid out files first, which may block files with stricter
> requirements later), people are rightfully complaining about too much
> magic in the build system already.
> (something to work on, but not as part of this effort)

I assume you're talking about things like crosreview.com/289491 as an
example for strict requirements. I'm not really familiar with the
details there so please correct me if I'm wrong, but if this is a blob
that needs to be placed in one exact offset (which it looks like?),
then I think we're approaching it from the completely wrong angle
trying to stuff it into CBFS anyway. Maybe it was originally
shoehorned into CBFS because that was the only thing we had at the
time, but if we're rearchitecting things from the ground up to make
FMAP a first class citizen this seems to be the right time to
reconsider that.

The whole point of FMAP is to allow exact placement in the ROM,
whereas the whole point of CBFS is to conveniently group files with
variable lengths and no strict placement requirements together in an
easily manageable way. I think the only placement restriction CBFS
should support in that new design is a (mild) minimum alignment, and
anything that needs to be placed more exactly than that (including the
bootblock and magic Intel blobs) should instead get its own FMAP
section.

>> HOST_FIRMWARE@0xff80 8M {
>> SI_ALL 2M {
>> SI_DESC 4K
>> SI_ME 0x1ff000
>> }
>> SI_BIOS {
>> RW_A 0xf {
>> VBLOCK_A 64K
>> RW_MAIN_A
>> RW_FWID_A 0x40
>> }
>> RW_B 0xf {
>> VBLOCK_B 64K
>> RW_MAIN_B
>> RW_FWID_B 0x40
>> }
>> RO@4M {
>> RO_VPD 16K
>> FMAP@0x1 2K
>> RO_FRID 0x40
>> GBB@0x11000
>> RO_MAIN 1M
>> }
>> }
>> }
> Just wanted to spell out the issues with this approach:
> First, it doesn't tell you where to put various build collateral. Is
> ramstage.elf to be added to SI_ME?

This was just an example of an existing (and I think oversimplified)
Chrome OS ROM that I pulled from an old discussion, to demonstrate the
format. We totally can (and should) redesign and rename some existing
parts for the new layouts, and we could also add new features to the
existing FMD parser if we deem them useful (e.g. some sort of
inclusion or macro mechanism to help deduplicate more). Stuff like
RO_MAIN and RW_MAIN_B are probably better named RO_CBFS and RW_CBFS_B,
and I think the bootblock should be split out into its own section
(since it is not a proper CBFS file, and it has strict placement
requirements).

In my current vision, you would then group all of coreboot's output
files (both stages and separate files) into "pre-boot-path-split" and
"post-boot-path-split" groups. I think the existing Makefile system
would be fine for that, but we could discuss that part separately.
Then, whatever controls the Chrome OS specific parts of the build
process (I think a Makefile would be enough, but again, that's a
separate issue from the ROM layout itself) will 'cbfstool add -r
RO_CBFS' the pre-boot-path-split files, 'cbfstool add -r
RO_CBFS,RW_CBFS_A,RW_CBFS_B' the post-boot-path-split files, and
finally 'cbfstool add -r RW_CBFS_A,RW_CBFS_B' the EC-RW images (which
are Chrome OS specific and should therefore probably only come into
play at that time).

The advantage is that the layout (which may need to be very flexible
due to all the weird corner cases that we keep encountering, and may
be duplicated a little bit in practice) only tells you *where* the
well-known sections are in a single, very easily readable and
modifiable place that can be parsed at a glance, while another (more
generic) part of the system can decide *what* to put in those
sections. I think this would be much easier to use in practice than a
solution where the mere decision which section is placed where gets
spread out through unordered lines in X different files that all
somehow work together to cook up the final layout in a process hardly
anyone will really understand (or be able to adjust).

> Second, this requires more or less one such description file per board
> and boot method configuration, which is even worse than Alexandru's
> concern about having a reenactment of horribly maintained cmos.layout
> files (they're "merely" per board, but otherwise stable).

I think it probably requires less than two layouts per SoC in practice
(one for Chrome OS, one for normal 

Re: [coreboot] Rebuilding coreboot image generation

[Julius Werner]

> What I'd like to see come out of this is a dts-like set of flash
> descriptors, with the rules left in the makefiles. At the very least,
> dts is not a new language.

Just wanted to point out (without trying to usurp any of the ongoing
discussion in that doc) that we already have an FMAP descriptor
language that was implemented in cbfstool half a year ago, because I
don't think that got much visibility upstream when it was merged.
That's the FMD (flash map descriptor) format some of us are talking
about in those comments, which was developed in anticipation of a
build system redesign like this. It's aiming to describe FMAPs in a
clean, flexible way without the cruft of a format (dts) that was never
really meant for that purpose, and has some nice features (like
variable-sized sections) built-in. The parser can be seen in
util/cbfstool/fmd.c, and a sample file would look something like this
(in general, '@offset' is the base offset to the parent section and
the number after the space is the size):

HOST_FIRMWARE@0xff80 8M {
SI_ALL 2M {
SI_DESC 4K
SI_ME 0x1ff000
}
SI_BIOS {
RW_A 0xf {
VBLOCK_A 64K
RW_MAIN_A
RW_FWID_A 0x40
}
RW_B 0xf {
VBLOCK_B 64K
RW_MAIN_B
RW_FWID_B 0x40
}
RO@4M {
RO_VPD 16K
FMAP@0x1 2K
RO_FRID 0x40
GBB@0x11000
RO_MAIN 1M
}
}
}

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Alex G.]

On 11/05/2015 09:55 PM, Julius Werner wrote:
> HOST_FIRMWARE@0xff80 8M {
> SI_ALL 2M {
> SI_DESC 4K
> SI_ME 0x1ff000
> }
> SI_BIOS {
> RW_A 0xf {
> VBLOCK_A 64K
> RW_MAIN_A
> RW_FWID_A 0x40
> }
> RW_B 0xf {
> VBLOCK_B 64K
> RW_MAIN_B
> RW_FWID_B 0x40
> }
> RO@4M {
> RO_VPD 16K
> FMAP@0x1 2K
> RO_FRID 0x40
> GBB@0x11000
> RO_MAIN 1M
> }
> }
> }

Yes, that's exactly what I was referring to! I like that.

Alex

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot

Re: [coreboot] Rebuilding coreboot image generation

[Alex G.]

My main concern with this is that we're introducing another language,
with something that may just as well end up being cmos.layout 2.0.

I see where the manifest language is coming from. It's sort of like a
FMAP dts, but then it really isn't.  I understand that we don't want to
use a per-board fmap.dts in order to prevent the duplication we ended up
having last time we did this (cmos.layout).

>From reading the manifest examples, these are the few things that jumped
out at as being unwanted:
(1) mixing region descriptors with rules. As soon as we agree to move
some rules to manifests and some to makefiles, we risk ending up with a
mess that's inconsistent between platforms, and hard to track down/debug.
(2) Undermining some of the existing infrastructure. We can extend the
concept of cbfs-files-y, to fmap-regions-y, whereas the FMAP regions are
dynamically declared. Of course, this is in stark contrast to the
per-module descriptors (which I also like, assuming they are pure
fmap.dts format). However, this inconsistency worries me.
(3) It changes everything we've learnt so far about building a firmware
image. Some platforms will end up using the new manifests, others will
use the existing build infrastructure. I'm worried we'll end up with an
inconsistent build system. Who is going to convert all the old hardware
to manifests?

Now to the good parts. I mentioned briefly the possibility of a
fmap-regions-y macro. Quite honestly, I much better like seeing the
fmap.dts in one place in order to get an idea of the flash layout.

What I'd like to see come out of this is a dts-like set of flash
descriptors, with the rules left in the makefiles. At the very least,
dts is not a new language.

Do you think we can do that without making a mess out of it?

Alex


On 11/05/2015 03:42 AM, Patrick Georgi wrote:
> Hey coreboot folks,
> 
> I'm looking for an approach to make building Chrome OS style coreboot
> images easier to do with regular coreboot tools, instead of the rather
> large post-processing pipeline we have in the Chrome OS build system.
> 
> The rationale is that we also push the Chrome OS capabilities (eg.
> verified boot) upstream, and actually using them shouldn't depend on
> checking out yet another custom build environment.
> 
> I wrote a proposal on how to do that, which can be found and commented
> at 
> https://docs.google.com/document/d/1o2bFl5HCHDFPccQsOwa-75A8TWojjFiGK3r0yeIc7Vo/edit
> 
> If this is to be implemented, I'd do that as part of coreboot's build
> system, and work on supporting the regular image types (with a
> "simple" bootblock and a fallback/normal switch configuration) we
> currently have.
> 
> I appreciate comments and concerns how this fits in with other use cases.
> 
> 
> Thanks,
> Patrick
> 

-- 
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot