Thanks; I'm starting to understand. It makes more sense to know this
all pre-dates Yocto (and maybe its use within bitbake/OE).
I've generated a lot of text below, but I think the priority for 1.5
should be to enhance the documention of the current approach, rather
than try to address any of these issues by changing the tool behavior
(e.g. adding overrides). If the discussion leads to a conclusion that
the linux-yocto system can be improved then consider such changes for
early in the 1.6 cycle; in the meantime I'll see what I can do for a
complex set of recipe-space BSPs within the current system.
Let's start by defining terms that I think don't exist or aren't used
consistently. In what follows:
* The YPLKDM is the Yocto Project Linux Kernel Development Manual.
* The "root BSP description" is the "BSP description file that most
closely matches the KMACHINE and LINUX_KERNEL_TYPE variables passed in
from the recipe" (YPLKDM). It is the "top-level kernel feature" that is
passed to the kernel build subsystem (YPLKDM B.1 step 1). This should
be a very special file, as it is this file and anything it recursively
references, augmented by KERNEL_FEATURES, that defines the final kernel
configuration.
* The "kernel metadata root" is the in-tree directory at which searches
for kernel metadata fragments/patches/descriptions/configuration files
is rooted. This is ./meta/cfg/kernel-cache as checked out from the
standard meta branch.
On 08/30/2013 03:28 PM, Bruce Ashfield wrote:
On 13-08-30 03:33 PM, Peter A. Bigot wrote:
There are two search directories documented:
* "in-tree" which ends up as ${S}/.meta/cfg/kernel-cache through tool
magic guided by KMETA (where the directory name .meta is derived from
the branch identified by ${KMETA})
* "recipe-space" which ends up as ${WORKDIR} for material identified in
the SRC_URI of a recipe
In fact, recipe-space turns into in-tree, since they are all migrated
into the tree during the patch/setup phase. Makes things much more
uniform and enables easier sharing of features.
Putting everything into one tree before processing is good, but the way
it's done is not quite consistent with the mental model for the same
process used in bitbake and recipes.
I'd been thinking of kernel metadata as being managed much the same as
recipes: It comes from multiple sources (layers), and a developer can
include, exclude, mask, append, and prioritize to ensure the selected
contributions are what s/he wants. Some recipes/extensions get
submitted up back in open-embedded/poky/meta-oe, and some remain in
specialized layers without the expectation that other layers may use
them. A name identifies a capability, but the user gets to determine
who supplies its implementation. Layer-specific modifications are expected.
My current understanding is that kernel metadata is collected from
multiple sources and combined into a singly-rooted directory hierarchy
in-tree during patch/setup. Bare file names will subsequently be
located from the kernel metadata root within this hierarchy, using a
to-be-documented search algorithm. This works because the model assumes
a single canonical source that holds all metadata so it can be shared,
coordinated, and managed across all users of the linux-yocto
infrastructure. For any given base file name there is only one supplier
of its capability, with this policy enforced by the maintainer of the
shared metadata repository (viz., Bruce Ashfield). Recipe-space
overrides are, for the most part, an exception.
The difference between these conceptualizations is the fundamental
disconnect that I think needs to be communicated clearly in the YPLKDM,
as developers like me who are coming from a linux-mainline or
linux-gumstix standard recipe approach will expect linux-yocto to be
substantially the same.
Earlier documentation assumed that only one or the other is used, but
since recipe-space material can now reference in-tree material a
priority needs to be defined.
Now: within an scc file you can do:
include features/media/media-camera.scc
and that file in turn can do:
kconf media-camera.cfg
which resolves automatically to features/media/media-camera.cfg. This
reveals that there is also a third search directory: the "current
directory". Is that the directory in which the fragment that contains
the directive is found? What is its priority relative to the
recipe-space and in-tree metadata roots?
The current directory is searched first, when that fails the rest of
the search paths are checked. The order is kernel-cache (in tree)
and then the features as they were specified on the SRC_URI.
This seems to be inconsistent with the statement that everything goes
into the in-tree metadata directory. After that the SRC_URI should no
longer be relevant: its prioritization should have applied at the time
the directory contents were installed. Preferring in-tree kernel-cache
content would complicate one of my main use cases, where the in-tree
kernel-cache has something I don't want and I want to replace it from
recipe-space rather than attempt to fix what it did after the fact.
Regardless, developers will need to know exactly how material provided
by the recipe is prioritized relative to the original "in-tree" material
and other recipe-space material. Should the recipe-space material
provided through SRC_URI in recipe-space follow the same hierarchy as
cfg/kernel-cache? Are they copied into the in-tree hierarchy at
locations determined by the linux-yocto patch/setup tools, or preserving
the relative hierarchy specified in SRC_URI, or all dumped in as files
at the top of in-tree hierarchy (as they are normally dumped into
${WORKDIR}).
Take a patch as an example. Normally a patch file in a SRC_URI is
automatically applied to the source tree. But kernel metadata includes
patches which (based on the description in section B.1 of the YPLKDM)
are instead applied only if the patch is identified in a patch directive
in an scc file that is processed.
This is confusing, and what it suggests to me is that recipe-space
kernel metadata should not be provided through SRC_URI, because the
standard semantics of that variable don't match how it needs to be used
for linux-yocto. I think it's a bad idea to have files in SRC_URI that
behave differently for linux-yocto than they would in any other recipe.
See the proposal outline below for an alternative.
I believe that in the past fragment file names that did not include a
path could be located anywhere under the kernel-cache directory (e.g.,
in cfg, features, patches, ktypes, or even bsp). What if
In the distant past, .cfg files had to be in the same directory as
their .scc file, the ability to search for them at all is a middle
aged feature. To make it easier to write relocatable fragments.
That is still the case, but the regex's have been tightened up to
prevent false positives. That's why the use of current directory
fragments is something that is more and more avoided, but still happens
in particular for BSPs when the names of their features tend to be
more unique than "foo" :)
media-camera.cfg did not exist in the "current directory" or the root of
any search directory; should it be found if present in a subdirectory?
There are existing .scc files that rely on this, so the answer still
is 'yes, they should be searched and found'. But again, it is really
bad form and trigger false positives, so I understand the concern.
I can always update the tools to emit a WARNING when this happens,
and then over time restrict the searching. Since good or bad, it
is what it is .. for now :)
If so the namespace gets flattened and as more BSPs add their metadata
to the Yocto standard one the likelihood of collision and
incompatibility increases.
This can definitely happen. That's why as we collect more meta data
fragments, I try and make sure that they are both unique (which is
an effort due to fail), have relative paths and are collected in a
central location (both for re-use and namespace preservation).
My strongest objection is to the practice of searching within the
metadata hierarchy to locate a file that doesn't have path components,
exactly because it requires active management to avoid conflicts. This
imposes a burden on people attempting to support a BSP from a layer
recipe, who in their ignorance may use feature identifiers (pathless
file names) that may be superseded by the addition of metadata with the
same base name, possibly in a different path, within the upstream shared
metadata repository.
That last observation identifies a potential fourth directory: The
directory containing the master BSP description file (the one with
KMACHINE/KTYPE/KARCH). By explicitly calling out this directory any
files in kernel-cache/bsp/* that belong to other BSPs will be ignored,
and media-camera.cfg can be overridden for a specific machine without
having to replicate media-camera.scc.
I read this a few times, and I'm not quite sure I'm quite getting it
(it's been a long week of 3.10 fun ..)
The .scc file that defines a BSP description, is indeed a bit special
but the directory that it sits in doesn't have any special meaning after
it is located.
If I can convince you to eliminate search for files without paths except
when in the including file's directory, this won't matter. If I can't,
then making its directory "special" simplifies the task of a BSP
collecting all the standard overrides in one location.
In any case, searching the entirety of the bsp subdirectory for features
seems unstable since if something is re-usable it shouldn't be hidden
down in a BSP directory, and including those areas increases the chance
of picking up the wrong implementation of, say, hardware.cfg or wifi.scc.
To make kernel metadata deterministic it would be necessary to provide a
mechanism to prioritize these four search directories, and perhaps also
I'm all for determinism. As a start, I can get the current algorithm
into the docs .. honestly, I'm happy that someone cares to talk about
it, no one has asked in the past, so it's not documented at the moment.
to change the algorithm so that unqualified files (no path) will not be
found in subdirectories of these directories.
Aha. I think I know what you meant above about the BSP directories.
You'd want them to be excluded, and the current BSP's could override
a generic one. If the .cfg is added with a relative path, that of
course wouldn't work, but if it is one that is searched, there's a
possibility of doing something like that.
I'm hesitant to give them that much of a special status .. but I'll
think on it over the weekend.
My first thought is to just restore some much older functionality
(i.e. pre-yocto .. who knew there was such a long, twisted story?)
that offered an "override" syntax.
We have:
force kconf non-hardware foo.cfg
Which ensure that an option makes it into any includes, and with
override, you also have:
override kconf non-hardware foo.cfg myfoo.cfg
Where the .cfg's can be specified by their relative path to the root
of the search dir.
As it stands, you can obviously let the common one be included and
then override the options that you don't want / like in your board
configuration.
As I mentioned I don't think changes are appropriate at this stage of
the 1.5 development cycle, and I'd like to see how far I can get with
the "let common do what it wants and try to clean up later" approach,
though philosophically I dislike it since in the future "common" may do
something I didn't anticipate in my cleanup.
However, it may be possible to change things in 1.6 to make kernel
metadata management more consistent with recipe searches while not
disrupting existing practice too much. Here's a rough outline of a
candidate process:
SRC_URI is used exactly the same as in other recipes. It normally
contains a single entry which is the location of the git repository,
with KBRANCH and maybe KMETA branches or directories. If it does have
other stuff (such as patches), it is treated as in other recipes.
KMETA behaves exactly as it does now, if that means it ensures there is
a directory of metadata placed somewhere in-tree. I expect the root of
that directory to contain the cfg, patches, and scripts directories just
as the root of the meta directory in the meta branch of yocto-3.10 does,
and that within it is the kernel metadata root (cfg/kernel-cache).
SRC_KMETA_URI is a special variable which contains the recipe-space
metadata currently supplied through SRC_URI. Its contents are
transferred into the kernel metadata root instead of ${WORKDIR}. The
URIs are not applied as with SRC_URI even if they are patches, but tar
files, git branches, and other archives should be unpacked over the
previous contents preserving any internal hierarchy (support
striplevel). Files should be copied by shifting so that if
SRC_KMETA_URI identifies cfg/kernel-cache/features/media/v4l2.cfg as an
input the corresponding v4l2.cfg will be found at features/meta/v4l2.cfg
by the description parsing code. All overriding will be done in the
order of appearance in SRC_KMETA_URI.
This produces a single in-tree metadata directory, without attempting to
interpret the contents of any of the files. It should be a separate
task in the recipe, allowing the user to manipulate the recipe until the
contents of that metadata hierarchy are as desired.
The root BSP description is then identified per YPLKDM section B.1,
possibly restricted to search the bsp subhierarchy of the kernel
metadata root. It is processed, followed by processing the implicit
include of everything in KERNEL_FEATURES.
In all cases where a directive has an unadorned file like:
include foo
the tools must find file foo in the directory containing the file from
which the directive was read (the kernel metadata root for files in
KERNEL_FEATURES). Failure to find the file in that directory is an
error. Override is possible because the recipe could ensure the desired
version of that file was written into place through SRC_KMETA_URI.
If the directive specifies the file with a path, like:
kconf features/media/v4l2.cfg
then it must be found following that path from the kernel metadata
root. Again override through SRC_KMETA_URI works.
I understand this would be a change requiring BSPs to update their
configuration files. I also don't see that task taking more than five
or ten minutes, so it should be worth the effort to produce a more
consistent model of how the processing is done.
Peter
Cheers,
Bruce
Comments and discussion?
Peter
[1]
http://article.gmane.org/gmane.comp.handhelds.openembedded.core/41298/
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5090
_______________________________________________
linux-yocto mailing list
linux-yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/linux-yocto
_______________________________________________
linux-yocto mailing list
linux-yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/linux-yocto
