On 13-08-31 12:07 PM, Peter A. Bigot wrote:
Thanks; I'm starting to understand. It makes more sense to know this
all pre-dates Yocto (and maybe its use within bitbake/OE).
heh. It first existed in 2006, in one form or another. The Yocto mapping
is the latest incarnation of the tools. So it has been around for a while
now.
I've generated a lot of text below, but I think the priority for 1.5
should be to enhance the documention of the current approach, rather
than try to address any of these issues by changing the tool behavior
(e.g. adding overrides). If the discussion leads to a conclusion that
the linux-yocto system can be improved then consider such changes for
early in the 1.6 cycle; in the meantime I'll see what I can do for a
complex set of recipe-space BSPs within the current system.
Let's start by defining terms that I think don't exist or aren't used
consistently. In what follows:
Some of what you may be missing is in the "kernel architecture" manuals,
which I wrote ages ago .. and which we subsequently moved to a harder
to find area of the Yocto site .. since they contained too much detailed
information that wasn't generally useful.
* The YPLKDM is the Yocto Project Linux Kernel Development Manual.
* The "root BSP description" is the "BSP description file that most
closely matches the KMACHINE and LINUX_KERNEL_TYPE variables passed in
from the recipe" (YPLKDM). It is the "top-level kernel feature" that is
passed to the kernel build subsystem (YPLKDM B.1 step 1). This should
be a very special file, as it is this file and anything it recursively
references, augmented by KERNEL_FEATURES, that defines the final kernel
configuration.
It can also be augmented by bare .scc/.cfg files on the SRC_URI (aka
recipe space features), which are applied before KERNEL_FEATURES.
* The "kernel metadata root" is the in-tree directory at which searches
for kernel metadata fragments/patches/descriptions/configuration files
is rooted. This is ./meta/cfg/kernel-cache as checked out from the
standard meta branch.
Both definitions are correct, with my minor addition to the first
one.
On 08/30/2013 03:28 PM, Bruce Ashfield wrote:
On 13-08-30 03:33 PM, Peter A. Bigot wrote:
There are two search directories documented:
* "in-tree" which ends up as ${S}/.meta/cfg/kernel-cache through tool
magic guided by KMETA (where the directory name .meta is derived from
the branch identified by ${KMETA})
* "recipe-space" which ends up as ${WORKDIR} for material identified in
the SRC_URI of a recipe
In fact, recipe-space turns into in-tree, since they are all migrated
into the tree during the patch/setup phase. Makes things much more
uniform and enables easier sharing of features.
Putting everything into one tree before processing is good, but the way
it's done is not quite consistent with the mental model for the same
process used in bitbake and recipes.
I'd been thinking of kernel metadata as being managed much the same as
recipes: It comes from multiple sources (layers), and a developer can
include, exclude, mask, append, and prioritize to ensure the selected
contributions are what s/he wants. Some recipes/extensions get
submitted up back in open-embedded/poky/meta-oe, and some remain in
specialized layers without the expectation that other layers may use
them. A name identifies a capability, but the user gets to determine
who supplies its implementation. Layer-specific modifications are
expected.
It's not much different than this .. see below.
My current understanding is that kernel metadata is collected from
multiple sources and combined into a singly-rooted directory hierarchy
in-tree during patch/setup. Bare file names will subsequently be
located from the kernel metadata root within this hierarchy, using a
to-be-documented search algorithm. This works because the model assumes
a single canonical source that holds all metadata so it can be shared,
coordinated, and managed across all users of the linux-yocto
infrastructure. For any given base file name there is only one supplier
of its capability, with this policy enforced by the maintainer of the
shared metadata repository (viz., Bruce Ashfield). Recipe-space
overrides are, for the most part, an exception.
The difference between these conceptualizations is the fundamental
disconnect that I think needs to be communicated clearly in the YPLKDM,
as developers like me who are coming from a linux-mainline or
linux-gumstix standard recipe approach will expect linux-yocto to be
substantially the same.
That's a fairly close description of what happens, with the tweak that
the linux-yocto shipped meta data, is an integrated form of an out
of tree "kernel-cache" that is grafted into the tree. This allows the
baseline meta data to follow the tree, and arrive with a single clone
and use model.
Due to the complex interactions of kernel features (and not just
configuration), in order to build, test and consistently combine
features having them all in tree is essential versus being spread out
across multiple disjoint layers and repositories.
But that is only the baseline configuration that ends up in tree,
other layers and recipes are free to extend, modify and mask the
configuration as they see fit. If a feature or fragment is suitable for
more general use, it is encouraged to be merged into the main tree.
Let's put it this way, we can clone the linux-yocto tree, point a directory
full of cross toolchains at it, and then build each and every BSP and
feature combination in a fraction of the time and effort that it would
otherwise take (see the build-all scripts in the kern-tools repo).
Earlier documentation assumed that only one or the other is used, but
since recipe-space material can now reference in-tree material a
priority needs to be defined.
Now: within an scc file you can do:
include features/media/media-camera.scc
and that file in turn can do:
kconf media-camera.cfg
which resolves automatically to features/media/media-camera.cfg. This
reveals that there is also a third search directory: the "current
directory". Is that the directory in which the fragment that contains
the directive is found? What is its priority relative to the
recipe-space and in-tree metadata roots?
The current directory is searched first, when that fails the rest of
the search paths are checked. The order is kernel-cache (in tree)
and then the features as they were specified on the SRC_URI.
This seems to be inconsistent with the statement that everything goes
into the in-tree metadata directory. After that the SRC_URI should no
longer be relevant: its prioritization should have applied at the time
the directory contents were installed. Preferring in-tree kernel-cache
content would complicate one of my main use cases, where the in-tree
kernel-cache has something I don't want and I want to replace it from
recipe-space rather than attempt to fix what it did after the fact.
Maybe I didn't describe it clearly enough, it really is in that order
that things are searched and used. But what's in the tree is the
baseline data, and what is used to both construct the tree and validate
branches/patches at build time.
After that, it can be extended. In a bitbake format, that means patches,
.cfgs and .scc's specified on the SRC_URI or via KERNEL_FEATURES.
Regardless, developers will need to know exactly how material provided
by the recipe is prioritized relative to the original "in-tree" material
and other recipe-space material. Should the recipe-space material
provided through SRC_URI in recipe-space follow the same hierarchy as
cfg/kernel-cache? Are they copied into the in-tree hierarchy at
locations determined by the linux-yocto patch/setup tools, or preserving
the relative hierarchy specified in SRC_URI, or all dumped in as files
at the top of in-tree hierarchy (as they are normally dumped into
${WORKDIR}).
They shouldn't follow the kernel-caches's format, they are free to,
but don't have to. Out of tree .scc, .cfg and patches are kept separate
from in-tree data. They are migrated into the tree structure into
paths that match their recipe space origin and location. That keeps
all the namespaces separate and consistent.
You can't overwrite what's there, or all repeatability and consistency
is gone. Which is the point of the baseline meta data.
Take a patch as an example. Normally a patch file in a SRC_URI is
automatically applied to the source tree. But kernel metadata includes
patches which (based on the description in section B.1 of the YPLKDM)
are instead applied only if the patch is identified in a patch directive
in an scc file that is processed.
Patches are typically not applied from the meta-data, it's there as a
reference for integrated features. Patches are managed via git, and
the .scc files simply validate against the branch that a patch is applied.
This is confusing, and what it suggests to me is that recipe-space
kernel metadata should not be provided through SRC_URI, because the
standard semantics of that variable don't match how it needs to be used
for linux-yocto. I think it's a bad idea to have files in SRC_URI that
behave differently for linux-yocto than they would in any other recipe.
See the proposal outline below for an alternative.
They don't behave any differently than any other recipe .. they are
patched or pushed onto a branch in the order they are specified in
the SRC_URI. What's in the tree is *already* applied.
I believe that in the past fragment file names that did not include a
path could be located anywhere under the kernel-cache directory (e.g.,
in cfg, features, patches, ktypes, or even bsp). What if
In the distant past, .cfg files had to be in the same directory as
their .scc file, the ability to search for them at all is a middle
aged feature. To make it easier to write relocatable fragments.
That is still the case, but the regex's have been tightened up to
prevent false positives. That's why the use of current directory
fragments is something that is more and more avoided, but still happens
in particular for BSPs when the names of their features tend to be
more unique than "foo" :)
media-camera.cfg did not exist in the "current directory" or the root of
any search directory; should it be found if present in a subdirectory?
There are existing .scc files that rely on this, so the answer still
is 'yes, they should be searched and found'. But again, it is really
bad form and trigger false positives, so I understand the concern.
I can always update the tools to emit a WARNING when this happens,
and then over time restrict the searching. Since good or bad, it
is what it is .. for now :)
If so the namespace gets flattened and as more BSPs add their metadata
to the Yocto standard one the likelihood of collision and
incompatibility increases.
This can definitely happen. That's why as we collect more meta data
fragments, I try and make sure that they are both unique (which is
an effort due to fail), have relative paths and are collected in a
central location (both for re-use and namespace preservation).
My strongest objection is to the practice of searching within the
metadata hierarchy to locate a file that doesn't have path components,
exactly because it requires active management to avoid conflicts. This
imposes a burden on people attempting to support a BSP from a layer
recipe, who in their ignorance may use feature identifiers (pathless
file names) that may be superseded by the addition of metadata with the
same base name, possibly in a different path, within the upstream shared
metadata repository.
That's something that is already being phased out as I mentioned, the
current directory search will absolutely stay, but leaving the current
feature directory to locate a fragment or config won't be possible
in the future.
If you don't look in the current directory (i.e. #include "foo") versus
(#include <foo>), the feature fragments are not relocatable, and that's
a problem with maintenance that is unacceptable.
That last observation identifies a potential fourth directory: The
directory containing the master BSP description file (the one with
KMACHINE/KTYPE/KARCH). By explicitly calling out this directory any
files in kernel-cache/bsp/* that belong to other BSPs will be ignored,
and media-camera.cfg can be overridden for a specific machine without
having to replicate media-camera.scc.
I read this a few times, and I'm not quite sure I'm quite getting it
(it's been a long week of 3.10 fun ..)
The .scc file that defines a BSP description, is indeed a bit special
but the directory that it sits in doesn't have any special meaning after
it is located.
If I can convince you to eliminate search for files without paths except
when in the including file's directory, this won't matter. If I can't,
then making its directory "special" simplifies the task of a BSP
collecting all the standard overrides in one location.
See above, that is already happening. I left in my description above,
even after reading this .. just so I can be clear why the current
directory needs to always be on the search path.
In any case, searching the entirety of the bsp subdirectory for features
seems unstable since if something is re-usable it shouldn't be hidden
down in a BSP directory, and including those areas increases the chance
of picking up the wrong implementation of, say, hardware.cfg or wifi.scc.
To make kernel metadata deterministic it would be necessary to provide a
mechanism to prioritize these four search directories, and perhaps also
I'm all for determinism. As a start, I can get the current algorithm
into the docs .. honestly, I'm happy that someone cares to talk about
it, no one has asked in the past, so it's not documented at the moment.
to change the algorithm so that unqualified files (no path) will not be
found in subdirectories of these directories.
Aha. I think I know what you meant above about the BSP directories.
You'd want them to be excluded, and the current BSP's could override
a generic one. If the .cfg is added with a relative path, that of
course wouldn't work, but if it is one that is searched, there's a
possibility of doing something like that.
I'm hesitant to give them that much of a special status .. but I'll
think on it over the weekend.
My first thought is to just restore some much older functionality
(i.e. pre-yocto .. who knew there was such a long, twisted story?)
that offered an "override" syntax.
We have:
force kconf non-hardware foo.cfg
Which ensure that an option makes it into any includes, and with
override, you also have:
override kconf non-hardware foo.cfg myfoo.cfg
Where the .cfg's can be specified by their relative path to the root
of the search dir.
As it stands, you can obviously let the common one be included and
then override the options that you don't want / like in your board
configuration.
As I mentioned I don't think changes are appropriate at this stage of
the 1.5 development cycle, and I'd like to see how far I can get with
the "let common do what it wants and try to clean up later" approach,
though philosophically I dislike it since in the future "common" may do
something I didn't anticipate in my cleanup.
The change would be completely transparent to any existing meta
data and users, so there's no risk. I may still bring it in, since the
code already exists :)
However, it may be possible to change things in 1.6 to make kernel
metadata management more consistent with recipe searches while not
disrupting existing practice too much. Here's a rough outline of a
candidate process:
SRC_URI is used exactly the same as in other recipes. It normally
contains a single entry which is the location of the git repository,
with KBRANCH and maybe KMETA branches or directories. If it does have
other stuff (such as patches), it is treated as in other recipes.
KMETA behaves exactly as it does now, if that means it ensures there is
a directory of metadata placed somewhere in-tree. I expect the root of
that directory to contain the cfg, patches, and scripts directories just
as the root of the meta directory in the meta branch of yocto-3.10 does,
and that within it is the kernel metadata root (cfg/kernel-cache).
SRC_KMETA_URI is a special variable which contains the recipe-space
metadata currently supplied through SRC_URI. Its contents are
transferred into the kernel metadata root instead of ${WORKDIR}. The
Everything bounces through ${WORKDIR}, that's how the fetcher works,
and we want the fetcher to take care of the details of getting SRC_URI
entries and making them available. It works well, and there's no reason
to duplicate its functionality.
It's after they are in WORKDIR they make it into the kernel meta
directory (in their own namespace).
But that detail is largely irrelevant.
I already have a feature that slipped from 1.5 that would allow full
recipe space directory hierarchies to be specified, without any extra
variables or special processing. Simply put the directory on the
SRC_URI and the processing triggers, it'll go in early in the 1.6 cycle.
I have TomZ's email asking me about it from 2012 staring at me!
URIs are not applied as with SRC_URI even if they are patches, but tar
files, git branches, and other archives should be unpacked over the
previous contents preserving any internal hierarchy (support
This is all fetcher functionality already, not something we want to
duplicate.
striplevel). Files should be copied by shifting so that if
SRC_KMETA_URI identifies cfg/kernel-cache/features/media/v4l2.cfg as an
input the corresponding v4l2.cfg will be found at features/meta/v4l2.cfg
I read this about 4 times, and it still doesn't read properly to me.
So I'm not sure what you are trying to describe here.
by the description parsing code. All overriding will be done in the
order of appearance in SRC_KMETA_URI.
If I'm reading it right, it can't happen as you've described it, plus
special kernel specific variables are bad. I've eliminated pretty much
every one that I had to introduce in yocto 0.9 to now and new ones are
a hard sell.
What happens in recipe space and extensions cannot directly clobber
what's already in the tree. Validation will fail, patch failures will
occur and many other types of issues .. I've lived through them all, and
won't go back. They must be explicitly called out in .scc files, where
the writer of the file knows what they are doing.
This produces a single in-tree metadata directory, without attempting to
interpret the contents of any of the files. It should be a separate
task in the recipe, allowing the user to manipulate the recipe until the
contents of that metadata hierarchy are as desired.
The user shouldn't be interpreting the hierarchy or manipulating it at
all .. just overriding from their own .scc files. Anything else is
complicated and binds everything to a very specific structure.
The root BSP description is then identified per YPLKDM section B.1,
possibly restricted to search the bsp subhierarchy of the kernel
metadata root. It is processed, followed by processing the implicit
include of everything in KERNEL_FEATURES.
In all cases where a directive has an unadorned file like:
include foo
This part has already been happening for sometime, and will more than
likely be the case regardless in 1.6.
the tools must find file foo in the directory containing the file from
which the directive was read (the kernel metadata root for files in
KERNEL_FEATURES). Failure to find the file in that directory is an
error. Override is possible because the recipe could ensure the desired
version of that file was written into place through SRC_KMETA_URI.
If you mean the overwriting of "foo" by the outside override, no, that
won't happen.
If the directive specifies the file with a path, like:
kconf features/media/v4l2.cfg
then it must be found following that path from the kernel metadata
root. Again override through SRC_KMETA_URI works.
I understand this would be a change requiring BSPs to update their
configuration files. I also don't see that task taking more than five
I can make the searching changes in 1.6, as they were already planned,
but anything more invasive than that is a problem, since I don't have
access to every BSP description that exists :)
The original design of all this, is that any feature, any BSP can
manipulate was has already been specified in the baseline configuration
.. to a point. You can't mess with patches, git merges or anything that
modifies the structure of the baseline tree. Horrible patch errors ensue
and the
entire thing collapses under it's own weight. The current system was
created after watching patch and configuration management collapse.
What *any* feature can do, is influence the configuration and ordering
of configs and optional features. That is done via commands and
operations in their .scc files. The entire base configuration can be in
a read only location, and you can still manipulate it.
That's what the 'override' command that I mentioned does, as do some
other "trigger" or mapping commands.
A .scc file injected from an external location (i.e recipe space) can do
this:
override features/media/v4l2.cfg my_features/media/v4l2.cfg
or
override features/edf/patch01.patch my_features/edf/abetterpatch.foo
.. etc.
The other part of the entire model is that if the baseline configuration,
features, patches, etc are NOT useful to someone's BSP or feature, then
they could be wrong, we should discuss and adjust the descriptions
accordingly. i.e. the need to override and manipulate goes away over
time, with the override mechanisms filling the gaps temporarily.
Great discussion BTW, I really do appreciate it, kept me up until 1:30
AM .. again!
Cheers,
Bruce
or ten minutes, so it should be worth the effort to produce a more
consistent model of how the processing is done.
Peter
Cheers,
Bruce
Comments and discussion?
Peter
[1]
http://article.gmane.org/gmane.comp.handhelds.openembedded.core/41298/
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5090
_______________________________________________
linux-yocto mailing list
linux-yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/linux-yocto
_______________________________________________
linux-yocto mailing list
linux-yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/linux-yocto
