Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[Bruce Ashfield]

On 2016-09-01 11:18 PM, Bruce Ashfield wrote:

On 2016-09-01 12:14 PM, André Draszik wrote:

On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote:

On 2016-08-31 4:54 AM, André Draszik wrote:


On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote:


Can you clarify for me if you are are using SRC_URI items tagged with
'kmeta', i.e. a directory of fragments, or are you just adding
.cfg/.scc
items directly to the SRC_URI ?


I do both:

in the 1st layer, my kernel recipe boils down to:

SRC_URI = "\
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-
stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \
file://0001-a-patch.patch \
file://kernel-meta;type=kmeta;destsuffix=${KMETA} \
"

KERNEL_FEATURES_append = " patches/some-patches.scc"
KERNEL_FEATURES_append = " patches/more-patches.scc"
KERNEL_FEATURES_append = " patches/even-more-patches.scc"

some of the above in turn include additional .scc items recursively.


In the 2nd layer, my .bbappend boils down to:

FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:"

KERNEL_FEATURES_append_tgm = " patches/tgm.scc"

2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd
layer. I don't specifically add another SRC_URI item tagged with
'kmeta'
(or
any other explicit SRC_URI item for that matter).


One more clarification. From our problem description, are you saying
that in your runs, only the meta data from layer2 is getting copied
and not that from layer1 ?


That is what I had inferred from the error message:

| DEBUG: Executing shell function do_kernel_metadata
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files:
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch
patches/some-patches.scc patches/more-patches.scc
patches/even-more-patches.scc patches/tgm.scc
|See /tmp/tmp.mPP1jtatxm for details
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files:
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch
patches/some-patches.scc patches/more-patches.scc
patches/even-more-patches.scc patches/tgm.scc
|See /tmp/tmp.v9LU9ccpbl for details
| WARNING: exit code 1 from a shell command.



I ran my sanity test, and saw meta data from both my layers copied ..
so I'm worried that I misunderstood the issue you are seeing.


I can see that they are actually copied indeed, but it doesn't find
them...
The file /tmp/tmp.v9LU9ccpbl mentioned above is empty.


But looking at the code, I can definitely do some minor tweaks in this
area .. I'm just trying to get the reproducer nailed down.


I can try to create one for you if you prefer tomorrow, rather than
wasting
your time with this, if you don't get any further?


I was able to trigger a similar trap to the above. And yes, it was sort
of working by luck before.


I spoke too soon. The error that I was able to trigger was valid,
and in fact, when I tried the same test on krogoth it also failed
there as well.

So there is something I still don't have right in the two layers to
trigger the same scenario you are seeing.

If you could do that mock up, it would help immensely.

Also, if you are open to changing the layer structure a bit, I could
always suggest a new way to organize things.

Cheers,

Bruce



I need to stare at this for a few hours and figure out the right way to
restore similar behaviour. But now .. sleep.

Cheers,

Bruce




Andre'





--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[Bruce Ashfield]

On 2016-09-01 12:14 PM, André Draszik wrote:

On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote:

On 2016-08-31 4:54 AM, André Draszik wrote:


On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote:


Can you clarify for me if you are are using SRC_URI items tagged with
'kmeta', i.e. a directory of fragments, or are you just adding
.cfg/.scc
items directly to the SRC_URI ?


I do both:

in the 1st layer, my kernel recipe boils down to:

SRC_URI = "\
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-
stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \
file://0001-a-patch.patch \
file://kernel-meta;type=kmeta;destsuffix=${KMETA} \
"

KERNEL_FEATURES_append = " patches/some-patches.scc"
KERNEL_FEATURES_append = " patches/more-patches.scc"
KERNEL_FEATURES_append = " patches/even-more-patches.scc"

some of the above in turn include additional .scc items recursively.


In the 2nd layer, my .bbappend boils down to:

FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:"

KERNEL_FEATURES_append_tgm = " patches/tgm.scc"

2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd
layer. I don't specifically add another SRC_URI item tagged with 'kmeta'
(or
any other explicit SRC_URI item for that matter).


One more clarification. From our problem description, are you saying
that in your runs, only the meta data from layer2 is getting copied
and not that from layer1 ?


That is what I had inferred from the error message:

| DEBUG: Executing shell function do_kernel_metadata
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files: 
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig 
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch 
patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc 
patches/tgm.scc
|See /tmp/tmp.mPP1jtatxm for details
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files: 
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig 
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch 
patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc 
patches/tgm.scc
|See /tmp/tmp.v9LU9ccpbl for details
| WARNING: exit code 1 from a shell command.



I ran my sanity test, and saw meta data from both my layers copied ..
so I'm worried that I misunderstood the issue you are seeing.


I can see that they are actually copied indeed, but it doesn't find them...
The file /tmp/tmp.v9LU9ccpbl mentioned above is empty.


But looking at the code, I can definitely do some minor tweaks in this
area .. I'm just trying to get the reproducer nailed down.


I can try to create one for you if you prefer tomorrow, rather than wasting
your time with this, if you don't get any further?


I was able to trigger a similar trap to the above. And yes, it was sort
of working by luck before.

I need to stare at this for a few hours and figure out the right way to
restore similar behaviour. But now .. sleep.

Cheers,

Bruce




Andre'



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[Bruce Ashfield]

On 2016-09-01 12:14 PM, André Draszik wrote:

On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote:

On 2016-08-31 4:54 AM, André Draszik wrote:


On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote:


Can you clarify for me if you are are using SRC_URI items tagged with
'kmeta', i.e. a directory of fragments, or are you just adding
.cfg/.scc
items directly to the SRC_URI ?


I do both:

in the 1st layer, my kernel recipe boils down to:

SRC_URI = "\
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-
stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \
file://0001-a-patch.patch \
file://kernel-meta;type=kmeta;destsuffix=${KMETA} \
"

KERNEL_FEATURES_append = " patches/some-patches.scc"
KERNEL_FEATURES_append = " patches/more-patches.scc"
KERNEL_FEATURES_append = " patches/even-more-patches.scc"

some of the above in turn include additional .scc items recursively.


In the 2nd layer, my .bbappend boils down to:

FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:"

KERNEL_FEATURES_append_tgm = " patches/tgm.scc"

2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd
layer. I don't specifically add another SRC_URI item tagged with 'kmeta'
(or
any other explicit SRC_URI item for that matter).


One more clarification. From our problem description, are you saying
that in your runs, only the meta data from layer2 is getting copied
and not that from layer1 ?


That is what I had inferred from the error message:

| DEBUG: Executing shell function do_kernel_metadata
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files: 
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig 
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch 
patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc 
patches/tgm.scc
|See /tmp/tmp.mPP1jtatxm for details
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files: 
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig 
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch 
patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc 
patches/tgm.scc
|See /tmp/tmp.v9LU9ccpbl for details
| WARNING: exit code 1 from a shell command.



I ran my sanity test, and saw meta data from both my layers copied ..
so I'm worried that I misunderstood the issue you are seeing.


I can see that they are actually copied indeed, but it doesn't find them...
The file /tmp/tmp.v9LU9ccpbl mentioned above is empty.


Yah, that's a temporary file used in the processing. I'm going to clean
up the logging a bit as I work through this.

I think I can fix things with some path manipulations, but am still
working on it.




But looking at the code, I can definitely do some minor tweaks in this
area .. I'm just trying to get the reproducer nailed down.


I can try to create one for you if you prefer tomorrow, rather than wasting
your time with this, if you don't get any further?


I'm going to be trying this later tonight (Thursday, Eastern Time), but
even if you can throw a mock up out, it will help me debug more.

Bruce




Andre'



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[André Draszik]

On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote:
> On 2016-08-31 4:54 AM, André Draszik wrote:
> > 
> > On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote:
> > > 
> > > Can you clarify for me if you are are using SRC_URI items tagged with
> > > 'kmeta', i.e. a directory of fragments, or are you just adding
> > > .cfg/.scc
> > > items directly to the SRC_URI ?
> > 
> > I do both:
> > 
> > in the 1st layer, my kernel recipe boils down to:
> > 
> > SRC_URI = "\
> > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-
> > stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \
> > file://0001-a-patch.patch \
> > file://kernel-meta;type=kmeta;destsuffix=${KMETA} \
> > "
> > 
> > KERNEL_FEATURES_append = " patches/some-patches.scc"
> > KERNEL_FEATURES_append = " patches/more-patches.scc"
> > KERNEL_FEATURES_append = " patches/even-more-patches.scc"
> > 
> > some of the above in turn include additional .scc items recursively.
> > 
> > 
> > In the 2nd layer, my .bbappend boils down to:
> > 
> > FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:"
> > 
> > KERNEL_FEATURES_append_tgm = " patches/tgm.scc"
> > 
> > 2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd
> > layer. I don't specifically add another SRC_URI item tagged with 'kmeta'
> > (or
> > any other explicit SRC_URI item for that matter).
> 
> One more clarification. From our problem description, are you saying
> that in your runs, only the meta data from layer2 is getting copied
> and not that from layer1 ?

That is what I had inferred from the error message:

| DEBUG: Executing shell function do_kernel_metadata
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files: 
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig 
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch 
patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc 
patches/tgm.scc
|See /tmp/tmp.mPP1jtatxm for details
| ERROR. input file "patches/some-patches.scc" does not exist
| ERROR: could not process input files: 
<2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig 
<1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch 
patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc 
patches/tgm.scc
|See /tmp/tmp.v9LU9ccpbl for details
| WARNING: exit code 1 from a shell command.


> I ran my sanity test, and saw meta data from both my layers copied ..
> so I'm worried that I misunderstood the issue you are seeing.

I can see that they are actually copied indeed, but it doesn't find them...
The file /tmp/tmp.v9LU9ccpbl mentioned above is empty.

> But looking at the code, I can definitely do some minor tweaks in this
> area .. I'm just trying to get the reproducer nailed down.

I can try to create one for you if you prefer tomorrow, rather than wasting
your time with this, if you don't get any further?


Andre'

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[Bruce Ashfield]

On 2016-08-31 4:54 AM, André Draszik wrote:

On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote:

Can you clarify for me if you are are using SRC_URI items tagged with
'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc
items directly to the SRC_URI ?


I do both:

in the 1st layer, my kernel recipe boils down to:

SRC_URI = "\

git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git;protocol=git;nocheckout=1;branch=${KBRANCH}
 \
file://0001-a-patch.patch \
file://kernel-meta;type=kmeta;destsuffix=${KMETA} \
"

KERNEL_FEATURES_append = " patches/some-patches.scc"
KERNEL_FEATURES_append = " patches/more-patches.scc"
KERNEL_FEATURES_append = " patches/even-more-patches.scc"

some of the above in turn include additional .scc items recursively.


In the 2nd layer, my .bbappend boils down to:

FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:"

KERNEL_FEATURES_append_tgm = " patches/tgm.scc"

2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd
layer. I don't specifically add another SRC_URI item tagged with 'kmeta' (or
any other explicit SRC_URI item for that matter).


One more clarification. From our problem description, are you saying
that in your runs, only the meta data from layer2 is getting copied
and not that from layer1 ?

I ran my sanity test, and saw meta data from both my layers copied ..
so I'm worried that I misunderstood the issue you are seeing.

But looking at the code, I can definitely do some minor tweaks in this
area .. I'm just trying to get the reproducer nailed down.

Bruce




Andre'



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[André Draszik]

On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote:
> Can you clarify for me if you are are using SRC_URI items tagged with
> 'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc
> items directly to the SRC_URI ?

I do both:

in the 1st layer, my kernel recipe boils down to:

SRC_URI = "\

git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git;protocol=git;nocheckout=1;branch=${KBRANCH}
 \
file://0001-a-patch.patch \
file://kernel-meta;type=kmeta;destsuffix=${KMETA} \
"

KERNEL_FEATURES_append = " patches/some-patches.scc"
KERNEL_FEATURES_append = " patches/more-patches.scc"
KERNEL_FEATURES_append = " patches/even-more-patches.scc"

some of the above in turn include additional .scc items recursively.


In the 2nd layer, my .bbappend boils down to:

FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:"

KERNEL_FEATURES_append_tgm = " patches/tgm.scc"

2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd
layer. I don't specifically add another SRC_URI item tagged with 'kmeta' (or
any other explicit SRC_URI item for that matter).


Andre'

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[Bruce Ashfield]

On 2016-08-30 10:19 AM, André Draszik wrote:

On Di, 2016-08-30 at 09:05 -0400, Bruce Ashfield wrote:

On 2016-08-30 05:05 AM, André Draszik wrote:


My kmeta stuff is located directly inside two meta-* layers (i.e. not in
an
extra git repo), where the 2nd layer's kernel recipes .bbappends to the
1st
one with additional kmeta things.

This used to work fine, but since this commit it doesn't anymore as only
the
kmeta stuff from the 2nd layer is being copied into workdir.

Have I been doing something that wasn't guaranteed / supposed to work in
the
first place? Or should that still work? Is it worth fixing or should I
rethink my approach?


That was complexity that I had tried to remove, but thinking on it a
bit, I think I can restore that without adding much complexity.

I'm just finishing work on the 4.8 kernel today, and can look at this
tomorrow.


Thank you Bruce!


Can you clarify for me if you are are using SRC_URI items tagged with
'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc
items directly to the SRC_URI ?

Bruce




Do you have any public layers that I can reference to pull together a
test case ?


Unfortunately not :-(

But I'm happy to test and report back!


Cheers,
Andre'



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[André Draszik]

On Di, 2016-08-30 at 09:05 -0400, Bruce Ashfield wrote:
> On 2016-08-30 05:05 AM, André Draszik wrote:
> > 
> > My kmeta stuff is located directly inside two meta-* layers (i.e. not in
> > an
> > extra git repo), where the 2nd layer's kernel recipes .bbappends to the
> > 1st
> > one with additional kmeta things.
> > 
> > This used to work fine, but since this commit it doesn't anymore as only
> > the
> > kmeta stuff from the 2nd layer is being copied into workdir.
> > 
> > Have I been doing something that wasn't guaranteed / supposed to work in
> > the
> > first place? Or should that still work? Is it worth fixing or should I
> > rethink my approach?
> 
> That was complexity that I had tried to remove, but thinking on it a
> bit, I think I can restore that without adding much complexity.
> 
> I'm just finishing work on the 4.8 kernel today, and can look at this
> tomorrow.

Thank you Bruce!

> Do you have any public layers that I can reference to pull together a
> test case ?

Unfortunately not :-(

But I'm happy to test and report back!


Cheers,
Andre'

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[Bruce Ashfield]

On 2016-08-30 05:05 AM, André Draszik wrote:

My kmeta stuff is located directly inside two meta-* layers (i.e. not in an
extra git repo), where the 2nd layer's kernel recipes .bbappends to the 1st
one with additional kmeta things.

This used to work fine, but since this commit it doesn't anymore as only the
kmeta stuff from the 2nd layer is being copied into workdir.

Have I been doing something that wasn't guaranteed / supposed to work in the
first place? Or should that still work? Is it worth fixing or should I
rethink my approach?


That was complexity that I had tried to remove, but thinking on it a
bit, I think I can restore that without adding much complexity.

I'm just finishing work on the 4.8 kernel today, and can look at this
tomorrow.

Do you have any public layers that I can reference to pull together a
test case ?

Bruce





Cheers,
Andre'


On Mo, 2016-08-15 at 14:26 -0400, Bruce Ashfield wrote:

We've been running with a set of kern-tools that were designed to work
with build systems that knew nothing about git, trees, commits, etc.

As such, there's been a set of shims/wrappers in place to work with
within bitbake/oe-core. These were the *me scripts: createme, updateme,
patchme and configme.

With this commit, we strip that legacy code and use the tools directly.
This means less complexity, fewer corner cases .. and no surprises
when the tools are arunning. As another benefit, the tools consume
much less time during a typical build and have no noticeable impact
on the overall build time.

Existing .scc files, features, and processing are not impacted as
these tools are compatible with existing feature descriptions and
kerne configuration fragments.

The audit of kernel configuration fragments is now detached
from the linux-yocto build structure and process. This means that
they can eventually be tweaked to offer kernel audit to any type of
kernel build and configuration process.

Additionally, the kernel symbol audit phase can now resolve symbol
dependencies and offer guidance when a symbol is missing:

   WARNING: linux-yocto-4.4.15+gitAUTOINC+b030d96c7b_f5e2c49d58-r0
do_kernel_configcheck: [kernel config]: specified values did not make it
into the kernel's final configuration:

   -- CONFIG_BT_6LOWPAN -
   Config: CONFIG_BT_6LOWPAN
   From: /home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel-
source/.kernel-meta/configs/standard/features/bluetooth/bluetooth.cfg
   Requested value:  CONFIG_BT_6LOWPAN=y
   Actual value:

   Config 'BT_6LOWPAN' has the following conditionals:
 BT_LE && 6LOWPAN (value: "n")
   Dependency values are:
 BT_LE [y] 6LOWPAN [n]

Signed-off-by: Bruce Ashfield 
---
 meta/classes/kernel-yocto.bbclass  | 143 --
---
 .../kern-tools/kern-tools-native_git.bb|   4 +-
 2 files changed, 55 insertions(+), 92 deletions(-)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-
yocto.bbclass
index a9d4205..8650e55 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -119,77 +119,42 @@ do_kernel_metadata() {
patches="${@" ".join(find_patches(d))}"
feat_dirs="${@" ".join(find_kernel_feature_dirs(d))}"

-   # add any explicitly referenced features onto the end of the
feature
-   # list that is passed to the kernel build scripts.
-   if [ -n "${KERNEL_FEATURES}" ]; then
-   for feat in ${KERNEL_FEATURES}; do
-   addon_features="$addon_features --feature $feat"
-   done
-   fi
-
# check for feature directories/repos/branches that were part of
the
# SRC_URI. If they were supplied, we convert them into include
directives
# for the update part of the process
-   if [ -n "${feat_dirs}" ]; then
-   for f in ${feat_dirs}; do
+   for f in ${feat_dirs}; do
if [ -d "${WORKDIR}/$f/meta" ]; then
-   includes="$includes -I${WORKDIR}/$f/meta"
+   includes="$includes -I${WORKDIR}/$f/meta"
elif [ -d "${WORKDIR}/$f" ]; then
-   includes="$includes -I${WORKDIR}/$f"
+   includes="$includes -I${WORKDIR}/$f"
fi
-   done
-   fi
+   done
+   for s in ${sccs}; do
+   sdir=$(dirname $s)
+   includes="$includes -I${sdir}"
+   done

-   # updates or generates the target description
-   updateme ${updateme_flags}
-DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \
- ${includes} ${addon_features} ${ARCH}
${KMACHINE} ${sccs} ${patches}
-   if [ $? -ne 0 ]; then
-   bbfatal_log "Could not update ${machine_branch}"
-   fi
+   # expand kernel features into their full path equivalents
+   bsp_definition=$(spp ${includes} --find -DKMACHINE=${KMACHINE}
-DKTYPE=${LINUX_KERNEL_TYPE})
+   meta_dir=$(kgit --meta)
+
+   # 

Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[André Draszik]

My kmeta stuff is located directly inside two meta-* layers (i.e. not in an
extra git repo), where the 2nd layer's kernel recipes .bbappends to the 1st
one with additional kmeta things.

This used to work fine, but since this commit it doesn't anymore as only the
kmeta stuff from the 2nd layer is being copied into workdir.

Have I been doing something that wasn't guaranteed / supposed to work in the
first place? Or should that still work? Is it worth fixing or should I
rethink my approach?


Cheers,
Andre'


On Mo, 2016-08-15 at 14:26 -0400, Bruce Ashfield wrote:
> We've been running with a set of kern-tools that were designed to work
> with build systems that knew nothing about git, trees, commits, etc.
> 
> As such, there's been a set of shims/wrappers in place to work with
> within bitbake/oe-core. These were the *me scripts: createme, updateme,
> patchme and configme.
> 
> With this commit, we strip that legacy code and use the tools directly.
> This means less complexity, fewer corner cases .. and no surprises
> when the tools are arunning. As another benefit, the tools consume
> much less time during a typical build and have no noticeable impact
> on the overall build time.
> 
> Existing .scc files, features, and processing are not impacted as
> these tools are compatible with existing feature descriptions and
> kerne configuration fragments.
> 
> The audit of kernel configuration fragments is now detached
> from the linux-yocto build structure and process. This means that
> they can eventually be tweaked to offer kernel audit to any type of
> kernel build and configuration process.
> 
> Additionally, the kernel symbol audit phase can now resolve symbol
> dependencies and offer guidance when a symbol is missing:
> 
>    WARNING: linux-yocto-4.4.15+gitAUTOINC+b030d96c7b_f5e2c49d58-r0
> do_kernel_configcheck: [kernel config]: specified values did not make it
> into the kernel's final configuration:
> 
>    -- CONFIG_BT_6LOWPAN -
>    Config: CONFIG_BT_6LOWPAN
>    From: /home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel-
> source/.kernel-meta/configs/standard/features/bluetooth/bluetooth.cfg
>    Requested value:  CONFIG_BT_6LOWPAN=y
>    Actual value:
> 
>    Config 'BT_6LOWPAN' has the following conditionals:
>  BT_LE && 6LOWPAN (value: "n")
>    Dependency values are:
>  BT_LE [y] 6LOWPAN [n]
> 
> Signed-off-by: Bruce Ashfield 
> ---
>  meta/classes/kernel-yocto.bbclass  | 143 --
> ---
>  .../kern-tools/kern-tools-native_git.bb|   4 +-
>  2 files changed, 55 insertions(+), 92 deletions(-)
> 
> diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-
> yocto.bbclass
> index a9d4205..8650e55 100644
> --- a/meta/classes/kernel-yocto.bbclass
> +++ b/meta/classes/kernel-yocto.bbclass
> @@ -119,77 +119,42 @@ do_kernel_metadata() {
>   patches="${@" ".join(find_patches(d))}"
>   feat_dirs="${@" ".join(find_kernel_feature_dirs(d))}"
>  
> - # add any explicitly referenced features onto the end of the
> feature
> - # list that is passed to the kernel build scripts.
> - if [ -n "${KERNEL_FEATURES}" ]; then
> - for feat in ${KERNEL_FEATURES}; do
> - addon_features="$addon_features --feature $feat"
> - done
> - fi
> -
>   # check for feature directories/repos/branches that were part of
> the
>   # SRC_URI. If they were supplied, we convert them into include
> directives
>   # for the update part of the process
> - if [ -n "${feat_dirs}" ]; then
> - for f in ${feat_dirs}; do
> + for f in ${feat_dirs}; do
>   if [ -d "${WORKDIR}/$f/meta" ]; then
> - includes="$includes -I${WORKDIR}/$f/meta"
> + includes="$includes -I${WORKDIR}/$f/meta"
>   elif [ -d "${WORKDIR}/$f" ]; then
> - includes="$includes -I${WORKDIR}/$f"
> + includes="$includes -I${WORKDIR}/$f"
>   fi
> - done
> - fi
> + done
> + for s in ${sccs}; do
> + sdir=$(dirname $s)
> + includes="$includes -I${sdir}"
> + done
>  
> - # updates or generates the target description
> - updateme ${updateme_flags}
> -DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \
> - ${includes} ${addon_features} ${ARCH}
> ${KMACHINE} ${sccs} ${patches}
> - if [ $? -ne 0 ]; then
> - bbfatal_log "Could not update ${machine_branch}"
> - fi
> + # expand kernel features into their full path equivalents
> + bsp_definition=$(spp ${includes} --find -DKMACHINE=${KMACHINE}
> -DKTYPE=${LINUX_KERNEL_TYPE})
> + meta_dir=$(kgit --meta)
> +
> + # run1: pull all the configuration fragments, no matter where
> they come from
> + scc --force -o ${S}/${meta_dir}:cfg,meta ${includes}
> ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES}
> +
> + # run2: only 

[OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases

[Bruce Ashfield]

We've been running with a set of kern-tools that were designed to work
with build systems that knew nothing about git, trees, commits, etc.

As such, there's been a set of shims/wrappers in place to work with
within bitbake/oe-core. These were the *me scripts: createme, updateme,
patchme and configme.

With this commit, we strip that legacy code and use the tools directly.
This means less complexity, fewer corner cases .. and no surprises
when the tools are arunning. As another benefit, the tools consume
much less time during a typical build and have no noticeable impact
on the overall build time.

Existing .scc files, features, and processing are not impacted as
these tools are compatible with existing feature descriptions and
kerne configuration fragments.

The audit of kernel configuration fragments is now detached
from the linux-yocto build structure and process. This means that
they can eventually be tweaked to offer kernel audit to any type of
kernel build and configuration process.

Additionally, the kernel symbol audit phase can now resolve symbol
dependencies and offer guidance when a symbol is missing:

   WARNING: linux-yocto-4.4.15+gitAUTOINC+b030d96c7b_f5e2c49d58-r0 
do_kernel_configcheck: [kernel config]: specified values did not make it into 
the kernel's final configuration:

   -- CONFIG_BT_6LOWPAN -
   Config: CONFIG_BT_6LOWPAN
   From: 
/home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel-source/.kernel-meta/configs/standard/features/bluetooth/bluetooth.cfg
   Requested value:  CONFIG_BT_6LOWPAN=y
   Actual value:

   Config 'BT_6LOWPAN' has the following conditionals:
 BT_LE && 6LOWPAN (value: "n")
   Dependency values are:
 BT_LE [y] 6LOWPAN [n]

Signed-off-by: Bruce Ashfield 
---
 meta/classes/kernel-yocto.bbclass  | 143 -
 .../kern-tools/kern-tools-native_git.bb|   4 +-
 2 files changed, 55 insertions(+), 92 deletions(-)

diff --git a/meta/classes/kernel-yocto.bbclass 
b/meta/classes/kernel-yocto.bbclass
index a9d4205..8650e55 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -119,77 +119,42 @@ do_kernel_metadata() {
patches="${@" ".join(find_patches(d))}"
feat_dirs="${@" ".join(find_kernel_feature_dirs(d))}"
 
-   # add any explicitly referenced features onto the end of the feature
-   # list that is passed to the kernel build scripts.
-   if [ -n "${KERNEL_FEATURES}" ]; then
-   for feat in ${KERNEL_FEATURES}; do
-   addon_features="$addon_features --feature $feat"
-   done
-   fi
-
# check for feature directories/repos/branches that were part of the
# SRC_URI. If they were supplied, we convert them into include 
directives
# for the update part of the process
-   if [ -n "${feat_dirs}" ]; then
-   for f in ${feat_dirs}; do
+   for f in ${feat_dirs}; do
if [ -d "${WORKDIR}/$f/meta" ]; then
-   includes="$includes -I${WORKDIR}/$f/meta"
+   includes="$includes -I${WORKDIR}/$f/meta"
elif [ -d "${WORKDIR}/$f" ]; then
-   includes="$includes -I${WORKDIR}/$f"
+   includes="$includes -I${WORKDIR}/$f"
fi
-   done
-   fi
+   done
+   for s in ${sccs}; do
+   sdir=$(dirname $s)
+   includes="$includes -I${sdir}"
+   done
 
-   # updates or generates the target description
-   updateme ${updateme_flags} -DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \
- ${includes} ${addon_features} ${ARCH} ${KMACHINE} 
${sccs} ${patches}
-   if [ $? -ne 0 ]; then
-   bbfatal_log "Could not update ${machine_branch}"
-   fi
+   # expand kernel features into their full path equivalents
+   bsp_definition=$(spp ${includes} --find -DKMACHINE=${KMACHINE} 
-DKTYPE=${LINUX_KERNEL_TYPE})
+   meta_dir=$(kgit --meta)
+
+   # run1: pull all the configuration fragments, no matter where they come 
from
+   scc --force -o ${S}/${meta_dir}:cfg,meta ${includes} ${bsp_definition} 
${sccs} ${patches} ${KERNEL_FEATURES}
+
+   # run2: only generate patches for elements that have been passed on the 
SRC_URI
+   scc --force -o ${S}/${meta_dir}:patch --cmds patch ${includes} ${sccs} 
${patches} ${KERNEL_FEATURES}
 }
 
 do_patch() {
cd ${S}
 
-   # executes and modifies the source tree as required
-   patchme ${KMACHINE}
-   if [ $? -ne 0 ]; then
-   bberror "Could not apply patches for ${KMACHINE}."
-   bbfatal_log "Patch failures can be resolved in the linux source 
directory ${S})"
-   fi
-
-   # check to see if the specified SRCREV is reachable from the final 
branch.
-   # if it wasn't something wrong has happened, and we should error.
-