Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On 2016-09-01 11:18 PM, Bruce Ashfield wrote: On 2016-09-01 12:14 PM, André Draszik wrote: On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote: On 2016-08-31 4:54 AM, André Draszik wrote: On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote: Can you clarify for me if you are are using SRC_URI items tagged with 'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc items directly to the SRC_URI ? I do both: in the 1st layer, my kernel recipe boils down to: SRC_URI = "\ git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux- stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \ file://0001-a-patch.patch \ file://kernel-meta;type=kmeta;destsuffix=${KMETA} \ " KERNEL_FEATURES_append = " patches/some-patches.scc" KERNEL_FEATURES_append = " patches/more-patches.scc" KERNEL_FEATURES_append = " patches/even-more-patches.scc" some of the above in turn include additional .scc items recursively. In the 2nd layer, my .bbappend boils down to: FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:" KERNEL_FEATURES_append_tgm = " patches/tgm.scc" 2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd layer. I don't specifically add another SRC_URI item tagged with 'kmeta' (or any other explicit SRC_URI item for that matter). One more clarification. From our problem description, are you saying that in your runs, only the meta data from layer2 is getting copied and not that from layer1 ? That is what I had inferred from the error message: | DEBUG: Executing shell function do_kernel_metadata | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.mPP1jtatxm for details | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.v9LU9ccpbl for details | WARNING: exit code 1 from a shell command. I ran my sanity test, and saw meta data from both my layers copied .. so I'm worried that I misunderstood the issue you are seeing. I can see that they are actually copied indeed, but it doesn't find them... The file /tmp/tmp.v9LU9ccpbl mentioned above is empty. But looking at the code, I can definitely do some minor tweaks in this area .. I'm just trying to get the reproducer nailed down. I can try to create one for you if you prefer tomorrow, rather than wasting your time with this, if you don't get any further? I was able to trigger a similar trap to the above. And yes, it was sort of working by luck before. I spoke too soon. The error that I was able to trigger was valid, and in fact, when I tried the same test on krogoth it also failed there as well. So there is something I still don't have right in the two layers to trigger the same scenario you are seeing. If you could do that mock up, it would help immensely. Also, if you are open to changing the layer structure a bit, I could always suggest a new way to organize things. Cheers, Bruce I need to stare at this for a few hours and figure out the right way to restore similar behaviour. But now .. sleep. Cheers, Bruce Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On 2016-09-01 12:14 PM, André Draszik wrote: On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote: On 2016-08-31 4:54 AM, André Draszik wrote: On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote: Can you clarify for me if you are are using SRC_URI items tagged with 'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc items directly to the SRC_URI ? I do both: in the 1st layer, my kernel recipe boils down to: SRC_URI = "\ git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux- stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \ file://0001-a-patch.patch \ file://kernel-meta;type=kmeta;destsuffix=${KMETA} \ " KERNEL_FEATURES_append = " patches/some-patches.scc" KERNEL_FEATURES_append = " patches/more-patches.scc" KERNEL_FEATURES_append = " patches/even-more-patches.scc" some of the above in turn include additional .scc items recursively. In the 2nd layer, my .bbappend boils down to: FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:" KERNEL_FEATURES_append_tgm = " patches/tgm.scc" 2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd layer. I don't specifically add another SRC_URI item tagged with 'kmeta' (or any other explicit SRC_URI item for that matter). One more clarification. From our problem description, are you saying that in your runs, only the meta data from layer2 is getting copied and not that from layer1 ? That is what I had inferred from the error message: | DEBUG: Executing shell function do_kernel_metadata | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.mPP1jtatxm for details | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.v9LU9ccpbl for details | WARNING: exit code 1 from a shell command. I ran my sanity test, and saw meta data from both my layers copied .. so I'm worried that I misunderstood the issue you are seeing. I can see that they are actually copied indeed, but it doesn't find them... The file /tmp/tmp.v9LU9ccpbl mentioned above is empty. But looking at the code, I can definitely do some minor tweaks in this area .. I'm just trying to get the reproducer nailed down. I can try to create one for you if you prefer tomorrow, rather than wasting your time with this, if you don't get any further? I was able to trigger a similar trap to the above. And yes, it was sort of working by luck before. I need to stare at this for a few hours and figure out the right way to restore similar behaviour. But now .. sleep. Cheers, Bruce Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On 2016-09-01 12:14 PM, André Draszik wrote: On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote: On 2016-08-31 4:54 AM, André Draszik wrote: On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote: Can you clarify for me if you are are using SRC_URI items tagged with 'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc items directly to the SRC_URI ? I do both: in the 1st layer, my kernel recipe boils down to: SRC_URI = "\ git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux- stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \ file://0001-a-patch.patch \ file://kernel-meta;type=kmeta;destsuffix=${KMETA} \ " KERNEL_FEATURES_append = " patches/some-patches.scc" KERNEL_FEATURES_append = " patches/more-patches.scc" KERNEL_FEATURES_append = " patches/even-more-patches.scc" some of the above in turn include additional .scc items recursively. In the 2nd layer, my .bbappend boils down to: FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:" KERNEL_FEATURES_append_tgm = " patches/tgm.scc" 2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd layer. I don't specifically add another SRC_URI item tagged with 'kmeta' (or any other explicit SRC_URI item for that matter). One more clarification. From our problem description, are you saying that in your runs, only the meta data from layer2 is getting copied and not that from layer1 ? That is what I had inferred from the error message: | DEBUG: Executing shell function do_kernel_metadata | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.mPP1jtatxm for details | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.v9LU9ccpbl for details | WARNING: exit code 1 from a shell command. I ran my sanity test, and saw meta data from both my layers copied .. so I'm worried that I misunderstood the issue you are seeing. I can see that they are actually copied indeed, but it doesn't find them... The file /tmp/tmp.v9LU9ccpbl mentioned above is empty. Yah, that's a temporary file used in the processing. I'm going to clean up the logging a bit as I work through this. I think I can fix things with some path manipulations, but am still working on it. But looking at the code, I can definitely do some minor tweaks in this area .. I'm just trying to get the reproducer nailed down. I can try to create one for you if you prefer tomorrow, rather than wasting your time with this, if you don't get any further? I'm going to be trying this later tonight (Thursday, Eastern Time), but even if you can throw a mock up out, it will help me debug more. Bruce Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On Mi, 2016-08-31 at 16:17 -0400, Bruce Ashfield wrote: > On 2016-08-31 4:54 AM, André Draszik wrote: > > > > On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote: > > > > > > Can you clarify for me if you are are using SRC_URI items tagged with > > > 'kmeta', i.e. a directory of fragments, or are you just adding > > > .cfg/.scc > > > items directly to the SRC_URI ? > > > > I do both: > > > > in the 1st layer, my kernel recipe boils down to: > > > > SRC_URI = "\ > > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux- > > stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \ > > file://0001-a-patch.patch \ > > file://kernel-meta;type=kmeta;destsuffix=${KMETA} \ > > " > > > > KERNEL_FEATURES_append = " patches/some-patches.scc" > > KERNEL_FEATURES_append = " patches/more-patches.scc" > > KERNEL_FEATURES_append = " patches/even-more-patches.scc" > > > > some of the above in turn include additional .scc items recursively. > > > > > > In the 2nd layer, my .bbappend boils down to: > > > > FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:" > > > > KERNEL_FEATURES_append_tgm = " patches/tgm.scc" > > > > 2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd > > layer. I don't specifically add another SRC_URI item tagged with 'kmeta' > > (or > > any other explicit SRC_URI item for that matter). > > One more clarification. From our problem description, are you saying > that in your runs, only the meta data from layer2 is getting copied > and not that from layer1 ? That is what I had inferred from the error message: | DEBUG: Executing shell function do_kernel_metadata | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.mPP1jtatxm for details | ERROR. input file "patches/some-patches.scc" does not exist | ERROR: could not process input files: <2nd_layer>/recipes-kernel/linux/linux-ramips-4.4/tgm/defconfig <1st_layer>/recipes-kernel/linux/linux-ramips-4.4/0001-a-patch.patch patches/some-patches.scc patches/more-patches.scc patches/even-more-patches.scc patches/tgm.scc |See /tmp/tmp.v9LU9ccpbl for details | WARNING: exit code 1 from a shell command. > I ran my sanity test, and saw meta data from both my layers copied .. > so I'm worried that I misunderstood the issue you are seeing. I can see that they are actually copied indeed, but it doesn't find them... The file /tmp/tmp.v9LU9ccpbl mentioned above is empty. > But looking at the code, I can definitely do some minor tweaks in this > area .. I'm just trying to get the reproducer nailed down. I can try to create one for you if you prefer tomorrow, rather than wasting your time with this, if you don't get any further? Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On 2016-08-31 4:54 AM, André Draszik wrote: On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote: Can you clarify for me if you are are using SRC_URI items tagged with 'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc items directly to the SRC_URI ? I do both: in the 1st layer, my kernel recipe boils down to: SRC_URI = "\ git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \ file://0001-a-patch.patch \ file://kernel-meta;type=kmeta;destsuffix=${KMETA} \ " KERNEL_FEATURES_append = " patches/some-patches.scc" KERNEL_FEATURES_append = " patches/more-patches.scc" KERNEL_FEATURES_append = " patches/even-more-patches.scc" some of the above in turn include additional .scc items recursively. In the 2nd layer, my .bbappend boils down to: FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:" KERNEL_FEATURES_append_tgm = " patches/tgm.scc" 2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd layer. I don't specifically add another SRC_URI item tagged with 'kmeta' (or any other explicit SRC_URI item for that matter). One more clarification. From our problem description, are you saying that in your runs, only the meta data from layer2 is getting copied and not that from layer1 ? I ran my sanity test, and saw meta data from both my layers copied .. so I'm worried that I misunderstood the issue you are seeing. But looking at the code, I can definitely do some minor tweaks in this area .. I'm just trying to get the reproducer nailed down. Bruce Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On Di, 2016-08-30 at 14:35 -0400, Bruce Ashfield wrote: > Can you clarify for me if you are are using SRC_URI items tagged with > 'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc > items directly to the SRC_URI ? I do both: in the 1st layer, my kernel recipe boils down to: SRC_URI = "\ git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git;protocol=git;nocheckout=1;branch=${KBRANCH} \ file://0001-a-patch.patch \ file://kernel-meta;type=kmeta;destsuffix=${KMETA} \ " KERNEL_FEATURES_append = " patches/some-patches.scc" KERNEL_FEATURES_append = " patches/more-patches.scc" KERNEL_FEATURES_append = " patches/even-more-patches.scc" some of the above in turn include additional .scc items recursively. In the 2nd layer, my .bbappend boils down to: FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}-4.4:" KERNEL_FEATURES_append_tgm = " patches/tgm.scc" 2nd-layer.scc is located in ${THISDIR}/kernel-meta/patches/ in the 2nd layer. I don't specifically add another SRC_URI item tagged with 'kmeta' (or any other explicit SRC_URI item for that matter). Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On 2016-08-30 10:19 AM, André Draszik wrote: On Di, 2016-08-30 at 09:05 -0400, Bruce Ashfield wrote: On 2016-08-30 05:05 AM, André Draszik wrote: My kmeta stuff is located directly inside two meta-* layers (i.e. not in an extra git repo), where the 2nd layer's kernel recipes .bbappends to the 1st one with additional kmeta things. This used to work fine, but since this commit it doesn't anymore as only the kmeta stuff from the 2nd layer is being copied into workdir. Have I been doing something that wasn't guaranteed / supposed to work in the first place? Or should that still work? Is it worth fixing or should I rethink my approach? That was complexity that I had tried to remove, but thinking on it a bit, I think I can restore that without adding much complexity. I'm just finishing work on the 4.8 kernel today, and can look at this tomorrow. Thank you Bruce! Can you clarify for me if you are are using SRC_URI items tagged with 'kmeta', i.e. a directory of fragments, or are you just adding .cfg/.scc items directly to the SRC_URI ? Bruce Do you have any public layers that I can reference to pull together a test case ? Unfortunately not :-( But I'm happy to test and report back! Cheers, Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On Di, 2016-08-30 at 09:05 -0400, Bruce Ashfield wrote: > On 2016-08-30 05:05 AM, André Draszik wrote: > > > > My kmeta stuff is located directly inside two meta-* layers (i.e. not in > > an > > extra git repo), where the 2nd layer's kernel recipes .bbappends to the > > 1st > > one with additional kmeta things. > > > > This used to work fine, but since this commit it doesn't anymore as only > > the > > kmeta stuff from the 2nd layer is being copied into workdir. > > > > Have I been doing something that wasn't guaranteed / supposed to work in > > the > > first place? Or should that still work? Is it worth fixing or should I > > rethink my approach? > > That was complexity that I had tried to remove, but thinking on it a > bit, I think I can restore that without adding much complexity. > > I'm just finishing work on the 4.8 kernel today, and can look at this > tomorrow. Thank you Bruce! > Do you have any public layers that I can reference to pull together a > test case ? Unfortunately not :-( But I'm happy to test and report back! Cheers, Andre' -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
On 2016-08-30 05:05 AM, André Draszik wrote: My kmeta stuff is located directly inside two meta-* layers (i.e. not in an extra git repo), where the 2nd layer's kernel recipes .bbappends to the 1st one with additional kmeta things. This used to work fine, but since this commit it doesn't anymore as only the kmeta stuff from the 2nd layer is being copied into workdir. Have I been doing something that wasn't guaranteed / supposed to work in the first place? Or should that still work? Is it worth fixing or should I rethink my approach? That was complexity that I had tried to remove, but thinking on it a bit, I think I can restore that without adding much complexity. I'm just finishing work on the 4.8 kernel today, and can look at this tomorrow. Do you have any public layers that I can reference to pull together a test case ? Bruce Cheers, Andre' On Mo, 2016-08-15 at 14:26 -0400, Bruce Ashfield wrote: We've been running with a set of kern-tools that were designed to work with build systems that knew nothing about git, trees, commits, etc. As such, there's been a set of shims/wrappers in place to work with within bitbake/oe-core. These were the *me scripts: createme, updateme, patchme and configme. With this commit, we strip that legacy code and use the tools directly. This means less complexity, fewer corner cases .. and no surprises when the tools are arunning. As another benefit, the tools consume much less time during a typical build and have no noticeable impact on the overall build time. Existing .scc files, features, and processing are not impacted as these tools are compatible with existing feature descriptions and kerne configuration fragments. The audit of kernel configuration fragments is now detached from the linux-yocto build structure and process. This means that they can eventually be tweaked to offer kernel audit to any type of kernel build and configuration process. Additionally, the kernel symbol audit phase can now resolve symbol dependencies and offer guidance when a symbol is missing: WARNING: linux-yocto-4.4.15+gitAUTOINC+b030d96c7b_f5e2c49d58-r0 do_kernel_configcheck: [kernel config]: specified values did not make it into the kernel's final configuration: -- CONFIG_BT_6LOWPAN - Config: CONFIG_BT_6LOWPAN From: /home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel- source/.kernel-meta/configs/standard/features/bluetooth/bluetooth.cfg Requested value: CONFIG_BT_6LOWPAN=y Actual value: Config 'BT_6LOWPAN' has the following conditionals: BT_LE && 6LOWPAN (value: "n") Dependency values are: BT_LE [y] 6LOWPAN [n] Signed-off-by: Bruce Ashfield --- meta/classes/kernel-yocto.bbclass | 143 -- --- .../kern-tools/kern-tools-native_git.bb| 4 +- 2 files changed, 55 insertions(+), 92 deletions(-) diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel- yocto.bbclass index a9d4205..8650e55 100644 --- a/meta/classes/kernel-yocto.bbclass +++ b/meta/classes/kernel-yocto.bbclass @@ -119,77 +119,42 @@ do_kernel_metadata() { patches="${@" ".join(find_patches(d))}" feat_dirs="${@" ".join(find_kernel_feature_dirs(d))}" - # add any explicitly referenced features onto the end of the feature - # list that is passed to the kernel build scripts. - if [ -n "${KERNEL_FEATURES}" ]; then - for feat in ${KERNEL_FEATURES}; do - addon_features="$addon_features --feature $feat" - done - fi - # check for feature directories/repos/branches that were part of the # SRC_URI. If they were supplied, we convert them into include directives # for the update part of the process - if [ -n "${feat_dirs}" ]; then - for f in ${feat_dirs}; do + for f in ${feat_dirs}; do if [ -d "${WORKDIR}/$f/meta" ]; then - includes="$includes -I${WORKDIR}/$f/meta" + includes="$includes -I${WORKDIR}/$f/meta" elif [ -d "${WORKDIR}/$f" ]; then - includes="$includes -I${WORKDIR}/$f" + includes="$includes -I${WORKDIR}/$f" fi - done - fi + done + for s in ${sccs}; do + sdir=$(dirname $s) + includes="$includes -I${sdir}" + done - # updates or generates the target description - updateme ${updateme_flags} -DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \ - ${includes} ${addon_features} ${ARCH} ${KMACHINE} ${sccs} ${patches} - if [ $? -ne 0 ]; then - bbfatal_log "Could not update ${machine_branch}" - fi + # expand kernel features into their full path equivalents + bsp_definition=$(spp ${includes} --find -DKMACHINE=${KMACHINE} -DKTYPE=${LINUX_KERNEL_TYPE}) + meta_dir=$(kgit --meta) + + # run1: pull all the configurati
Re: [OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
My kmeta stuff is located directly inside two meta-* layers (i.e. not in an extra git repo), where the 2nd layer's kernel recipes .bbappends to the 1st one with additional kmeta things. This used to work fine, but since this commit it doesn't anymore as only the kmeta stuff from the 2nd layer is being copied into workdir. Have I been doing something that wasn't guaranteed / supposed to work in the first place? Or should that still work? Is it worth fixing or should I rethink my approach? Cheers, Andre' On Mo, 2016-08-15 at 14:26 -0400, Bruce Ashfield wrote: > We've been running with a set of kern-tools that were designed to work > with build systems that knew nothing about git, trees, commits, etc. > > As such, there's been a set of shims/wrappers in place to work with > within bitbake/oe-core. These were the *me scripts: createme, updateme, > patchme and configme. > > With this commit, we strip that legacy code and use the tools directly. > This means less complexity, fewer corner cases .. and no surprises > when the tools are arunning. As another benefit, the tools consume > much less time during a typical build and have no noticeable impact > on the overall build time. > > Existing .scc files, features, and processing are not impacted as > these tools are compatible with existing feature descriptions and > kerne configuration fragments. > > The audit of kernel configuration fragments is now detached > from the linux-yocto build structure and process. This means that > they can eventually be tweaked to offer kernel audit to any type of > kernel build and configuration process. > > Additionally, the kernel symbol audit phase can now resolve symbol > dependencies and offer guidance when a symbol is missing: > > WARNING: linux-yocto-4.4.15+gitAUTOINC+b030d96c7b_f5e2c49d58-r0 > do_kernel_configcheck: [kernel config]: specified values did not make it > into the kernel's final configuration: > > -- CONFIG_BT_6LOWPAN - > Config: CONFIG_BT_6LOWPAN > From: /home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel- > source/.kernel-meta/configs/standard/features/bluetooth/bluetooth.cfg > Requested value: CONFIG_BT_6LOWPAN=y > Actual value: > > Config 'BT_6LOWPAN' has the following conditionals: > BT_LE && 6LOWPAN (value: "n") > Dependency values are: > BT_LE [y] 6LOWPAN [n] > > Signed-off-by: Bruce Ashfield > --- > meta/classes/kernel-yocto.bbclass | 143 -- > --- > .../kern-tools/kern-tools-native_git.bb| 4 +- > 2 files changed, 55 insertions(+), 92 deletions(-) > > diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel- > yocto.bbclass > index a9d4205..8650e55 100644 > --- a/meta/classes/kernel-yocto.bbclass > +++ b/meta/classes/kernel-yocto.bbclass > @@ -119,77 +119,42 @@ do_kernel_metadata() { > patches="${@" ".join(find_patches(d))}" > feat_dirs="${@" ".join(find_kernel_feature_dirs(d))}" > > - # add any explicitly referenced features onto the end of the > feature > - # list that is passed to the kernel build scripts. > - if [ -n "${KERNEL_FEATURES}" ]; then > - for feat in ${KERNEL_FEATURES}; do > - addon_features="$addon_features --feature $feat" > - done > - fi > - > # check for feature directories/repos/branches that were part of > the > # SRC_URI. If they were supplied, we convert them into include > directives > # for the update part of the process > - if [ -n "${feat_dirs}" ]; then > - for f in ${feat_dirs}; do > + for f in ${feat_dirs}; do > if [ -d "${WORKDIR}/$f/meta" ]; then > - includes="$includes -I${WORKDIR}/$f/meta" > + includes="$includes -I${WORKDIR}/$f/meta" > elif [ -d "${WORKDIR}/$f" ]; then > - includes="$includes -I${WORKDIR}/$f" > + includes="$includes -I${WORKDIR}/$f" > fi > - done > - fi > + done > + for s in ${sccs}; do > + sdir=$(dirname $s) > + includes="$includes -I${sdir}" > + done > > - # updates or generates the target description > - updateme ${updateme_flags} > -DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \ > - ${includes} ${addon_features} ${ARCH} > ${KMACHINE} ${sccs} ${patches} > - if [ $? -ne 0 ]; then > - bbfatal_log "Could not update ${machine_branch}" > - fi > + # expand kernel features into their full path equivalents > + bsp_definition=$(spp ${includes} --find -DKMACHINE=${KMACHINE} > -DKTYPE=${LINUX_KERNEL_TYPE}) > + meta_dir=$(kgit --meta) > + > + # run1: pull all the configuration fragments, no matter where > they come from > + scc --force -o ${S}/${meta_dir}:cfg,meta ${includes} > ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES} > + > + # run2: only generate patches for elements that
[OE-core] [PATCH 5/6] kernel-yocto: streamline patch, configuration and audit phases
We've been running with a set of kern-tools that were designed to work with build systems that knew nothing about git, trees, commits, etc. As such, there's been a set of shims/wrappers in place to work with within bitbake/oe-core. These were the *me scripts: createme, updateme, patchme and configme. With this commit, we strip that legacy code and use the tools directly. This means less complexity, fewer corner cases .. and no surprises when the tools are arunning. As another benefit, the tools consume much less time during a typical build and have no noticeable impact on the overall build time. Existing .scc files, features, and processing are not impacted as these tools are compatible with existing feature descriptions and kerne configuration fragments. The audit of kernel configuration fragments is now detached from the linux-yocto build structure and process. This means that they can eventually be tweaked to offer kernel audit to any type of kernel build and configuration process. Additionally, the kernel symbol audit phase can now resolve symbol dependencies and offer guidance when a symbol is missing: WARNING: linux-yocto-4.4.15+gitAUTOINC+b030d96c7b_f5e2c49d58-r0 do_kernel_configcheck: [kernel config]: specified values did not make it into the kernel's final configuration: -- CONFIG_BT_6LOWPAN - Config: CONFIG_BT_6LOWPAN From: /home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel-source/.kernel-meta/configs/standard/features/bluetooth/bluetooth.cfg Requested value: CONFIG_BT_6LOWPAN=y Actual value: Config 'BT_6LOWPAN' has the following conditionals: BT_LE && 6LOWPAN (value: "n") Dependency values are: BT_LE [y] 6LOWPAN [n] Signed-off-by: Bruce Ashfield --- meta/classes/kernel-yocto.bbclass | 143 - .../kern-tools/kern-tools-native_git.bb| 4 +- 2 files changed, 55 insertions(+), 92 deletions(-) diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass index a9d4205..8650e55 100644 --- a/meta/classes/kernel-yocto.bbclass +++ b/meta/classes/kernel-yocto.bbclass @@ -119,77 +119,42 @@ do_kernel_metadata() { patches="${@" ".join(find_patches(d))}" feat_dirs="${@" ".join(find_kernel_feature_dirs(d))}" - # add any explicitly referenced features onto the end of the feature - # list that is passed to the kernel build scripts. - if [ -n "${KERNEL_FEATURES}" ]; then - for feat in ${KERNEL_FEATURES}; do - addon_features="$addon_features --feature $feat" - done - fi - # check for feature directories/repos/branches that were part of the # SRC_URI. If they were supplied, we convert them into include directives # for the update part of the process - if [ -n "${feat_dirs}" ]; then - for f in ${feat_dirs}; do + for f in ${feat_dirs}; do if [ -d "${WORKDIR}/$f/meta" ]; then - includes="$includes -I${WORKDIR}/$f/meta" + includes="$includes -I${WORKDIR}/$f/meta" elif [ -d "${WORKDIR}/$f" ]; then - includes="$includes -I${WORKDIR}/$f" + includes="$includes -I${WORKDIR}/$f" fi - done - fi + done + for s in ${sccs}; do + sdir=$(dirname $s) + includes="$includes -I${sdir}" + done - # updates or generates the target description - updateme ${updateme_flags} -DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \ - ${includes} ${addon_features} ${ARCH} ${KMACHINE} ${sccs} ${patches} - if [ $? -ne 0 ]; then - bbfatal_log "Could not update ${machine_branch}" - fi + # expand kernel features into their full path equivalents + bsp_definition=$(spp ${includes} --find -DKMACHINE=${KMACHINE} -DKTYPE=${LINUX_KERNEL_TYPE}) + meta_dir=$(kgit --meta) + + # run1: pull all the configuration fragments, no matter where they come from + scc --force -o ${S}/${meta_dir}:cfg,meta ${includes} ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES} + + # run2: only generate patches for elements that have been passed on the SRC_URI + scc --force -o ${S}/${meta_dir}:patch --cmds patch ${includes} ${sccs} ${patches} ${KERNEL_FEATURES} } do_patch() { cd ${S} - # executes and modifies the source tree as required - patchme ${KMACHINE} - if [ $? -ne 0 ]; then - bberror "Could not apply patches for ${KMACHINE}." - bbfatal_log "Patch failures can be resolved in the linux source directory ${S})" - fi - - # check to see if the specified SRCREV is reachable from the final branch. - # if it wasn't something wrong has happened, and we should error. - machine_srcrev="${SRCREV_machine}" - if [ -z "${mac