Re: [OE-core] weston/wayland: OpenGL vs GLES2
Hi Khem, > opengl in OE context is usually not the same as desktop GL, it means the > distro > needs OpenGL supported from the platform. > It is then accentuated by other things like egl glesv2, glx etc. I think we > could > make it a bit better but this is something that sits in the middle of distro > and > machine features. OK, thanks for clarifying. So my takeaway is that if I see a recipe that conflates the 'opengl' DISTRO_FEATURE with desktop OpenGL, then it is a bug, right? Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#192837): https://lists.openembedded.org/g/openembedded-core/message/192837 Mute This Topic: https://lists.openembedded.org/mt/103286278/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] weston/wayland: OpenGL vs GLES2
Hi all,
Just want to check my understanding on something. It seems to me that the
Weston having:
REQUIRED_DISTRO_FEATURES = "wayland opengl
${@oe.utils.conditional('VIRTUAL-RUNTIME_init_manager', 'systemd', 'pam', '',
d)}"
...is overly restrictive because 'opengl' implies desktop OpenGL, whereas
Weston is perfectly capable of running on gles2. At the very least, some other
recipes in the system seem to conflate 'opengl' with desktop OpenGL.
For example, meta-oe/recipes-benchmark/glmark2/glmark2_git.bb.
Is my understanding correct? Am I missing something?
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192809):
https://lists.openembedded.org/g/openembedded-core/message/192809
Mute This Topic: https://lists.openembedded.org/mt/103286278/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] kernel-yocto, devtool-source.bbclass: fix 'devtool modify' for kernels
Hi Alexandre,
> The series causes:
>
> 2023-10-17 23:07:15,509 - oe-selftest - INFO -
> devtool.DevtoolUpgradeTests.test_devtool_modify_kernel_overrides
> (subunit.RemotedTestCase)
> 2023-10-17 23:07:15,675 - oe-selftest - INFO - ... FAIL
> Stderr:
> 2023-10-17 22:28:35,773 - oe-selftest - INFO - Adding: "include selftest.inc"
> in
> /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-
> 2667546/conf/local.conf
> 2023-10-17 22:28:35,773 - oe-selftest - INFO - Adding: "include bblayers.inc"
> in
> bblayers.conf
> 2023-10-17 23:07:15,680 - oe-selftest - INFO - 7: 10/32 116/544 (1008.52s) (0
> failed) (devtool.DevtoolUpgradeTests.test_devtool_modify_kernel_overrides)
> 2023-10-17 23:07:15,681 - oe-selftest - INFO -
> testtools.testresult.real._StringException: Traceback (most recent call last):
> File "/home/pokybuild/yocto-worker/oe-selftest-
> centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 2236, in
> test_devtool_modify_kernel_overrides
> self.assertSetEqual(set(tags), {"devtool-base", "devtool-patched"})
> File "/usr/lib64/python3.9/unittest/case.py", line 1097, in assertSetEqual
> self.fail(self._formatMessage(msg, standardMsg))
> File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
> raise self.failureException(msg)
> AssertionError: Items in the second set but not the first:
> 'devtool-base'
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobu
> ilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F79%2Fbuilds%2F592
> 7%2Fsteps%2F14%2Flogs%2Fstdio=05%7C01%7Cchris.laplante%40agilen
> t.com%7Ce69a6029f8c8428f140208dbd009d811%7Ca9c0bc098b46420693512b
> a12fb4a5c0%7C0%7C0%7C638332512937141361%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C3000%7C%7C%7C=ptHcRf3P8%2Fg6G%2BgA2ZXR6SSVuKJcZGOR
> UwUzbYoJ70E%3D=0
I'm unable to reproduce this failure locally. Is there any way to grab the temp
directories or otherwise access the filesystem on the Autobuilder to see what's
going on? I suspect the answer is 'no', but just thought I'd ask.
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189475):
https://lists.openembedded.org/g/openembedded-core/message/189475
Mute This Topic: https://lists.openembedded.org/mt/101926734/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] kernel-yocto, devtool-source.bbclass: fix 'devtool modify' for kernels
> Hello,
>
> The series causes:
>
> 2023-10-17 23:07:15,509 - oe-selftest - INFO -
> devtool.DevtoolUpgradeTests.test_devtool_modify_kernel_overrides
> (subunit.RemotedTestCase)
> 2023-10-17 23:07:15,675 - oe-selftest - INFO - ... FAIL
> Stderr:
> 2023-10-17 22:28:35,773 - oe-selftest - INFO - Adding: "include selftest.inc"
> in
> /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-
> 2667546/conf/local.conf
> 2023-10-17 22:28:35,773 - oe-selftest - INFO - Adding: "include bblayers.inc"
> in
> bblayers.conf
> 2023-10-17 23:07:15,680 - oe-selftest - INFO - 7: 10/32 116/544 (1008.52s) (0
> failed) (devtool.DevtoolUpgradeTests.test_devtool_modify_kernel_overrides)
> 2023-10-17 23:07:15,681 - oe-selftest - INFO -
> testtools.testresult.real._StringException: Traceback (most recent call last):
> File "/home/pokybuild/yocto-worker/oe-selftest-
> centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 2236, in
> test_devtool_modify_kernel_overrides
> self.assertSetEqual(set(tags), {"devtool-base", "devtool-patched"})
> File "/usr/lib64/python3.9/unittest/case.py", line 1097, in assertSetEqual
> self.fail(self._formatMessage(msg, standardMsg))
> File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
> raise self.failureException(msg)
> AssertionError: Items in the second set but not the first:
> 'devtool-base'
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuil
> der.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F79%2Fbuilds%2F5927%
> 2Fsteps%2F14%2Flogs%2Fstdio=05%7C01%7Cchris.laplante%40agilent.co
> m%7Ce69a6029f8c8428f140208dbd009d811%7Ca9c0bc098b46420693512ba12
> fb4a5c0%7C0%7C0%7C638332512937141361%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000%7C%7C%7C=ptHcRf3P8%2Fg6G%2BgA2ZXR6SSVuKJcZGORUw
> UzbYoJ70E%3D=0
I'll take a look, thanks. This series is not ready for other reasons besides
this though :(
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189419):
https://lists.openembedded.org/g/openembedded-core/message/189419
Mute This Topic: https://lists.openembedded.org/mt/101926734/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] kernel-yocto, devtool-source.bbclass: fix 'devtool modify' for kernels
Hi Bruce, > > Is setting up MACHINE too invasive? Or do we come up with a way of only > setting up override branches for overrides that are valid for the given > local.conf? I lean towards the latter approach for simplicity. > > > > I admit to not using devtool, and not following much of what has been going on > in the series. > > But unless we are changing the machine for other devtool modify workflows, I > wouldn't' see why we'd want to do that in the kernel either. > > Any type of valid override should work, as long as you get all the required > variables setup for that override. > > To the error you mentioned in the original email, what exactly is blowing up ? > KBUILD_DEFCONFIG is completely optional, and not used in most builds, so the > empty / unset case shouldn't have any issues. This is specifically for the case of in-tree defconfig, in which case KBUILD_DEFCONFIG is required. I'm working with meta-xilinx which AFAIK uses this exclusively for its kernel (https://github.com/Xilinx/meta-xilinx/blob/1a2cb511ff20c9de24e973df3b556febb9659df4/meta-xilinx-core/recipes-kernel/linux/linux-xlnx.inc#L30). I guess the workaround for now is to switch to using a out-of-tree defconfig. We are not setting MACHINE for other 'devtool modify' workflows, which makes me hesitant to add it here. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189417): https://lists.openembedded.org/g/openembedded-core/message/189417 Mute This Topic: https://lists.openembedded.org/mt/101926734/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] kernel-yocto, devtool-source.bbclass: fix 'devtool modify' for kernels
> Ugh, actually this needs more work. If a recipe sets KBUILD_DEFCONFIG based > on MACHINE overrides, then the no-override case will explode when we call > 'kernel_pre_patch(localdata)'. Since in that case, KBUILD_DEFCONFIG will be > unset. Actually, this approach is problematic in general because calling do_kernel_metadata with the overrides changed isn't enough - we'd also have to change MACHINE. E.g.: machine-a.conf: KBUILD_DEFCONFIG:machine-a = "" machine-b.conf: KBUILD_DEFCONFIG:machine-b = "" kernel-recipe.bb: SRC_URI = "" SRC_URI:append:machine-a = " " SRC_URI:append:machine-b = " " Assuming MACHINE=machine-a in local.conf, then 'devtool modify ' will fail to setup kernel metadata for 'machine-b'. Is setting up MACHINE too invasive? Or do we come up with a way of only setting up override branches for overrides that are valid for the given local.conf? I lean towards the latter approach for simplicity. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189415): https://lists.openembedded.org/g/openembedded-core/message/189415 Mute This Topic: https://lists.openembedded.org/mt/101926734/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] kernel-yocto, devtool-source.bbclass: fix 'devtool modify' for kernels
> From: Chris Laplante
>
> Fixes a couple of different issues that all conspired to break 'devtool
> modify' for
> many use cases with kernel-yocto recipes.
>
> To explain, we need to consider the basic flow of how 'devtool modify'
> works for a recipe using kernel-yocto.bbclass:
>
> ┌──┐
> │do_kernel_checkout│
> ├──┘
> │ Sets up ${S}
> │ devtool_post_unpack: sets up 'devtool' branch
> ▼
> ┌┐
> │do_validate_branches│
> ├┘
> │ Checks out the machine branch (derived from ${KBRANCH}) and performs
> validation
> ▼
> ┌──┐
> │do_kernel_metadata│
> ├──┘
> │ Generates the config and patch series.
> │
> ▼
> ┌┐
> │do_patch│
> └┘
>Applies patches from patch series.
Ugh, actually this needs more work. If a recipe sets KBUILD_DEFCONFIG based on
MACHINE overrides, then the no-override case will explode when we call
'kernel_pre_patch(localdata)'. Since in that case, KBUILD_DEFCONFIG will be
unset.
If anyone has ideas I'm all ears.
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189414):
https://lists.openembedded.org/g/openembedded-core/message/189414
Mute This Topic: https://lists.openembedded.org/mt/101926734/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/3] oeqa/selftest/devtool: add test for YOCTO# 14723
From: Chris Laplante
This relatively exhaustive test is designed to exercise the 'devtool
modify' workflow for kernel recipes, with a focus on SRC_URI override
branches.
Signed-off-by: Chris Laplante
---
meta/lib/oeqa/selftest/cases/devtool.py | 132
1 file changed, 132 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py
b/meta/lib/oeqa/selftest/cases/devtool.py
index a5ef0329df..1072e23c21 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -2197,3 +2197,135 @@ class DevtoolUpgradeTests(DevtoolBase):
#Step 4.5
runCmd("grep %s %s" % (modconfopt, codeconfigfile))
+
+def test_devtool_modify_kernel_overrides(self):
+"""
+[YOCTO #14723]
+
+Test that patches in SRC_URI overrides round-trip correctly through
devtool modify, especially when
+appends/prepends are present.
+"""
+import typing
+
+# Perform some initial setup
+kernel_provider = self.td['PREFERRED_PROVIDER_virtual/kernel']
+self.track_for_cleanup(self.workspacedir)
+self.add_command_to_tearDown('bitbake -c clean %s' % kernel_provider)
+self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+
+# Collect some information for later
+machine = get_bb_var("MACHINE")
+recipefile = get_bb_var('FILE', kernel_provider)
+recipedir = os.path.dirname(recipefile)
+res = re.search('recipes-.*', recipedir)
+self.assertTrue(res, 'Unable to find recipe subdirectory')
+recipesubdir = res[0]
+
+new_file_contents = "A new file"
+
+bitbake('%s -c clean' % kernel_provider)
+
+# We are going to call 'devtool modify' multiple times in this test,
so we use a context manager for the temp
+# dir rather than call 'track_for_cleanup', since we want the tempdirs
destroyed between sub-tests. Also it's
+# easier and clearer this way.
+with tempfile.TemporaryDirectory(prefix='devtoolqa') as tempdir:
+runCmd('devtool modify %s -x %s' % (kernel_provider, tempdir))
+self._check_src_repo(tempdir)
+
+tags = runCmd('git tag --points-at HEAD',
cwd=tempdir).output.strip().splitlines()
+self.assertSetEqual(set(tags), {"devtool-base", "devtool-patched"})
+
+# Construct a patch to add a file
+runCmd(f'echo "{new_file_contents}" > devtool-new-file',
cwd=tempdir)
+runCmd('git add devtool-new-file', cwd=tempdir)
+# Need to add the Upstream-Status otherwise the patch will be
rejected next time we 'modify'.
+runCmd('git commit -m "Add a new file\n\nUpstream-Status:
Inappropriate [OE self-test specific]"', cwd=tempdir)
+self.add_command_to_tearDown('rm -rf %s' %
os.path.join(self.testlayer_path, recipesubdir))
+runCmd('devtool finish %s meta-selftest' % kernel_provider)
+
+# Check patch was created
+patchfile = os.path.join(self.testlayer_path, recipesubdir,
kernel_provider, "0001-Add-a-new-file.patch")
+self.assertExists(patchfile, "Patch file doesn't exist")
+
+# Check bbappend was created
+appendfn = os.path.join(self.testlayer_path, recipesubdir,
'%s_%%.bbappend' % kernel_provider)
+self.assertExists(appendfn, "bbappend doesn't exist")
+
+def _assert_new_file_presence(exists: bool):
+file_path = os.path.join(tempdir, "devtool-new-file")
+if exists:
+with open(file_path, "r") as f:
+contents = f.read()
+self.assertEqual(contents, f"{new_file_contents}\n")
+else:
+self.assertNotExists(file_path)
+
+# Check that the patch round-trips
+with tempfile.TemporaryDirectory(prefix='devtoolqa') as tempdir:
+result = runCmd(f'devtool modify {kernel_provider} -x {tempdir}')
+self._check_src_repo(tempdir)
+_assert_new_file_presence(True)
+runCmd(f'devtool reset {kernel_provider}')
+
+def _modify_append_file(fn: typing.Callable[[str], str]):
+with open(appendfn, "r+") as f:
+contents = f.read()
+contents = fn(contents)
+f.seek(0)
+f.write(contents)
+f.truncate()
+
+# Modify bbappend to conditionally apply the patch
+_modify_append_file(lambda contents: contents.replace('SRC_URI += "',
f'SRC_URI:append:{machine} = " '))
+
+# Check that the patch still applies and that all branches/tags are
created as expected
+with tempfile.TemporaryDirectory(prefix='devtoolqa') as tempdir:
+result = runCmd(f'devtool modify {kernel_provider} -x {tempdir}')
+self._check_src_repo(tempdir)
+_assert_new_file_presence(True)
+runCmd(f'devtool reset {kernel_provider}')
+
+
[OE-core] [PATCH 2/3] oeqa/selftest/devtool: strengthen test_devtool_virtual_kernel_modify test
From: Chris Laplante
Call _check_src_repo to confirm that the 'devtool' branch is setup as we
expect. This would have caught the basic case of the bug (i.e. where
override branches are not involved).
Signed-off-by: Chris Laplante
---
meta/lib/oeqa/selftest/cases/devtool.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py
b/meta/lib/oeqa/selftest/cases/devtool.py
index b577f6d62a..a5ef0329df 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -2166,6 +2166,7 @@ class DevtoolUpgradeTests(DevtoolBase):
bitbake('%s -c clean' % kernel_provider)
#Step 4.1
runCmd('devtool modify virtual/kernel -x %s' % tempdir)
+self._check_src_repo(tempdir)
self.assertExists(os.path.join(tempdir, 'Makefile'), 'Extracted source
could not be found')
#Step 4.2
configfile = os.path.join(tempdir,'.config')
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189022):
https://lists.openembedded.org/g/openembedded-core/message/189022
Mute This Topic: https://lists.openembedded.org/mt/101926733/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/3] kernel-yocto, devtool-source.bbclass: fix 'devtool modify' for kernels
From: Chris Laplante
Fixes a couple of different issues that all conspired to break 'devtool
modify' for many use cases with kernel-yocto recipes.
To explain, we need to consider the basic flow of how 'devtool modify'
works for a recipe using kernel-yocto.bbclass:
┌──┐
│do_kernel_checkout│
├──┘
│ Sets up ${S}
│ devtool_post_unpack: sets up 'devtool' branch
▼
┌┐
│do_validate_branches│
├┘
│ Checks out the machine branch (derived from ${KBRANCH}) and performs
validation
▼
┌──┐
│do_kernel_metadata│
├──┘
│ Generates the config and patch series.
│
▼
┌┐
│do_patch│
└┘
Applies patches from patch series.
The first issue becomes clear when visualizing the flow above. The
'devtool' branch is checked out during 'do_kernel_checkout', but then
'do_validate_branches' stomps on it by checking out its machine branch.
So fix (1) is to add a postfunc to 'do_validate_branches' to checkout
the 'devtool' branch again.
Next, we need to look at the flow and consider how things work when
SRC_URI override branches are involved.
Consider:
SRC_URI:append:fake-machine = " file://0001-my-patch.patch"
SRC_URI:append:fake-machine-2 = " file://0001-my-patch.patch"
Assuming neither overrides are active, we'd expect a 'devtool' branch
that just points to the initial rev, then
'devtool-override-fake-machine' and 'devtool-override-fake-machine-2'
branches that each point to the same commit for 0001-my-patch.
Setting aside the matter of how the override branch set is determined,
the flow looks like this:
┌──┐
│do_kernel_checkout│
├──┘
│ Sets up ${S}
│ devtool_post_unpack: sets up 'devtool' branch
▼
┌┐
│do_validate_branches│
├┘
│ Checks out the machine branch (derived from ${KBRANCH} and performs
validation
▼
┌──┐
│do_kernel_metadata│
├──┘
│ Generates the config and patch series.
│
▼
┌┐
│do_patch│
└┘
Applies patches from patch series.
devtool_post_patch:
┌─► for each extra override... ──┐
│▼
│ create devtool-override- branch
││
│▼
│ set OVERRIDES/FILESOVERRIDES
││
│┌───▼┐
││do_patch│
│└───┬┘
││
└┘
In the loop, we set OVERRIDES & FILESOVERRIDES such that
SRC_URI contains the correct patches for each override. But
when we call 'do_patch', it is still using the patch series file that
was generated during the call to 'do_kernel_metadata'. So the correct
patches are not applied.
The solution to this issue is to insert a call to 'do_kernel_metadata'
in between setting OVERRIDES & FILESOVERRIDES and the call to
'do_patch'. We do need to slightly tweak 'do_kernel_metadata' to be able
to clear out the previous 'fence post' files, otherwise in the example
above, the 0001-my-patch.patch would only be applied to the first
override branch that is processed.
[YOCTO #14723]
Signed-off-by: Chris Laplante
---
meta/classes-recipe/kernel-yocto.bbclass | 4
meta/classes/devtool-source.bbclass | 22 ++
2 files changed, 26 insertions(+)
diff --git a/meta/classes-recipe/kernel-yocto.bbclass
b/meta/classes-recipe/kernel-yocto.bbclass
index 4ac977b122..a5fb6e42f3 100644
--- a/meta/classes-recipe/kernel-yocto.bbclass
+++ b/meta/classes-recipe/kernel-yocto.bbclass
@@ -332,6 +332,10 @@ do_patch() {
if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
kgit_extra_args="--commit-sha author"
fi
+ if [ "${_DEVTOOL_RUNNING_DO_KERNEL_METADATA}" = "1" ]; then
+ # see devtool-source.bbclass for explanation
+ kgit-s2q --clean
+ fi
kgit-s2q --gen -v $kgit_extra_args --patches .kernel-meta/
if [ $? -ne 0 ]; then
bberror "Could not apply patches for ${KMACHINE}."
diff --git a/meta/classes/devtool-source.bbclass
b/meta/classes/devtool-source.bbclass
index a02b1e9b0e..b037c5612b 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -57,6 +57,7 @@ python() {
if is_kernel_yocto:
Re: [OE-core] [PATCH 2/2] oeqa/selftest/devtool: fail if non-selfest workspace layer present
> FYI I also sent a patch to fix this issue the day just before you :) You can > find it > here: > https://lists.op/ > enembedded.org%2Fg%2Fopenembedded- > core%2Fmessage%2F188767=05%7C01%7Cchris.laplante%40agilent.com > %7C34ce8f731f6f4590172108dbc89c6739%7Ca9c0bc098b46420693512ba12fb > 4a5c0%7C0%7C0%7C638324346300462677%7CUnknown%7CTWFpbGZsb3d8e > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D% > 7C3000%7C%7C%7C=X3V9v4pUxmNJh%2Fb0w9OVdzgvqiDXgfjQYB8iUb > 904BU%3D=0 > > I do a check directly on the setUpModule. I also noticed that > devtool.DevtoolAddTests.test_devtool_add just hangs if there is a local > workspace but I didn't investigate it ;) > > Cheers > Julien Ah, very nice :). It was really frustrating me (plus it was a Friday) which is why I spent the time to debug it. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188841): https://lists.openembedded.org/g/openembedded-core/message/188841 Mute This Topic: https://lists.openembedded.org/mt/101820643/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/2] recipetool: use context manager for tinfoil to avoid hang
Hi Alexandre, > > Thank you for your patches! > > Could you remember changing your git config for your next submission: > https://docs.yo/ > ctoproject.org%2Fnext%2Fcontributor-guide%2Fsubmit- > changes.html%23fixing-your-from- > identity=05%7C01%7Cchris.laplante%40agilent.com%7Cd8e4bf9a3b674c > 281b2408dbc841e009%7Ca9c0bc098b46420693512ba12fb4a5c0%7C0%7C0%7 > C638323957497350867%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM > DAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C > =edAZCGs8naAEfH%2FqL1IkJZ1xnUtaTzKd9OM28HHNeeg%3D > d=0 > > Thanks! Ah, my mistake, sorry! I will fix it right now. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188827): https://lists.openembedded.org/g/openembedded-core/message/188827 Mute This Topic: https://lists.openembedded.org/mt/101820642/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/2] recipetool: use context manager for tinfoil to avoid hang
If you individually run some of the devtool selftests, e.g. with:
oe-selftest -r devtool.DevtoolAddTests.test_devtool_add
then you bypass the sanity check in test_create_workspace that checks
for an existing workspace layer. Eventually, recipetool will be called
in a test, and tinfoil.prepare will fail. But because tinfoil.shutdown
doesn't get called, the thread spawned by bb.server.process.BBUIEventQueue
spins forever and recipetool never exits, hanging the test.
Signed-off-by: Chris Laplante
---
scripts/recipetool | 16 +---
1 file changed, 5 insertions(+), 11 deletions(-)
diff --git a/scripts/recipetool b/scripts/recipetool
index e2d585d2c5..9eaecfbde6 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -22,13 +22,6 @@ logger = scriptutils.logger_create('recipetool')
plugins = []
-def tinfoil_init(parserecipes):
-import bb.tinfoil
-import logging
-tinfoil = bb.tinfoil.Tinfoil(tracking=True)
-tinfoil.logger.setLevel(logger.getEffectiveLevel())
-tinfoil.prepare(not parserecipes)
-return tinfoil
def main():
@@ -67,8 +60,11 @@ def main():
scriptutils.logger_setup_color(logger, global_args.color)
-tinfoil = tinfoil_init(False)
-try:
+import bb.tinfoil
+with bb.tinfoil.Tinfoil(tracking=True) as tinfoil:
+tinfoil.logger.setLevel(logger.getEffectiveLevel())
+tinfoil.prepare(True)
+
for path in (tinfoil.config_data.getVar('BBPATH').split(':')
+ [scripts_path]):
pluginpath = os.path.join(path, 'lib', 'recipetool')
@@ -100,8 +96,6 @@ def main():
ret = args.func(args)
except bb.BBHandledException:
ret = 1
-finally:
-tinfoil.shutdown()
return ret
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188797):
https://lists.openembedded.org/g/openembedded-core/message/188797
Mute This Topic: https://lists.openembedded.org/mt/101820642/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/2] oeqa/selftest/devtool: fail if non-selfest workspace layer present
The tests will fail anyway (since you will have two 'workspacelayer'
layers), so might as well make it fail faster and be clear.
Signed-off-by: Chris Laplante
---
meta/lib/oeqa/selftest/cases/devtool.py | 36 ++---
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py
b/meta/lib/oeqa/selftest/cases/devtool.py
index b577f6d62a..2c38eeca17 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -3,7 +3,7 @@
#
# SPDX-License-Identifier: MIT
#
-
+import dataclasses
import os
import re
import shutil
@@ -11,6 +11,7 @@ import tempfile
import glob
import fnmatch
import unittest
+from typing import List
from oeqa.selftest.case import OESelftestTestCase
from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer
@@ -90,6 +91,12 @@ def tearDownModule():
bb.utils.edit_bblayers_conf(bblayers_conf, None, None,
bblayers_edit_cb)
shutil.rmtree(templayerdir)
+@dataclasses.dataclass
+class BBLayerConfEntry:
+layer: str
+path: str
+priority: int
+
class DevtoolTestCase(OESelftestTestCase):
def setUp(self):
@@ -100,6 +107,29 @@ class DevtoolTestCase(OESelftestTestCase):
'This test cannot be run with a workspace directory '
'under the build directory')
+# Ensure there is no actual workspace dir enabled
+self.assertFalse(self.is_external_workspace_layer_enabled(),
+ "This test cannot be run with a workspace layer in
bblayers.conf")
+
+def is_selftest_workspace_layer_enabled(self):
+"""Returns true if the selftest workspace layer is enabled in
bblayers.conf"""
+return any([layer for layer in self._read_bblayers() if
+layer.layer == "workspacelayer" and layer.path ==
self.workspacedir])
+
+def is_external_workspace_layer_enabled(self):
+"""Returns true if a workspace layer (that wasn't installed by
selftest) is enabled in bblayers.conf"""
+return any([layer for layer in self._read_bblayers() if
+layer.layer == "workspacelayer" and layer.path !=
self.workspacedir])
+
+def _read_bblayers(self) -> List[BBLayerConfEntry]:
+"""Call bitbake-layers show-layers and return list of layers"""
+ret = []
+for line in runCmd('bitbake-layers show-layers').output.splitlines():
+line = line.strip()
+if match := re.match(r'^(\S+) +([^\n]*?) +(\d+)$', line):
+ret.append(BBLayerConfEntry(match[1], match[2], match[3]))
+return ret
+
def _check_src_repo(self, repo_dir):
"""Check srctree git repository"""
self.assertTrue(os.path.isdir(os.path.join(repo_dir, '.git')),
@@ -327,8 +357,8 @@ class DevtoolTests(DevtoolBase):
def test_create_workspace(self):
# Check preconditions
-result = runCmd('bitbake-layers show-layers')
-self.assertTrue('\nworkspace' not in result.output, 'This test cannot
be run with a workspace layer in bblayers.conf')
+self.assertFalse(self.is_selftest_workspace_layer_enabled(),
+ "This test cannot be run with the selftest workspace
layer in bblayers.conf")
# remove conf/devtool.conf to avoid it corrupting tests
devtoolconf = os.path.join(self.builddir, 'conf', 'devtool.conf')
self.track_for_cleanup(devtoolconf)
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188798):
https://lists.openembedded.org/g/openembedded-core/message/188798
Mute This Topic: https://lists.openembedded.org/mt/101820643/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Useful for me to send more Python 3.12 patches?
Hi Richard, Thanks for accepting my patch from a few days ago :). Just wanted to check if you would like me to send more Python 3.12 compatibility patches. If there are bigger fish to fry I will hold off, otherwise I'm happy to help with making everything compatible. P.S. I ask because I didn't realize that Python 3.12 released on the same day as when I sent the patch. The tool I use to maintain Python envs (rye) dutifully updated to the latest Python version on the day I started some work on poky. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188774): https://lists.openembedded.org/g/openembedded-core/message/188774 Mute This Topic: https://lists.openembedded.org/mt/101799331/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] recipetool/create_buildsys_python: use importlib instead of imp
'imp' was deprecated in Python 3.4 and removed in 3.12. The
piece of importlib we use has been around since 3.3.
Signed-off-by: Chris Laplante
---
scripts/lib/recipetool/create_buildsys_python.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/recipetool/create_buildsys_python.py
b/scripts/lib/recipetool/create_buildsys_python.py
index 4675cc68fa..92468b2254 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -10,7 +10,7 @@ import codecs
import collections
import setuptools.command.build_py
import email
-import imp
+import importlib
import glob
import itertools
import logging
@@ -561,7 +561,6 @@ class PythonRecipeHandler(RecipeHandler):
return deps
def parse_pkgdata_for_python_packages(self):
-suffixes = [t[0] for t in imp.get_suffixes()]
pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
ldata = tinfoil.config_data.createCopy()
@@ -585,7 +584,7 @@ class PythonRecipeHandler(RecipeHandler):
continue
for fn in files_info:
-for suffix in suffixes:
+for suffix in importlib.machinery.all_suffixes():
if fn.endswith(suffix):
break
else:
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188662):
https://lists.openembedded.org/g/openembedded-core/message/188662
Mute This Topic: https://lists.openembedded.org/mt/101742170/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
> On Thu, 14 Sept 2023 at 21:54, Richard Purdie > wrote: > > The tools are already supposed to support doing this with local file > > sstate sources, they just do a bad job at getting the diffs right. One > > intent of this work item was to try and understand why they don't work > > and address that so at least for filesystem sstate mirrors, you can > > get better results. I don't know how we solve the remote http issue as yet. > > I ran a few experiments with bitbake -S printdiff, and so far I haven't yet > seen > an unhelpful result. Here's the most impressive one: > > $ bitbake libsolv-native (uses cmake-native) $ git revert version > update> $ rm -rf build/tmp/ $ bitbake -S printdiff libsolv-native ... > The differences between the current build and any cached tasks start at the > following tasks: > /srv/work/alex/poky/meta/recipes-devtools/cmake/cmake- > native_3.27.4.bb:do_recipe_qa > NOTE: Writing task signature files > Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc > > Task cmake-native:do_recipe_qa couldn't be used from the cache because: > We need hash > 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb, > closest matching task was > 85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7 > basehash changed from > 7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to > 7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09 > Variable PV value changed from '3.27.5' to '3.27.4' > Variable SRC_URI[sha256sum] value changed from > '5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9' > to > '0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af' > > That's pretty good, isn't it? It does print both what needs to be re-run, and > *why* as well. > > I have no idea yet what kind of magic it does to find the 'closest matching > task' > in sstate, but if this breaks down in some other scenarios, we need to find > them > to get a starting point for making the tools better. Ideas? I'm ready to try > them > :) That is very impressive and I'd also love to hear about what heuristics it uses. We use Artifactory to host our sstate. (Artifactory doesn’t have specific support for it, it's just basically acting as a generic HTTP server. But it makes things like LDAP easy). I have for a while been thinking of building a tool that tries to find the "closest" sstate on the server and then recursively runs bitbake-diffsig on it. The tool was going to be called 'why-not-sstate'. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188008): https://lists.openembedded.org/g/openembedded-core/message/188008 Mute This Topic: https://lists.openembedded.org/mt/101501384/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe] [OE-core] Introducing yb - a new tool for Yocto environment setup/management
> The bitbake-layers command is needed only to generate the layer config file > (in json format) out of an already existing yocto setup. It does not perform > the actual layer fetch/setup from the json. That's the feature I like the most > in my proposal: none of the other tools bootstrap the config file in this way, > you need to always write it by hand. > > The actual layer setup (using the json) is performed by an independent, self- > contained python script (copied from a template in oe-core), which you can > place anywhere you want (e.g. in a product layer repo, or in a 'config stream > repo'), and which does not require an active yocto environment or a poky > checkout. The equivalant of 'yb sync' would be to pull that repo to get latest > revisions of the script and the json, then run the script pointing to an > existing > checkout - it will sync everything to the revisions in json. Ah, I misunderstood your approach then. That all makes sense and sounds very nice. Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#168829): https://lists.openembedded.org/g/openembedded-core/message/168829 Mute This Topic: https://lists.openembedded.org/mt/92801786/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Introducing yb - a new tool for Yocto environment setup/management
Hi Alex, > do you think any of these pieces could be moved to 'official yocto', > specifically as bitbake-layers subcommands? I read through the README, and > it seems that things like 'status' and 'run' would fit very well there. I would agree that 'status' and 'run' would be useful additions, though I can't say I'll have time to volunteer to do it :(. > Also note that there's a proposal for a json schema and layer tooling that > would be provided directly in oe-core: > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.y > octoproject.org%2Fpoky-contrib%2Flog%2F%3Fh%3Dakanavin%2Fsetup- > layersdata=05%7C01%7Cchris.laplante%40agilent.com%7C533ad5afc07 > 54a885d7808da75907dff%7Ca9c0bc098b46420693512ba12fb4a5c0%7C0%7C0 > %7C637951560728773611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA > wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C > %7Csdata=hN7QRlYqAPxigCSQfNlaCM7nWZ4vvSLv5IWzckYWAFU%3D > reserved=0 I saw that a few weeks ago and it is certainly good work :). (as are the other existing solutions, like kas, whisk, etc.) But IMHO bundling the layer setup tool inside bitbake itself feels a bit like the chicken-and-egg problem. I'd like my spec file (or JSON file, kas configuration file, whatever) to be a complete manifest of things to download. But with bitbake-layers, first I need to know the right version of poky and where to get it. Also, I think it would be hard to implement 'yb sync'-like functionality there, specifically if we wanted to be able to change between poky branches. Overall the thing I like about yb is that it is independent of Yocto/BitBake and easier to setup. Like an IDE, I can install it once and use it across multiple projects. > I can't help but say that there's a certain bit of irony in referring to > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxkcd > .com%2F1987%2Fdata=05%7C01%7Cchris.laplante%40agilent.com%7C > 533ad5afc0754a885d7808da75907dff%7Ca9c0bc098b46420693512ba12fb4a5c0 > %7C0%7C0%7C637951560728773611%7CUnknown%7CTWFpbGZsb3d8eyJWIj > oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3 > 000%7C%7C%7Csdata=uTq3p2HeA0Xfe3DMZ1fxkAxlHfOJNXM4za2XIC > v5OcA%3Dreserved=0 - when introducing yet another external tool > for yocto layer management :-) The irony is not lost on me :), especially given the layer setup tool conversation last month. Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#168827): https://lists.openembedded.org/g/openembedded-core/message/168827 Mute This Topic: https://lists.openembedded.org/mt/92800761/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Introducing yb - a new tool for Yocto environment setup/management
Hi all, Today I'm excited to publish a tool I've been developing internally for about a year now. It is called 'yb', and you can think of it like a cross between kas, Google's repo, and myrepos (mr). Project page: https://github.com/Agilent/yb Download it here: https://github.com/Agilent/yb/releases/tag/0.0.11 The tool is written in Rust. It is statically linked, so all you need to do is download the binary and put it somewhere in PATH. The primary contribution I believe this tool makes is the ability to not only setup Yocto environments, but keep them in sync with the rest of your team as your product(s) evolve. This is realized via the 'stream' mechanism - a git repo that holds your specs (basically like kas' configuration files). Most operations in yb automatically fetch updates to the active stream before doing anything. If you need to add or remove a layer from your build, just do it in the stream. Anyone using that stream will automatically get the update and be informed to update their environment. I am also very proud of the 'status' command. It works even with vanilla Yocto environments (i.e. what you're using today) as long as the tool can find "bitbake" on PATH. The "status" command will automatically do a 'git fetch' on each layer. When used along with streams/specs, it will also first check for updates to the stream. I hope you can give it a try and let me know what you find useful/broken. It is still in relatively early development (in particular the 'yb sync' command) but it is already helpful for my teams' day-to-day work. (P.S. Please forgive the multi-mailing list post) Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#168825): https://lists.openembedded.org/g/openembedded-core/message/168825 Mute This Topic: https://lists.openembedded.org/mt/92800761/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] How to use devicetree.bbclass?
We're using honister. We have a recipe that generates a device tree using
devicetree.bbclass. It doesn't do anything special:
SRC_URI = file://system-top.dts
DT_INCLUDE:append = " ${THISDIR}/files"
COMPATIBLE_MACHINE:zynq = "zynq"
inherit devicetree
It's unclear to me how this is actually supposed to work, however. There are no
examples of using the class in poky or meta-openembedded that I can find.
What I'm leaning towards now is creating a bbappend for the kernel recipe that
just installs the generated device tree into arch/arm/boot/dts. But if anyone
has a better way I'd be glad to hear it.
I've done a little more experimenting and I think I understand now. Please
someone correct me if I'm wrong.
If you want an in-tree (kernel source tree, that is) device tree, you set
KERNEL_DEVICETREE. AFAICT the device tree doesn't actually get baked into the
kernel. Rather, it is exported alongside the kernel to be picked up by a
fitImage, for example.
If you want an out-of-tree device tree, you can write a recipe that inherits
devicetree.bbclass (if you want to use the device tree compiler), or I suppose
you could write a recipe that just deploys a pre-built .dtb in a similar way to
how devicetree.bbclass does. Either way, you *don't* set KERNEL_DEVICETREE.
Then the fitimage process will pick up the .dtb.
In my case, I'm creating a fitimage without an initramfs. It's just the kernel
and dtb. This is done by adding this to my MACHINE conf:
KERNEL_CLASSES += "kernel-fitimage"
KERNEL_IMAGETYPES += "fitImage"
You also need to tell the kernel-fitimage.bbclass where to get the dtb from,
which is done somewhat indirectly by setting a preferred provider:
PREFERRED_PROVIDER_virtual/dtb = "my-device-tree"
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166755):
https://lists.openembedded.org/g/openembedded-core/message/166755
Mute This Topic: https://lists.openembedded.org/mt/91625593/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] How to use devicetree.bbclass?
We're using honister. We have a recipe that generates a device tree using
devicetree.bbclass. It doesn't do anything special:
SRC_URI = file://system-top.dts
DT_INCLUDE:append = " ${THISDIR}/files"
COMPATIBLE_MACHINE:zynq = "zynq"
inherit devicetree
It's unclear to me how this is actually supposed to work, however. There are no
examples of using the class in poky or meta-openembedded that I can find.
What I'm leaning towards now is creating a bbappend for the kernel recipe that
just installs the generated device tree into arch/arm/boot/dts. But if anyone
has a better way I'd be glad to hear it.
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166754):
https://lists.openembedded.org/g/openembedded-core/message/166754
Mute This Topic: https://lists.openembedded.org/mt/91625593/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] How to use devicetree.bbclass?
Hello,
We're using honister. We have a recipe that generates a device tree using
devicetree.bbclass. It doesn't do anything special:
SRC_URI = file://system-top.dts
DT_INCLUDE:append = " ${THISDIR}/files"
COMPATIBLE_MACHINE:zynq = "zynq"
inherit devicetree
It's unclear to me how this is actually supposed to work, however. There are no
examples of using the class in poky or meta-openembedded that I can find.
In my MACHINE conf, I tried to set:
KERNEL_DEVICETREE = "system-top.dtb"
but then Yocto thinks you're talking about a system-top.dtb that lives in the
kernel source arch/arm/boot/dtb.
I see in kernel-fitimage.bbclass there is something to do with
EXTERNAL_KERNEL_DEVICETREE, but as far as I can tell that has to do with
packaging the dtb as part of the fitimage. Which maybe is what I want? But then
how do I tell Yocto not to bundle the dtb with the kernel?
It is a shame that there is no documentation in the kernel dev manual about how
to deal with device trees. I'd offer to write it, but then of course first I'd
need to understand what I'm doing :(
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166753):
https://lists.openembedded.org/g/openembedded-core/message/166753
Mute This Topic: https://lists.openembedded.org/mt/91625593/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] bitbake-diffsigs with multiconfig is not helpful
Hi all, We are using Yocto Zeus (yes, I know...) and I'm trying to track down some reproducibility issues. My go-to tool is bitbake-diffsigs, but ever since we started using multiconfig, the tool is pretty much useless with results like this: (where mc-one and mc-two are multiconfigs) NOTE: Starting bitbake server... runtaskdeps changed: [-libidn/libidn2_2.2.0.bb:do_package:mc:mc-one libidn/libidn2_2.2.0.bb:do_packagedata:mc:mc-one pseudo/pseudo_git.bb:do_populate_sysroot:virtual:native:mc:mc-one rpm/rpm_4.14.2.1.bb:do_populate_sysroot:virtual:native:mc:mc-one, +libidn/libidn2_2.2.0.bb:do_package:mc:mc-two libidn/libidn2_2.2.0.bb:do_packagedata:mc:mc-two pseudo/pseudo_git.bb:do_populate_sysroot:virtual:native:mc:mc-two rpm/rpm_4.14.2.1.bb:do_populate_sysroot:virtual:native:mc:mc-two] libidn/libidn2_2.2.0.bb:do_package:mc:mc-one with hash 08ce20399ecf48464970bb3340b5cd5600661a296e679fc260d2bcac29ab2ce8 changed to libidn/libidn2_2.2.0.bb:do_package:mc:mc-two with hash a3a359d58c1cbe494935b7689b0a443251ac78666839742a53fd7083969e6e7d libidn/libidn2_2.2.0.bb:do_packagedata:mc:mc-one with hash 2c5da719673ea0b95c6c8010dad39fae7c6a403d4b72d22786a7caed45606ca3 changed to libidn/libidn2_2.2.0.bb:do_packagedata:mc:mc-two with hash 6082ecc1935134fc64f318bf998d65c22b5522aaa6c509026512787495b547f3 rpm/rpm_4.14.2.1.bb:do_populate_sysroot:virtual:native:mc:mc-one with hash 575c6ab8920d8699e291d10ad6211bfae36f4f3b5c682cceb660ef6cfab111e4 changed to rpm/rpm_4.14.2.1.bb:do_populate_sysroot:virtual:native:mc:mc-two with hash fed841d6093be9faf415bac2b9dd10f4aafdf6b5bb5075f7da45f2e7028d036d Dependency on task libidn/libidn2_2.2.0.bb:do_package:mc:mc-two was added with hash a3a359d58c1cbe494935b7689b0a443251ac78666839742a53fd7083969e6e7d Dependency on task rpm/rpm_4.14.2.1.bb:do_populate_sysroot:virtual:native:mc:mc-two was added with hash fed841d6093be9faf415bac2b9dd10f4aafdf6b5bb5075f7da45f2e7028d036d Dependency on task libidn/libidn2_2.2.0.bb:do_packagedata:mc:mc-two was added with hash 6082ecc1935134fc64f318bf998d65c22b5522aaa6c509026512787495b547f3 Dependency on task rpm/rpm_4.14.2.1.bb:do_populate_sysroot:virtual:native:mc:mc-one was removed with hash 575c6ab8920d8699e291d10ad6211bfae36f4f3b5c682cceb660ef6cfab111e4 Dependency on task libidn/libidn2_2.2.0.bb:do_packagedata:mc:mc-one was removed with hash 2c5da719673ea0b95c6c8010dad39fae7c6a403d4b72d22786a7caed45606ca3 Dependency on task libidn/libidn2_2.2.0.bb:do_package:mc:mc-one was removed with hash 08ce20399ecf48464970bb3340b5cd5600661a296e679fc260d2bcac29ab2ce8 The next thing to try is diffsigs on the do_package or the do_populate_sysroot siginfos. But so far in my testing, the rabbit hole never ends (granted I've only gone down 6 or 7 levels before giving up). I'm going to cobble together an automated tool to follow the trail for me, but I thought I'd post this first to see if anyone has any ideas. I have a few questions: 1. I would that the name of the multiconfig (e.g. mc-one vs mc-two) doesn't factor into the hash calculation. Would I be correct? 2. Is anyone aware of the situation improving in newer versions of Yocto? 3. We are also using hash equivalence. Could this have any effect on what I'm seeing? Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#161534): https://lists.openembedded.org/g/openembedded-core/message/161534 Mute This Topic: https://lists.openembedded.org/mt/89010103/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [hardknott][PATCH] bitbake: server: Fix early parsing errors preventing zombie bitbake
Hi Anuj, > Thanks, this is the right way to request a backport. But since this is a > bitbake > change, should have been sent to the bitbake list. I had picked this though so > no need to re-send. Ah right. Thank you! Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#155976): https://lists.openembedded.org/g/openembedded-core/message/155976 Mute This Topic: https://lists.openembedded.org/mt/85254139/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [hardknott][PATCH] bitbake: server: Fix early parsing errors preventing zombie bitbake
Hello, > If the client process never sends cooker data, the server timeout will be 0.0, > not None. This will prevent the server from exiting, as it is waiting for a > new > client. In particular, the client will disconnect with a bad "INHERIT" line, > such > as: > > INHERIT += "this-class-does-not-exist" > > Instead of checking explicitly for None, check for a false value, which means > either 0.0 or None. > > (Bitbake rev: 7668057f8565655611de23d143432d1c05dae3ce) > > Signed-off-by: Joshua Watt > Signed-off-by: Richard Purdie > (cherry picked from commit 881005ec405f26b8f7f4b41505d69579e83b69c4) Was this the correct for me to request application of this patch to hardknott branch? I've never tried to submit a patch to something other than master. No rush obviously, just wanted to check that I was doing it right. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#155803): https://lists.openembedded.org/g/openembedded-core/message/155803 Mute This Topic: https://lists.openembedded.org/mt/85254139/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [hardknott][PATCH] bitbake: server: Fix early parsing errors preventing zombie bitbake
From: Joshua Watt
From: Chris Laplante
If the client process never sends cooker data, the server timeout will
be 0.0, not None. This will prevent the server from exiting, as it is
waiting for a new client. In particular, the client will disconnect with
a bad "INHERIT" line, such as:
INHERIT += "this-class-does-not-exist"
Instead of checking explicitly for None, check for a false value, which
means either 0.0 or None.
(Bitbake rev: 7668057f8565655611de23d143432d1c05dae3ce)
Signed-off-by: Joshua Watt
Signed-off-by: Richard Purdie
(cherry picked from commit 881005ec405f26b8f7f4b41505d69579e83b69c4)
---
bitbake/lib/bb/server/process.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index 155e8d131fea..a0955722e336 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -147,7 +147,7 @@ class ProcessServer():
conn = newconnections.pop(-1)
fds.append(conn)
self.controllersock = conn
-elif self.timeout is None and not ready:
+elif not self.timeout and not ready:
serverlog("No timeout, exiting.")
self.quit = True
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155491):
https://lists.openembedded.org/g/openembedded-core/message/155491
Mute This Topic: https://lists.openembedded.org/mt/85254139/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Status of 'devtool modify virtual/kernel' when using a separate SRC_URI for kernel metadata?
Hello all,
Can someone please clarify expected behavior of 'devtool modify virtual/kernel'
when the kernel recipe has an additional SRC_URI entry for the kernel-metadata
repository? For example, the 2021.1 linux-xlnx recipe in the meta-xilinx-bsp
layer (part of meta-xilkinx) uses the following:
KERNELURI ?=
"git://github.com/Xilinx/linux-xlnx.git;protocol=https;name=machine"
YOCTO_META ?=
"git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=yocto-kmeta"
SRC_URI = "${KERNELURI};${SRCBRANCHARG} ${YOCTO_META}"
I am using Hardknott. When I devtool modify this recipe, I get 'linux-xlnx'
unpacked as expected to workspace/sources. But I don't see yocto-kmeta unpacked
anywhere (except under tmp during the build, of course). Is this the expected
behavior?
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153940):
https://lists.openembedded.org/g/openembedded-core/message/153940
Mute This Topic: https://lists.openembedded.org/mt/84253603/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH 1/1] classes/npm: utilize lockfile in do_fetch to avoid multiconfig problems
> > But it would happen even without multiconfig, if there is overlap in > > the node modules that two recipes try to fetch. For some reason the > > npmsw fetcher (or the proxy fetcher it is using, see > > bitbake/lib/bb/fetch2/npmsw.py around lines 168-181) are not guarding > > against two recipes trying to download the same module concurrently. > > Sorry, right. If this isn't multiconfig safe, its also not safe against > concurrent > builds against DL_DIR which we do support and allow. > > It does sound like there are locks missing in the fetcher code and this is > working around that. Could we add the right locks to the fetcher? Yes, I will try to look into adding the locks. I had sent this patch in the meantime just to make everyone aware of the problem :) Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#146627): https://lists.openembedded.org/g/openembedded-core/message/146627 Mute This Topic: https://lists.openembedded.org/mt/79627859/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH 1/1] classes/npm: utilize lockfile in do_fetch to avoid multiconfig problems
Hi Richard,
> >
> > +do_fetch[lockfiles] += "${DL_DIR}/npm2_fetch.lock"
> > +
> > def npm_global_configs(d):
> > """Get the npm global configuration"""
> > configs = []
>
> If its just multilib, you could put a ${BPN} in the lock name? or is it
> conflicting
> general recipes?
multiconfig, not multilib.
But it would happen even without multiconfig, if there is overlap in the node
modules that two recipes try to fetch. For some reason the npmsw fetcher (or
the proxy fetcher it is using, see bitbake/lib/bb/fetch2/npmsw.py around lines
168-181) are not guarding against two recipes trying to download the same
module concurrently.
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#146623):
https://lists.openembedded.org/g/openembedded-core/message/146623
Mute This Topic: https://lists.openembedded.org/mt/79627859/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 1/1] classes/npm: utilize lockfile in do_fetch to avoid multiconfig problems
From: Chris Laplante
The npmsw fetcher doesn't play nice with multiconfig for some reason.
Concurrent do_fetch tasks for npm recipes can try to download modules at
the same time to ${DL_DIR}/npm2, resulting in races and failed
downloads.
Signed-off-by: Chris Laplante
---
meta/classes/npm.bbclass | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 068032a1e5..3e56765a2f 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -41,6 +41,8 @@ NPM_PACKAGE = "${WORKDIR}/npm-package"
NPM_CACHE = "${WORKDIR}/npm-cache"
NPM_BUILD = "${WORKDIR}/npm-build"
+do_fetch[lockfiles] += "${DL_DIR}/npm2_fetch.lock"
+
def npm_global_configs(d):
"""Get the npm global configuration"""
configs = []
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#146620):
https://lists.openembedded.org/g/openembedded-core/message/146620
Mute This Topic: https://lists.openembedded.org/mt/79627859/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 0/1] Workaround for npm + multiconfig
From: Chris Laplante I didn't have time to dig into the npmsw fetcher to understand exactly why the conflicts happen, but this fixes things in the meantime. Chris Laplante (1): classes/npm: utilize lockfile in do_fetch to avoid multiconfig problems meta/classes/npm.bbclass | 2 ++ 1 file changed, 2 insertions(+) -- 2.17.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#146619): https://lists.openembedded.org/g/openembedded-core/message/146619 Mute This Topic: https://lists.openembedded.org/mt/79627856/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] contrib/git-hooks: add a sendemail-validate example hook that adds FROM: lines to outgoing patch emails
From: Chris Laplante
This is useful for people using Microsoft Exchange / Office 360, which
butchers patches causing author identity to be lost.
Signed-off-by: Chris Laplante
---
contrib/git-hooks/sendemail-validate.sample | 78 +
1 file changed, 78 insertions(+)
create mode 100755 contrib/git-hooks/sendemail-validate.sample
diff --git a/contrib/git-hooks/sendemail-validate.sample
b/contrib/git-hooks/sendemail-validate.sample
new file mode 100755
index 00..af5d55cb00
--- /dev/null
+++ b/contrib/git-hooks/sendemail-validate.sample
@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+
+# Copyright (C) 2020 Agilent Technologies, Inc.
+# Author: Chris Laplante
+
+# This sendemail-validate hook injects 'From: ' header lines into outgoing
+# emails sent via 'git send-email', to ensure that accurate commit authorship
+# information is present. It was created because some email servers
+# (notably Microsoft Exchange / Office 360) seem to butcher outgoing patches,
+# resulting in incorrect authorship.
+
+# Current limitations:
+# 1. Assumes one per patch per email
+# 2. Minimal error checking
+#
+# Installation:
+# 1. Copy to .git/hooks/sendemail-validate
+# 2. chmod +x .git/hooks/sendemail-validate
+
+
+import enum
+import re
+import subprocess
+import sys
+
+
+class Subject(enum.IntEnum):
+NOT_SEEN = 0
+CONSUMING = 1
+SEEN = 2
+
+
+def make_from_line():
+cmd = ["git", "var", "GIT_COMMITTER_IDENT"]
+proc = subprocess.run(cmd, check=True, stdout=subprocess.PIPE,
universal_newlines=True)
+regex = re.compile(r"^(.*>).*$")
+match = regex.match(proc.stdout)
+assert match is not None
+return "From: {0}".format(match.group(1))
+
+
+def main():
+email = sys.argv[1]
+
+with open(email, "r") as f:
+email_lines = f.read().split("\n")
+
+subject_seen = Subject.NOT_SEEN
+first_body_line = None
+for i, line in enumerate(email_lines):
+if (subject_seen == Subject.NOT_SEEN) and line.startswith("Subject: "):
+subject_seen = Subject.CONSUMING
+continue
+if subject_seen == Subject.CONSUMING:
+if not line.strip():
+subject_seen = Subject.SEEN
+continue
+if subject_seen == Subject.SEEN:
+first_body_line = i
+break
+
+assert subject_seen == Subject.SEEN
+assert first_body_line is not None
+
+from_line = make_from_line()
+# Only add FROM line if it is not already there
+if email_lines[first_body_line] != from_line:
+email_lines.insert(first_body_line, from_line)
+email_lines.insert(first_body_line + 1, "")
+with open(email, "w") as f:
+f.write("\n".join(email_lines))
+
+return 0
+
+
+if __name__ == "__main__":
+sys.exit(main())
+
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#146171):
https://lists.openembedded.org/g/openembedded-core/message/146171
Mute This Topic: https://lists.openembedded.org/mt/79223367/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] cases/bbtests.py: ensure PACKAGE_CLASSES is set to RPM for bbtests.BitbakeTests.test_force_task_1
From: Chris Laplante
This is because the test expects to find "do_package_write_rpm" in the
bitbake output.
Signed-off-by: Chris Laplante
---
meta/lib/oeqa/selftest/cases/bbtests.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/lib/oeqa/selftest/cases/bbtests.py
b/meta/lib/oeqa/selftest/cases/bbtests.py
index dc423ec439..79390acc0d 100644
--- a/meta/lib/oeqa/selftest/cases/bbtests.py
+++ b/meta/lib/oeqa/selftest/cases/bbtests.py
@@ -89,6 +89,7 @@ class BitbakeTests(OESelftestTestCase):
image_dir = bb_vars['D']
pkgsplit_dir = bb_vars['PKGDEST']
man_dir = bb_vars['mandir']
+self.write_config("PACKAGE_CLASSES = \"package_rpm\"")
bitbake('-c clean %s' % test_recipe)
bitbake('-c package -f %s' % test_recipe)
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#144788):
https://lists.openembedded.org/g/openembedded-core/message/144788
Mute This Topic: https://lists.openembedded.org/mt/78350813/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/4] staging: Ensure cleaned dependencies are added
> > Is this patch something that should be backported to zeus and friends? > > Its fixing a very rare problem that I only ran into whilst doing "interesting" > things most people don't probably do. > > If people were running into those issues I'd backport but for master it was > more a correctness fix which did fix some weird corner cases I debugged... A colleague today did see something that sounded to me like it may be fixed by this patch. It was a native tool that should have been available during a do_configure that wasn't. We didn't take the time to dig into it though - it was resolved with a -c cleansstate of the -native recipe and the target recipe. Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#143054): https://lists.openembedded.org/g/openembedded-core/message/143054 Mute This Topic: https://lists.openembedded.org/mt/77174124/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/4] staging: Ensure cleaned dependencies are added
Hi Richard, > Most recipe-sysroot dependencies are handled by these prefuncs. "configure" > is special with a decidated task, prepare_recipe_sysroot which runs > beforehand. > > do_prepare_recipe_sysroot does not have to be run before/after > fetch/unpack/patch, they're independent tasks. If fetch/unpack/patch have > sysroot dependencies and those tasks rerun, stale items from the sysroot > could be uninstalled and since prepare_recipe_sysroot doesn't re-run, they > could be missing by the time configure runs. > > Fix this by adding the prefunc for do_configure which would ensure the > sysroot is in a good state. Is this patch something that should be backported to zeus and friends? Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#143044): https://lists.openembedded.org/g/openembedded-core/message/143044 Mute This Topic: https://lists.openembedded.org/mt/77174124/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/2] cve-check: add CVE_CHECK_REPORT_PATCHED variable to suppress reporting of patched CVEs
Default behavior is not changed. To suppress patched CVEs, set:
CVE_CHECK_REPORT_PATCHED = ""
Signed-off-by: Chris Laplante
---
meta/classes/cve-check.bbclass | 38 --
1 file changed, 22 insertions(+), 16 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index df28a93687..25cefda92e 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -41,14 +41,16 @@ CVE_CHECK_MANIFEST ?=
"${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cve
CVE_CHECK_COPY_FILES ??= "1"
CVE_CHECK_CREATE_MANIFEST ??= "1"
+CVE_CHECK_REPORT_PATCHED ??= "1"
+
# Whitelist for packages (PN)
CVE_CHECK_PN_WHITELIST ?= ""
# Whitelist for CVE. If a CVE is found, then it is considered patched.
# The value is a string containing space separated CVE values:
-#
+#
# CVE_CHECK_WHITELIST = 'CVE-2014-2524 CVE-2018-1234'
-#
+#
CVE_CHECK_WHITELIST ?= ""
python cve_save_summary_handler () {
@@ -332,12 +334,15 @@ def cve_write_data(d, patched, unpatched, whitelisted,
cve_data):
bb.utils.mkdirhier(os.path.dirname(cve_file))
for cve in sorted(cve_data):
+is_patched = cve in patched
+if is_patched and (d.getVar("CVE_CHECK_REPORT_PATCHED") != "1"):
+continue
write_string += "PACKAGE NAME: %s\n" % d.getVar("PN")
write_string += "PACKAGE VERSION: %s%s\n" % (d.getVar("EXTENDPE"),
d.getVar("PV"))
write_string += "CVE: %s\n" % cve
if cve in whitelisted:
write_string += "CVE STATUS: Whitelisted\n"
-elif cve in patched:
+elif is_patched:
write_string += "CVE STATUS: Patched\n"
else:
unpatched_cves.append(cve)
@@ -351,19 +356,20 @@ def cve_write_data(d, patched, unpatched, whitelisted,
cve_data):
if unpatched_cves:
bb.warn("Found unpatched CVE (%s), for more information check %s" % ("
".join(unpatched_cves),cve_file))
-with open(cve_file, "w") as f:
-bb.note("Writing file %s with CVE information" % cve_file)
-f.write(write_string)
-
-if d.getVar("CVE_CHECK_COPY_FILES") == "1":
-deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE")
-bb.utils.mkdirhier(os.path.dirname(deploy_file))
-with open(deploy_file, "w") as f:
+if write_string:
+with open(cve_file, "w") as f:
+bb.note("Writing file %s with CVE information" % cve_file)
f.write(write_string)
-if d.getVar("CVE_CHECK_CREATE_MANIFEST") == "1":
-cvelogpath = d.getVar("CVE_CHECK_SUMMARY_DIR")
-bb.utils.mkdirhier(cvelogpath)
+if d.getVar("CVE_CHECK_COPY_FILES") == "1":
+deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE")
+bb.utils.mkdirhier(os.path.dirname(deploy_file))
+with open(deploy_file, "w") as f:
+f.write(write_string)
+
+if d.getVar("CVE_CHECK_CREATE_MANIFEST") == "1":
+cvelogpath = d.getVar("CVE_CHECK_SUMMARY_DIR")
+bb.utils.mkdirhier(cvelogpath)
-with open(d.getVar("CVE_CHECK_TMP_FILE"), "a") as f:
-f.write("%s" % write_string)
+with open(d.getVar("CVE_CHECK_TMP_FILE"), "a") as f:
+f.write("%s" % write_string)
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142906):
https://lists.openembedded.org/g/openembedded-core/message/142906
Mute This Topic: https://lists.openembedded.org/mt/77199089/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/2] cve-check: introduce CVE_CHECK_RECIPE_FILE variable to allow changing of per-recipe check file
The addition of this variable also makes it possible to change the
output suffix of the check files, e.g. in local.conf:
CVE_CHECK_MANIFEST_append = ".txt"
CVE_CHECK_RECIPE_FILE_append = ".txt"
Signed-off-by: Chris Laplante
---
meta/classes/cve-check.bbclass | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 02fef7c205..df28a93687 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -36,6 +36,7 @@ CVE_CHECK_SUMMARY_FILE_NAME ?= "cve-summary"
CVE_CHECK_SUMMARY_FILE ?=
"${CVE_CHECK_SUMMARY_DIR}/${CVE_CHECK_SUMMARY_FILE_NAME}"
CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"
+CVE_CHECK_RECIPE_FILE ?= "${CVE_CHECK_DIR}/${PN}"
CVE_CHECK_MANIFEST ?=
"${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cve"
CVE_CHECK_COPY_FILES ??= "1"
CVE_CHECK_CREATE_MANIFEST ??= "1"
@@ -118,7 +119,7 @@ python cve_check_write_rootfs_manifest () {
import shutil
if d.getVar("CVE_CHECK_COPY_FILES") == "1":
-deploy_file = os.path.join(d.getVar("CVE_CHECK_DIR"), d.getVar("PN"))
+deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE")
if os.path.exists(deploy_file):
bb.utils.remove(deploy_file)
@@ -355,9 +356,8 @@ def cve_write_data(d, patched, unpatched, whitelisted,
cve_data):
f.write(write_string)
if d.getVar("CVE_CHECK_COPY_FILES") == "1":
-cve_dir = d.getVar("CVE_CHECK_DIR")
-bb.utils.mkdirhier(cve_dir)
-deploy_file = os.path.join(cve_dir, d.getVar("PN"))
+deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE")
+bb.utils.mkdirhier(os.path.dirname(deploy_file))
with open(deploy_file, "w") as f:
f.write(write_string)
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142905):
https://lists.openembedded.org/g/openembedded-core/message/142905
Mute This Topic: https://lists.openembedded.org/mt/77199088/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] bitbake.conf: add name of multiconfig to BUILDCFG_HEADER when multiconfig is active
Signed-off-by: Chris Laplante
---
meta/conf/bitbake.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a318d1ca58..e6338b0c75 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -701,7 +701,7 @@ PREFERRED_PROVIDER_virtual/fakeroot-native ?=
"pseudo-native"
##
# Pre-build configuration output
-BUILDCFG_HEADER = "Build Configuration:"
+BUILDCFG_HEADER = "Build Configuration${@" (mc:${BB_CURRENT_MC})" if
d.getVar("BBMULTICONFIG") else ""}:"
BUILDCFG_VARS = "BB_VERSION BUILD_SYS NATIVELSBSTRING TARGET_SYS MACHINE
DISTRO DISTRO_VERSION TUNE_FEATURES TARGET_FPU"
BUILDCFG_VARS[type] = "list"
BUILDCFG_NEEDEDVARS = "TARGET_ARCH TARGET_OS"
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142856):
https://lists.openembedded.org/g/openembedded-core/message/142856
Mute This Topic: https://lists.openembedded.org/mt/77176224/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/3] cve-update-db-native: be less magical about checking whether the cve-check class is enabled
Signed-off-by: Chris Laplante
---
meta/recipes-core/meta/cve-update-db-native.bb | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index e4e2451bfd..c2f713312f 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -13,8 +13,7 @@ deltask do_install
deltask do_populate_sysroot
python () {
-cve_check_db_file = d.getVar("CVE_CHECK_DB_FILE")
-if not cve_check_db_file:
+if not bb.data.inherits_class("cve-check", d):
raise bb.parse.SkipRecipe("Skip recipe when cve-check class is not
loaded.")
if os.path.exists("%s-journal" % cve_check_db_file ):
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142540):
https://lists.openembedded.org/g/openembedded-core/message/142540
Mute This Topic: https://lists.openembedded.org/mt/76844932/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/3] cve-update-db-native: remove unused variable
Signed-off-by: Chris Laplante
---
meta/recipes-core/meta/cve-update-db-native.bb | 1 -
1 file changed, 1 deletion(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index 242fbc2f30..cf2b251e21 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -33,7 +33,6 @@ python do_fetch() {
db_file = d.getVar("CVE_CHECK_DB_FILE")
db_dir = os.path.dirname(db_file)
-json_tmpfile = os.path.join(db_dir, 'nvd.json.gz')
if os.path.exists("{0}-journal".format(db_file)):
# If a journal is present the last update might have been interrupted.
In that case,
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142542):
https://lists.openembedded.org/g/openembedded-core/message/142542
Mute This Topic: https://lists.openembedded.org/mt/76844934/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/3] cve-update-db-native: move -journal checking into do_fetch
It was always questionable to do this in an anonymous function, but now
with multiconfig it is a critical mistake and leads to more strange
"Exception: sqlite3.OperationalError: disk I/O error" errors.
Signed-off-by: Chris Laplante
---
meta/recipes-core/meta/cve-update-db-native.bb | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index c2f713312f..242fbc2f30 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -15,12 +15,6 @@ deltask do_populate_sysroot
python () {
if not bb.data.inherits_class("cve-check", d):
raise bb.parse.SkipRecipe("Skip recipe when cve-check class is not
loaded.")
-
-if os.path.exists("%s-journal" % cve_check_db_file ):
-os.remove("%s-journal" % cve_check_db_file)
-
-if os.path.exists(cve_check_db_file):
-os.remove(cve_check_db_file)
}
python do_fetch() {
@@ -41,6 +35,14 @@ python do_fetch() {
db_dir = os.path.dirname(db_file)
json_tmpfile = os.path.join(db_dir, 'nvd.json.gz')
+if os.path.exists("{0}-journal".format(db_file)):
+# If a journal is present the last update might have been interrupted.
In that case,
+# just wipe any leftovers and force the DB to be recreated.
+os.remove("{0}-journal".format(db_file))
+
+if os.path.exists(db_file):
+os.remove(db_file)
+
# Don't refresh the database more than once an hour
try:
import time
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142541):
https://lists.openembedded.org/g/openembedded-core/message/142541
Mute This Topic: https://lists.openembedded.org/mt/76844933/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 2/4] cve-check/cve-update-db-native: use lockfile to fix usage under multiconfig
Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same across
multiconfigs which led to a race condition wherein multiple
cve-update-db-native:do_populate_cve_db tasks could attempt to write to
the same sqlite database. This led to the following task failure:
Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function:
0001:
*** 0002:do_populate_cve_db(d)
0003:
File:
'/mnt/data/agent/work/74f119cccb44f133/yocto/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb',
lineno: 103, function: do_populate_cve_db
0099:if year == date.today().year:
0100:cve_f.write('CVE database update : %s\n\n' %
date.today())
0101:
0102:cve_f.close()
*** 0103:conn.commit()
0104:conn.close()
0105:}
0106:
0107:def initialize_db(c):
Exception: sqlite3.OperationalError: disk I/O error
Use a lockfile to ensure multiple tasks don't step over each other.
Signed-off-by: Chris Laplante
---
meta/classes/cve-check.bbclass | 1 +
meta/recipes-core/meta/cve-update-db-native.bb | 5 +++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 0889e7544a..35b7d0f298 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -27,6 +27,7 @@ CVE_VERSION ??= "${PV}"
CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.1.db"
+CVE_CHECK_DB_FILE_LOCK ?= "${CVE_CHECK_DB_FILE}.lock"
CVE_CHECK_LOG ?= "${T}/cve.log"
CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index 2221825bf8..d22b66f6c7 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -52,8 +52,7 @@ python do_populate_cve_db() {
cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
-if not os.path.isdir(db_dir):
-os.mkdir(db_dir)
+bb.utils.mkdirhier(db_dir)
# Connect to database
conn = sqlite3.connect(db_file)
@@ -114,6 +113,8 @@ python do_populate_cve_db() {
conn.close()
}
+do_populate_cve_db[lockfiles] += "${CVE_CHECK_DB_FILE_LOCK}"
+
def initialize_db(c):
c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE
TEXT)")
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142315):
https://lists.openembedded.org/g/openembedded-core/message/142315
Mute This Topic: https://lists.openembedded.org/mt/76742013/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 1/4] cve-update-db-native: add progress handler
Signed-off-by: Chris Laplante
---
.../recipes-core/meta/cve-update-db-native.bb | 90 ++-
1 file changed, 47 insertions(+), 43 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index 32d6dbdffc..2221825bf8 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -29,6 +29,7 @@ python do_populate_cve_db() {
Update NVD database with json data feed
"""
import bb.utils
+import bb.progress
import sqlite3, urllib, urllib.parse, shutil, gzip
from datetime import date
@@ -60,54 +61,57 @@ python do_populate_cve_db() {
initialize_db(c)
-for year in range(YEAR_START, date.today().year + 1):
-year_url = BASE_URL + str(year)
-meta_url = year_url + ".meta"
-json_url = year_url + ".json.gz"
+with bb.progress.ProgressHandler(d) as ph:
+total_years = date.today().year + 1 - YEAR_START
+for i, year in enumerate(range(YEAR_START, date.today().year + 1)):
+ph.update((float(i + 1) / total_years) * 100)
+year_url = BASE_URL + str(year)
+meta_url = year_url + ".meta"
+json_url = year_url + ".json.gz"
-# Retrieve meta last modified date
-try:
-response = urllib.request.urlopen(meta_url)
-except urllib.error.URLError as e:
-cve_f.write('Warning: CVE db update error, Unable to fetch CVE
data.\n\n')
-bb.warn("Failed to fetch CVE data (%s)" % e.reason)
-return
-
-if response:
-for l in response.read().decode("utf-8").splitlines():
-key, value = l.split(":", 1)
-if key == "lastModifiedDate":
-last_modified = value
-break
-else:
-bb.warn("Cannot parse CVE metadata, update failed")
-return
-
-# Compare with current db last modified date
-c.execute("select DATE from META where YEAR = ?", (year,))
-meta = c.fetchone()
-if not meta or meta[0] != last_modified:
-# Clear products table entries corresponding to current year
-c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' %
year,))
-
-# Update db with current year json file
+# Retrieve meta last modified date
try:
-response = urllib.request.urlopen(json_url)
-if response:
-update_db(c,
gzip.decompress(response.read()).decode('utf-8'))
-c.execute("insert or replace into META values (?, ?)", [year,
last_modified])
+response = urllib.request.urlopen(meta_url)
except urllib.error.URLError as e:
-cve_f.write('Warning: CVE db update error, CVE data is
outdated.\n\n')
-bb.warn("Cannot parse CVE data (%s), update failed" % e.reason)
+cve_f.write('Warning: CVE db update error, Unable to fetch CVE
data.\n\n')
+bb.warn("Failed to fetch CVE data (%s)" % e.reason)
return
-# Update success, set the date to cve_check file.
-if year == date.today().year:
-cve_f.write('CVE database update : %s\n\n' % date.today())
-
-cve_f.close()
-conn.commit()
-conn.close()
+if response:
+for l in response.read().decode("utf-8").splitlines():
+key, value = l.split(":", 1)
+if key == "lastModifiedDate":
+last_modified = value
+break
+else:
+bb.warn("Cannot parse CVE metadata, update failed")
+return
+
+# Compare with current db last modified date
+c.execute("select DATE from META where YEAR = ?", (year,))
+meta = c.fetchone()
+if not meta or meta[0] != last_modified:
+# Clear products table entries corresponding to current year
+c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%'
% year,))
+
+# Update db with current year json file
+try:
+response = urllib.request.urlopen(json_url)
+if response:
+update_db(c,
gzip.decompress(response.read()).decode('utf-8'))
+c.execute("insert or replace into META values (?, ?)",
[year, last_modified])
+except urllib.error.URLError as e:
+cve_f.write('Warning: CVE db update error, CVE data is
outdated.\n\n')
+bb.warn("Cannot parse CVE data (%s), update failed" %
e.reason)
+return
+
+# Update success, set the date to cve_check file.
+if year == date.today().year:
+cve_f.write('CVE
[OE-core] [PATCH v2 3/4] cve-update-db-native: use context manager for cve_f
---
meta/recipes-core/meta/cve-update-db-native.bb | 5 +
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index d22b66f6c7..328f6ab364 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -50,8 +50,6 @@ python do_populate_cve_db() {
except OSError:
pass
-cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
-
bb.utils.mkdirhier(db_dir)
# Connect to database
@@ -60,7 +58,7 @@ python do_populate_cve_db() {
initialize_db(c)
-with bb.progress.ProgressHandler(d) as ph:
+with bb.progress.ProgressHandler(d) as ph,
open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') as cve_f:
total_years = date.today().year + 1 - YEAR_START
for i, year in enumerate(range(YEAR_START, date.today().year + 1)):
ph.update((float(i + 1) / total_years) * 100)
@@ -108,7 +106,6 @@ python do_populate_cve_db() {
if year == date.today().year:
cve_f.write('CVE database update : %s\n\n' % date.today())
-cve_f.close()
conn.commit()
conn.close()
}
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142316):
https://lists.openembedded.org/g/openembedded-core/message/142316
Mute This Topic: https://lists.openembedded.org/mt/76742014/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 4/4] cve-check: avoid FileNotFoundError if no do_cve_check task has run
For example, if you just run 'bitbake cve-update-db-native' in a clean
build system, |cve_tmp_file| won't exist yet.
Signed-off-by: Chris Laplante
---
meta/classes/cve-check.bbclass | 13 +++--
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 35b7d0f298..17f64a8a9c 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -63,14 +63,15 @@ python cve_save_summary_handler () {
timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
cve_summary_file = os.path.join(cvelogpath, "%s-%s.txt" %
(cve_summary_name, timestamp))
-shutil.copyfile(cve_tmp_file, cve_summary_file)
+if os.path.exists(cve_tmp_file):
+shutil.copyfile(cve_tmp_file, cve_summary_file)
-if cve_summary_file and os.path.exists(cve_summary_file):
-cvefile_link = os.path.join(cvelogpath, cve_summary_name)
+if cve_summary_file and os.path.exists(cve_summary_file):
+cvefile_link = os.path.join(cvelogpath, cve_summary_name)
-if os.path.exists(os.path.realpath(cvefile_link)):
-os.remove(cvefile_link)
-os.symlink(os.path.basename(cve_summary_file), cvefile_link)
+if os.path.exists(os.path.realpath(cvefile_link)):
+os.remove(cvefile_link)
+os.symlink(os.path.basename(cve_summary_file), cvefile_link)
}
addhandler cve_save_summary_handler
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142313):
https://lists.openembedded.org/g/openembedded-core/message/142313
Mute This Topic: https://lists.openembedded.org/mt/76742011/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 0/4] cve-check fixes/enhancements
v1: initial v2: use lockfile in cve-update-db-native as suggested by Ross Burton Chris Laplante (4): cve-update-db-native: add progress handler cve-check/cve-update-db-native: use lockfile to fix usage under multiconfig cve-update-db-native: use context manager for cve_f cve-check: avoid FileNotFoundError if no do_cve_check task has run meta/classes/cve-check.bbclass| 14 +-- .../recipes-core/meta/cve-update-db-native.bb | 96 ++- 2 files changed, 57 insertions(+), 53 deletions(-) -- 2.17.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142312): https://lists.openembedded.org/g/openembedded-core/message/142312 Mute This Topic: https://lists.openembedded.org/mt/76742010/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig
> > But the databases will be identical so that's basically a waste of > > time. How about adding a lock file so the second task waits for the > > first to complete, and then does nothing as the database is up to date? > > That's a good point. I also considered just making it a bb.event.BuildStarted > event handler, like how uninative works. Should have mentioned, I like your approach better and am working on a v2 patch series. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142311): https://lists.openembedded.org/g/openembedded-core/message/142311 Mute This Topic: https://lists.openembedded.org/mt/76737830/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig
> On Wed, 9 Sep 2020 at 18:35, Chris Laplante via lists.openembedded.org > wrote: > > > > Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same > across > > multiconfigs which led to a race condition wherein multiple > > cve-update-db-native:do_populate_cve_db tasks could attempt to write > > to the same sqlite database. This led to the following task failure: > > But the databases will be identical so that's basically a waste of time. How > about adding a lock file so the second task waits for the first to complete, > and then does nothing as the database is up to date? That's a good point. I also considered just making it a bb.event.BuildStarted event handler, like how uninative works. Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142310): https://lists.openembedded.org/g/openembedded-core/message/142310 Mute This Topic: https://lists.openembedded.org/mt/76737830/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/4] cve-update-db-native: add progress handler
Signed-off-by: Chris Laplante
---
.../recipes-core/meta/cve-update-db-native.bb | 90 ++-
1 file changed, 47 insertions(+), 43 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index 32d6dbdffc..2221825bf8 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -29,6 +29,7 @@ python do_populate_cve_db() {
Update NVD database with json data feed
"""
import bb.utils
+import bb.progress
import sqlite3, urllib, urllib.parse, shutil, gzip
from datetime import date
@@ -60,54 +61,57 @@ python do_populate_cve_db() {
initialize_db(c)
-for year in range(YEAR_START, date.today().year + 1):
-year_url = BASE_URL + str(year)
-meta_url = year_url + ".meta"
-json_url = year_url + ".json.gz"
+with bb.progress.ProgressHandler(d) as ph:
+total_years = date.today().year + 1 - YEAR_START
+for i, year in enumerate(range(YEAR_START, date.today().year + 1)):
+ph.update((float(i + 1) / total_years) * 100)
+year_url = BASE_URL + str(year)
+meta_url = year_url + ".meta"
+json_url = year_url + ".json.gz"
-# Retrieve meta last modified date
-try:
-response = urllib.request.urlopen(meta_url)
-except urllib.error.URLError as e:
-cve_f.write('Warning: CVE db update error, Unable to fetch CVE
data.\n\n')
-bb.warn("Failed to fetch CVE data (%s)" % e.reason)
-return
-
-if response:
-for l in response.read().decode("utf-8").splitlines():
-key, value = l.split(":", 1)
-if key == "lastModifiedDate":
-last_modified = value
-break
-else:
-bb.warn("Cannot parse CVE metadata, update failed")
-return
-
-# Compare with current db last modified date
-c.execute("select DATE from META where YEAR = ?", (year,))
-meta = c.fetchone()
-if not meta or meta[0] != last_modified:
-# Clear products table entries corresponding to current year
-c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' %
year,))
-
-# Update db with current year json file
+# Retrieve meta last modified date
try:
-response = urllib.request.urlopen(json_url)
-if response:
-update_db(c,
gzip.decompress(response.read()).decode('utf-8'))
-c.execute("insert or replace into META values (?, ?)", [year,
last_modified])
+response = urllib.request.urlopen(meta_url)
except urllib.error.URLError as e:
-cve_f.write('Warning: CVE db update error, CVE data is
outdated.\n\n')
-bb.warn("Cannot parse CVE data (%s), update failed" % e.reason)
+cve_f.write('Warning: CVE db update error, Unable to fetch CVE
data.\n\n')
+bb.warn("Failed to fetch CVE data (%s)" % e.reason)
return
-# Update success, set the date to cve_check file.
-if year == date.today().year:
-cve_f.write('CVE database update : %s\n\n' % date.today())
-
-cve_f.close()
-conn.commit()
-conn.close()
+if response:
+for l in response.read().decode("utf-8").splitlines():
+key, value = l.split(":", 1)
+if key == "lastModifiedDate":
+last_modified = value
+break
+else:
+bb.warn("Cannot parse CVE metadata, update failed")
+return
+
+# Compare with current db last modified date
+c.execute("select DATE from META where YEAR = ?", (year,))
+meta = c.fetchone()
+if not meta or meta[0] != last_modified:
+# Clear products table entries corresponding to current year
+c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%'
% year,))
+
+# Update db with current year json file
+try:
+response = urllib.request.urlopen(json_url)
+if response:
+update_db(c,
gzip.decompress(response.read()).decode('utf-8'))
+c.execute("insert or replace into META values (?, ?)",
[year, last_modified])
+except urllib.error.URLError as e:
+cve_f.write('Warning: CVE db update error, CVE data is
outdated.\n\n')
+bb.warn("Cannot parse CVE data (%s), update failed" %
e.reason)
+return
+
+# Update success, set the date to cve_check file.
+if year == date.today().year:
+cve_f.write('CVE
[OE-core] [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig
Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same across
multiconfigs which led to a race condition wherein multiple
cve-update-db-native:do_populate_cve_db tasks could attempt to write to
the same sqlite database. This led to the following task failure:
Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function:
0001:
*** 0002:do_populate_cve_db(d)
0003:
File:
'/mnt/data/agent/work/74f119cccb44f133/yocto/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb',
lineno: 103, function: do_populate_cve_db
0099:if year == date.today().year:
0100:cve_f.write('CVE database update : %s\n\n' %
date.today())
0101:
0102:cve_f.close()
*** 0103:conn.commit()
0104:conn.close()
0105:}
0106:
0107:def initialize_db(c):
Exception: sqlite3.OperationalError: disk I/O error
The fix is to add a multiconfig-specific subdirectory to act as a layer of
separation.
Signed-off-by: Chris Laplante
---
meta/classes/cve-check.bbclass | 2 +-
meta/recipes-core/meta/cve-update-db-native.bb | 3 +--
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 0889e7544a..485d147ef8 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -25,7 +25,7 @@
CVE_PRODUCT ??= "${BPN}"
CVE_VERSION ??= "${PV}"
-CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
+CVE_CHECK_DB_DIR ?= "${DL_DIR}/${BB_CURRENT_MC}/CVE_CHECK"
CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.1.db"
CVE_CHECK_LOG ?= "${T}/cve.log"
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index 2221825bf8..57368caf73 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -52,8 +52,7 @@ python do_populate_cve_db() {
cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
-if not os.path.isdir(db_dir):
-os.mkdir(db_dir)
+bb.utils.mkdirhier(db_dir)
# Connect to database
conn = sqlite3.connect(db_file)
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142307):
https://lists.openembedded.org/g/openembedded-core/message/142307
Mute This Topic: https://lists.openembedded.org/mt/76737830/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/4] cve-update-db-native: use context manager for cve_f
---
meta/recipes-core/meta/cve-update-db-native.bb | 5 +
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb
b/meta/recipes-core/meta/cve-update-db-native.bb
index 57368caf73..f8f13af97c 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -50,8 +50,6 @@ python do_populate_cve_db() {
except OSError:
pass
-cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
-
bb.utils.mkdirhier(db_dir)
# Connect to database
@@ -60,7 +58,7 @@ python do_populate_cve_db() {
initialize_db(c)
-with bb.progress.ProgressHandler(d) as ph:
+with bb.progress.ProgressHandler(d) as ph,
open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') as cve_f:
total_years = date.today().year + 1 - YEAR_START
for i, year in enumerate(range(YEAR_START, date.today().year + 1)):
ph.update((float(i + 1) / total_years) * 100)
@@ -108,7 +106,6 @@ python do_populate_cve_db() {
if year == date.today().year:
cve_f.write('CVE database update : %s\n\n' % date.today())
-cve_f.close()
conn.commit()
conn.close()
}
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142308):
https://lists.openembedded.org/g/openembedded-core/message/142308
Mute This Topic: https://lists.openembedded.org/mt/76737831/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 4/4] cve-check: avoid FileNotFoundError if no do_cve_check task has run
For example, if you just run 'bitbake cve-update-db-native' in a clean
build system, |cve_tmp_file| won't exist yet.
Signed-off-by: Chris Laplante
---
meta/classes/cve-check.bbclass | 13 +++--
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 485d147ef8..9f272cd60c 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -62,14 +62,15 @@ python cve_save_summary_handler () {
timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
cve_summary_file = os.path.join(cvelogpath, "%s-%s.txt" %
(cve_summary_name, timestamp))
-shutil.copyfile(cve_tmp_file, cve_summary_file)
+if os.path.exists(cve_tmp_file):
+shutil.copyfile(cve_tmp_file, cve_summary_file)
-if cve_summary_file and os.path.exists(cve_summary_file):
-cvefile_link = os.path.join(cvelogpath, cve_summary_name)
+if cve_summary_file and os.path.exists(cve_summary_file):
+cvefile_link = os.path.join(cvelogpath, cve_summary_name)
-if os.path.exists(os.path.realpath(cvefile_link)):
-os.remove(cvefile_link)
-os.symlink(os.path.basename(cve_summary_file), cvefile_link)
+if os.path.exists(os.path.realpath(cvefile_link)):
+os.remove(cvefile_link)
+os.symlink(os.path.basename(cve_summary_file), cvefile_link)
}
addhandler cve_save_summary_handler
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142306):
https://lists.openembedded.org/g/openembedded-core/message/142306
Mute This Topic: https://lists.openembedded.org/mt/76737829/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [meta-oe][PATCH 0/3] Log colorizer
Hi Richard, > This looks interesting. Could we add something to the manual about how to > use it and maybe a test for it as well please? Halfway through writing the tests, and I realized LOG_COLORIZER_SUPPRESS_COLORIZED_OUTPUT doesn’t quite work. I wrote a test recipe with a do_compile that purposely fails with a compile error, to check whether color diagnostics work. But I'm getting color text printed out after "Log data follows: " which is knotty's response to the bb.build.TaskFailed. This is because of some code of _exec_task in build.py. 581 try: 582 for func in (prefuncs or '').split(): 583 exec_func(func, localdata) 584 exec_func(task, localdata) 585 for func in (postfuncs or '').split(): 586 exec_func(func, localdata) 587 except bb.BBHandledException: 588 event.fire(TaskFailed(task, fn, logfn, localdata, True), localdata) 589 return 1 590 except Exception as exc: 591 if quieterr: 592 event.fire(TaskFailedSilent(task, fn, logfn, localdata), localdata) 593 else: 594 errprinted = errchk.triggered 595 logger.error(str(exc)) 596 event.fire(TaskFailed(task, fn, logfn, localdata, errprinted), localdata) 597 return 1 The color output is coming from line 595. The CommandFailed exception from bb.process.run carries with it the stdout of the process, which in this case has color text. I think at this point, I question whether I should just integrate with bit BitBake bake directly. If I were to do that, I could also fix the limitation of .color and .nocolor logs not having output from pre/postfuncs. I propose this: log-colorizer.bbclass: 1. Responsible for setting up CFLAGS 2. Responsible for passing the appropriate flags/environment variables depending on build system in use (for example, the little bit of code I have targeting CMake). BitBake: 1. Ensure progress handlers only see filtered output. At the BitBake level I can just do this directly and not mess around with proxy progress handlers. 2. Create a "--suppress-color" flag that filters out ANSI escape sequences from the console and the task log 3. Maintain the .nocolor and .color versions of the logs. Though I'm not sure how I'd keep LOG_COLORIZER_TASKS-like functionality (which only writes .nocolor/.color logs for tasks we actually expect to produce color output). Maybe I should do it automatically based on whether any escape sequences are detected. The user would still be responsible for INHERIT'ing log-colorizer globally or per-recipe. What do you think? Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141102): https://lists.openembedded.org/g/openembedded-core/message/141102 Mute This Topic: https://lists.openembedded.org/mt/75836416/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [meta-oe][PATCH 0/3] Log colorizer
Hi Richard, > You can run these tests individually with "oe-selftest -r imagefeatures" to > run > that file or "oe-selftest -r imagefeatures.ImageFeatures.test_bmap" as an > example of a specific test. I'm developing the tests for log colorizer and I had a question about oe-selftest. Is there a way to re-run the tests without having to delete the build-st directory and start from scratch? It would save me time. $ oe-selftest -r log_colorizer.LogColorizerTests.test_log_colorizer_basic 2020-07-29 10:26:55,619 - oe-selftest - INFO - Adding layer libraries: 2020-07-29 10:26:55,620 - oe-selftest - INFO - /home/laplante/repos/oe-core/meta/lib 2020-07-29 10:26:55,620 - oe-selftest - INFO - /home/laplante/repos/oe-core/meta-selftest/lib 2020-07-29 10:26:55,620 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2020-07-29 10:26:57,190 - oe-selftest - ERROR - Build directory /home/laplante/repos/oe-core/build-st already exists, aborting Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141096): https://lists.openembedded.org/g/openembedded-core/message/141096 Mute This Topic: https://lists.openembedded.org/mt/75836416/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [meta-oe][PATCH 0/3] Log colorizer
Hi Richard, > This looks interesting. Could we add something to the manual about how to > use it and maybe a test for it as well please? > > I did write something similar in a different context recently which applies > here near enough as well: > > """ > By documentation, I mean that kernel-fitimage.bbclass has no information in > the reference manual: > > https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#ref- > classes-kernel-fitimage > > generated from: > http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/ref- > manual/ref-classes.xml > > and also there is also no header at the start of the class saying what it does > or how to use it. > > Could we add something to these locations to give more information about > the class? > > For testing, have a look at > meta/lib/oeqa/selftest/cases/imagefeatures.py. Its testing target image > rootfs settings but I believe the concept is similar to how you'd test > something like the kernel-fitimage class. > > You can run these tests individually with "oe-selftest -r imagefeatures" to > run > that file or "oe-selftest -r imagefeatures.ImageFeatures.test_bmap" as an > example of a specific test. > """ > > although there may be a different selftest which would be a better example > to base a test off for this code. Yes certainly, I'll work on them and submit a v2. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141067): https://lists.openembedded.org/g/openembedded-core/message/141067 Mute This Topic: https://lists.openembedded.org/mt/75836416/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [meta-oe][PATCH 3/3] log-colorizer.bbclass: add new class
This bbclass turns on compiler color diagnostics to make it easier to
visually interpret compiler errors and warnings. It can be used
per-recipe or globally (via INHERIT in local.conf).
You can set the LOG_COLORIZER_SUPPRESS_COLORIZED_OUTPUT variable to turn
off color output - this is intended for usage in a CI environment.
log.do_compile and log.do_configure task logs will contain the colorized
output. Non-colorized versions (log.do_.nocolor) and explicitly
colorized versions (log.do_.color) are created as well, regardless
of LOG_COLORIZER_SUPPRESS_COLORIZED_OUTPUT.
Signed-off-by: Chris Laplante
---
meta/classes/log-colorizer.bbclass | 49 ++
1 file changed, 49 insertions(+)
create mode 100644 meta/classes/log-colorizer.bbclass
diff --git a/meta/classes/log-colorizer.bbclass
b/meta/classes/log-colorizer.bbclass
new file mode 100644
index 00..4271957e28
--- /dev/null
+++ b/meta/classes/log-colorizer.bbclass
@@ -0,0 +1,49 @@
+# Copyright (C) 2020 Agilent Technologies, Inc.
+# Author: Chris Laplante
+#
+# Released under the MIT license (see COPYING.MIT)
+
+LOG_COLORIZER_SUPPRESS_COLORIZED_OUTPUT ?= ""
+LOG_COLORIZER_SUPPRESS_COLORIZED_OUTPUT[doc] = "If set, then console output
from the colorized tasks will be stripped of ANSI escape codes."
+
+LOG_COLORIZER_TASKS ?= " \
+configure \
+compile \
+"
+
+BB_SIGNATURE_EXCLUDE_FLAGS += "originalprogress"
+
+CFLAGS += "-fdiagnostics-color"
+
+python log_colorizer_eventhandler() {
+def is_task_enabled(task):
+return task in (d.getVar("__BBTASKS") or []) and "noexec" not in
d.getVarFlags(task)
+
+for task in set((d.getVar("LOG_COLORIZER_TASKS") or "").split()):
+if not task.startswith("do_"):
+task = "do_{0}".format(task)
+
+if not is_task_enabled(task):
+continue
+
+ph = d.getVarFlag(task, "progress")
+if ph:
+# Stash away the original progress handler
+d.setVarFlag(task, "originalprogress", ph)
+
+d.setVarFlag(task, "progress",
"custom:oe.log_colorizer.LogColorizerProxyProgressHandler")
+}
+
+addhandler log_colorizer_eventhandler
+log_colorizer_eventhandler[eventmask] = " \
+bb.event.RecipeTaskPreProcess \
+"
+
+python __anonymous() {
+if bb.data.inherits_class("cmake", d):
+# Inject environment variable to ensure CMake/Ninja gives colorized
output
+func = d.getVar("cmake_do_compile", False)
+if "export CLICOLOR_FORCE=1" not in [line.strip() for line in
func.split("\n")]:
+func = "\texport CLICOLOR_FORCE=1\n" + func
+d.setVar("cmake_do_compile", func)
+}
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#141022):
https://lists.openembedded.org/g/openembedded-core/message/141022
Mute This Topic: https://lists.openembedded.org/mt/75836420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [meta-oe][PATCH 2/3] base.bbclass: make oe.log_colorizer available in global context
Signed-off-by: Chris Laplante
---
meta/classes/base.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 4c681cc870..5a969b207a 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -12,7 +12,7 @@ inherit logging
OE_EXTRA_IMPORTS ?= ""
-OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license
${OE_EXTRA_IMPORTS}"
+OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.log_colorizer
${OE_EXTRA_IMPORTS}"
OE_IMPORTS[type] = "list"
PACKAGECONFIG_CONFARGS ??= ""
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#141020):
https://lists.openembedded.org/g/openembedded-core/message/141020
Mute This Topic: https://lists.openembedded.org/mt/75836418/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [meta-oe][PATCH 1/3] lib/oe/log_colorizer.py: add LogColorizerProxyProgressHandler
This progress handler intercepts log output, stripping any ANSII color
escape codes. Then the stripped output is fed to the underlying progress
handler which will render the progress bar as usual.
Signed-off-by: Chris Laplante
---
meta/lib/oe/log_colorizer.py | 86
1 file changed, 86 insertions(+)
create mode 100644 meta/lib/oe/log_colorizer.py
diff --git a/meta/lib/oe/log_colorizer.py b/meta/lib/oe/log_colorizer.py
new file mode 100644
index 00..10021ef880
--- /dev/null
+++ b/meta/lib/oe/log_colorizer.py
@@ -0,0 +1,86 @@
+# Copyright (C) 2020 Agilent Technologies, Inc.
+# Author: Chris Laplante
+#
+# Released under the MIT license (see COPYING.MIT)
+
+from bb.progress import ProgressHandler
+from bb.build import create_progress_handler
+import contextlib
+import re
+import os.path
+
+# from https://stackoverflow.com/a/14693789/221061
+ansi_escape = re.compile(r'\x1B\[[0-?]*[ -/]*[@-~]')
+
+
+class LogColorizerProxyProgressHandler(ProgressHandler):
+"""
+This is a proxy progress handler. It intercepts log output, stripping any
+ANSII color escape codes. Then the stripped output is fed to the task's
+original progress handler which will render the progress bar as usual.
+"""
+def __init__(self, d, outfile=None, otherargs=None):
+self._task = d.getVar("BB_RUNTASK")
+self._color_log = None
+self._nocolor_log = None
+self._exit_stack = None
+self._original_ph = None
+
+self._suppress_color_output = not not
d.getVar("LOG_COLORIZER_SUPPRESS_COLORIZED_OUTPUT")
+
+self._tempdir = d.getVar("T")
+logfmt = (d.getVar('BB_LOGFMT') or 'log.{task}.{pid}')
+self._logbasename = logfmt.format(task=self._task, pid=os.getpid())
+
+# Setup courtesy symlinks
+for suffix in [".nocolor", ".color"]:
+loglink = os.path.join(self._tempdir,
'log.{0}{1}'.format(self._task, suffix))
+logfn = os.path.join(self._tempdir, self._logbasename + suffix)
+if loglink:
+bb.utils.remove(loglink)
+
+try:
+os.symlink(self._logbasename + suffix, loglink)
+except OSError:
+pass
+
+super().__init__(d, outfile)
+
+def __enter__(self):
+with contextlib.ExitStack() as es:
+self._color_log =
es.enter_context(open(os.path.join(self._tempdir, self._logbasename) +
".color", "w"))
+self._nocolor_log =
es.enter_context(open(os.path.join(self._tempdir, self._logbasename) +
".nocolor", "w"))
+
+# Reconstitute the original progress handler. We will feed
stripped output to it so
+# that the progress bar still shows up for the task.
+original_ph = self._data.getVarFlag(self._task, "originalprogress")
+if original_ph:
+# We don't want task output showing up on the screen twice, so
we'll just have
+# the original progress handler write to /dev/null.
+# Note the progress handler itself is responsible for closing
the devnull handler.
+devnull = open("/dev/null", "w")
+self._original_ph =
es.enter_context(create_progress_handler(self._task, original_ph, devnull,
self._data))
+
+self._exit_stack = es.pop_all()
+super().__enter__()
+
+def __exit__(self, *exc_info):
+if self._exit_stack:
+self._exit_stack.__exit__(*exc_info)
+super().__exit__(*exc_info)
+
+def write(self, string):
+# Filter out ANSI escape sequences using the regular expression
+filtered = ansi_escape.sub('', string)
+
+if self._color_log:
+self._color_log.write(string)
+
+if self._nocolor_log:
+self._nocolor_log.write(filtered)
+
+if self._original_ph:
+# Pass-through to the original progress handler so we get our
progress bar
+self._original_ph.write(filtered)
+
+super().write(filtered if self._suppress_color_output else string)
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#141021):
https://lists.openembedded.org/g/openembedded-core/message/141021
Mute This Topic: https://lists.openembedded.org/mt/75836419/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [meta-oe][PATCH 0/3] Log colorizer
This patch series turns on color compiler diagnostics. It is especially useful when doing edit-compile-test development work with devtool. It is based on one I've been using internally for about a year now. Limitation: Note that the *.nocolor and *.color logs will only contain the output of the task itself, not any prefuncs or postfuncs. This is because the color filtering is implemented using progress handlers, which don't see prefunc or postfunc output. Chris Laplante (3): lib/oe/log_colorizer.py: add LogColorizerProxyProgressHandler base.bbclass: make oe.log_colorizer available in global context log-colorizer.bbclass: add new class meta/classes/base.bbclass | 2 +- meta/classes/log-colorizer.bbclass | 49 + meta/lib/oe/log_colorizer.py | 86 ++ 3 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 meta/classes/log-colorizer.bbclass create mode 100644 meta/lib/oe/log_colorizer.py -- 2.17.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141019): https://lists.openembedded.org/g/openembedded-core/message/141019 Mute This Topic: https://lists.openembedded.org/mt/75836416/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Yocto Project Status WW25'20
Hi David, > > "Does anyone know what the intended audience is for the Developer Day > presentations?" > > The official description for the intended audience can be found here: > https://www.yoctoproject.org/yocto-project-dev-day-virtual-north-america- > 2020/ > > We will be covering a lot of topics that can give you ideas on how to use > OE/YP > in perhaps new ways, for example in containers, virtualization, micro > services, > and CI/CD frameworks. > > There will be practical instruction in such topics as image size reduction. > > There will also be hands-on exercises in the kernel development tools and user > space management. > > There will be many ways to expand your horizons with Yocto Project at > DevDay, to meet and hear the community experts behind all the magic, and > even bring them your questions directly. > > All of this plus being with people passionate about OE/YP and having fun at > the > same time! A bargain at any price! > > - David Excellent, thank you! I was going to attend anyway :). Was just trying to figure out if others on my team might also benefit, and it sounds like they will. Thanks again, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139887): https://lists.openembedded.org/g/openembedded-core/message/139887 Mute This Topic: https://lists.openembedded.org/mt/75061182/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Yocto Project Status WW25'20
> > Does anyone know what the intended audience is for the Developer Day > presentations? E.g. would someone who uses bitbake day-to-day but perhaps > doesn't edit recipes benefit? > > > > I would say look at the schedule and see if there are talks of interest to > you. I did, but the schedule says nothing about the level of the topics. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139850): https://lists.openembedded.org/g/openembedded-core/message/139850 Mute This Topic: https://lists.openembedded.org/mt/75061182/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Yocto Project Status WW25'20
> * Next week is ELC which is taking place virtually this year and Yocto Project > has a presence there. There is also a virtual Yocto Project Developer Day: > https://events.linuxfoundation.org/yocto-dev-day-north-america/ Does anyone know what the intended audience is for the Developer Day presentations? E.g. would someone who uses bitbake day-to-day but perhaps doesn't edit recipes benefit? Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139841): https://lists.openembedded.org/g/openembedded-core/message/139841 Mute This Topic: https://lists.openembedded.org/mt/75061182/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Changing DISTRO_FEATURES / build directory
> "contains" has special magic where it caches the specific value presence > rather than the whole string. I'd therefore recommend using the contains > functions where at all possible for that reason. So then regarding my other points, it is correctly handled by the signature code? If so, why doesn’t it show up in task dependencies? Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137449): https://lists.openembedded.org/g/openembedded-core/message/137449 Mute This Topic: https://lists.openembedded.org/mt/73225609/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Changing DISTRO_FEATURES / build directory
> > So, is changing DISTRO_FEATURES but not changing the build directory
> > expected to work?
>
> Should work but has bugs.
>
> There were some fixed around the thud/zeus timeframe.
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13702
>
> is a recently reported one that has patches in master-next (both to fix and
> add a test case).
>
> Is there a failure with reproducer you have? Which release?
I haven't encountered any bugs yet, no. I am working on adding some custom
DISTRO_FEATURES to our distro. I was trying to convince myself whether vardeps
are necessary for DISTRO_FEATURES, and couldn't decide. For instance, if I have:
do_unpack_append() {
echo "${@bb.utils.contains("DISTRO_FEATURES", "my-feature", "true",
"false", d)}" >> ${S}/whatever
}
bitbake-dumpsigs on the do_unpack sigdata reveals that DISTRO_FEATURES is *not*
part of the task dependencies. So I was wondering if I needed to do:
do_unpack[vardeps] += "DISTRO_FEATURES"
but looking around poky and meta-openembedded, I don't see anyone else doing
that kind of thing.
The interesting thing is that the bitbake-dumpsig does have some special
looking DISTRO_FEATURES output, such as:
DISTRO_FEATURES{usrmerge} = Unset
DISTRO_FEATURES{my-feature} = Set
So it seems there is some DISTRO_FEATURES specific processing?
It's also unclear to me whether the signature code is able to understand
bb.utils.contains and automatically pluck out the fact that DISTRO_FEATURES
should be accounted for.
Thanks,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#137447):
https://lists.openembedded.org/g/openembedded-core/message/137447
Mute This Topic: https://lists.openembedded.org/mt/73225609/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Changing DISTRO_FEATURES / build directory
There is a familiar message in sstate.bbclass that explains: "... It may be you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning those recipes should again resolve this error, however switching DISTRO_FEATURES on an existing build directory is not supported - you should really clean out tmp and rebuild (reusing sstate should be safe)..." But the release notes for the 2.0 release suggest that it should work: https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#migration-2.0-automatic-stale-sysroot-file-cleanup. So, is changing DISTRO_FEATURES but not changing the build directory expected to work? Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137433): https://lists.openembedded.org/g/openembedded-core/message/137433 Mute This Topic: https://lists.openembedded.org/mt/73225609/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Uninative for i386 native tool?
> There is a non-hypothetical i386 loader, it already exists. > Unfortunately the loader has to be downloaded and extracted in place > before anything gets run so it all happens very early. Its done centrally with > one uninative, not two or per recipe. > > I think you might be able to do something like: > > SSTATEPOSTUNPACKFUNCS_remove_pn-mytool = "uninative_changeinterp" > > and it may at least stop breaking it... What if we taught uninative how to identify whether it is appropriate to change the loader for the executable? If not, it could (maybe configurably) avoid doing so and/or give a warning. Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137328): https://lists.openembedded.org/g/openembedded-core/message/137328 Mute This Topic: https://lists.openembedded.org/mt/73137039/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Uninative for i386 native tool?
> > I suspect uninative assumes all binaries are for the main system > > architecture so its probably trying to change the interpreter to a 64 > > bit one which clearly won't work in this case. > > > > It would be a question of stopping it for these binaries somehow... > > Maybe a matter of doing something like UNINATIVE_LOADER_pn-my-tool = > "???" > > Where "???" would be the loader in a hypothetical i386 uninative. Or I suppose the existing i686 uninative. So the trick would be to get uninative.bbclass to fetch both. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137325): https://lists.openembedded.org/g/openembedded-core/message/137325 Mute This Topic: https://lists.openembedded.org/mt/73137039/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Uninative for i386 native tool?
> I suspect uninative assumes all binaries are for the main system architecture > so its probably trying to change the interpreter to a 64 bit one which clearly > won't work in this case. > > It would be a question of stopping it for these binaries somehow... Maybe a matter of doing something like UNINATIVE_LOADER_pn-my-tool = "???" Where "???" would be the loader in a hypothetical i386 uninative. Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137324): https://lists.openembedded.org/g/openembedded-core/message/137324 Mute This Topic: https://lists.openembedded.org/mt/73137039/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Uninative for i386 native tool?
One possible work around which I'm using for some prebuilt native binaries is
to call them with hosts loader (e.g. /lib32/ld-linux.so.2
${STAGING_BINDIR_NATIVE}/my-tool params)
Ooh, nice. Thanks for the tip! Definitely curious about a longer-term
workaround but this is good for the short term.
Thanks again,
Chris
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#137317):
https://lists.openembedded.org/g/openembedded-core/message/137317
Mute This Topic: https://lists.openembedded.org/mt/73137039/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Uninative for i386 native tool?
We have a vendor-provided cross-compiler toolchain that is distributed as 32-bit binaries. Naturally on my native system (outside of Yocto) I can run them by first adding the i386 architecture to dpkg, and then doing "apt-get install -y libc6-dbg:i386 lib32stdc++6 libzstd1:i386 libncurses5:i386" or similar. Outside of Yocto (a normal terminal): ldd recipe-sysroot-native/usr/bin/my-tool: linux-gate.so.1 => (0xf7f03000) libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xf7d48000) libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf7cf3000) libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf7cd6000) libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7b2) /home/laplante/yocto/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 => /lib/ld-linux.so.2 (0xf7f05000) But in a Yocto devshell: ldd recipe-sysroot-native/usr/bin/my-tool: not a dynamic executable Is there a way to get this working in Yocto? I assume it has something to do with uninative? Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137315): https://lists.openembedded.org/g/openembedded-core/message/137315 Mute This Topic: https://lists.openembedded.org/mt/73137039/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] why does oe-buildenv-internal check for -z "$BDIR"?
> i know that -- in fact, it's the only way i use it -- the question > is the purpose for checking if $BDIR is zero length first. that is, > under what circumstances could it *not* be? even invoking the way you > demonstrate, that variable is empty when tested, and i see nothing > that could cause it to have a value before that test. FWIW I was curious/bored and looked to see if 'BDIR' was ever used in poky (outside of the script of course), and I found no results. git rev-list --all | xargs git grep -w BDIR -- './*' ':!scripts/oe-buildenv-internal' Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137285): https://lists.openembedded.org/g/openembedded-core/message/137285 Mute This Topic: https://lists.openembedded.org/mt/73109744/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
