Re: [oe-core][PATCH 3/6] meta/lib/oe/spdx30_tasks.py: improve debug log in add_package_files
LGTM Reviewed-by: Joshua Watt On Tue, Oct 29, 2024 at 11:07 PM Hongxu Jia wrote: > > While SPDX_INCLUDE_SOURCES = "1" [1], there are mess of `Adding file' > in debug log > ''' > DEBUG: Adding file > tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/sources-unpack/COPYING > to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native > DEBUG: Adding file > tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/gettext-0.22.5/.tarball-version > to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native > ''' > > Summary the total number other than print for each file. > ''' > DEBUG: Added 7201 files to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/f5e0e04913ac4c595be791fc001d545a77519ed6ee8c743deef721ca0898bc94/document/gettext-minimal-native > ''' > > [1] > https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES > > Signed-off-by: Hongxu Jia > --- > meta/lib/oe/spdx30_tasks.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py > index e0b656d81f..5aeed5cd6f 100644 > --- a/meta/lib/oe/spdx30_tasks.py > +++ b/meta/lib/oe/spdx30_tasks.py > @@ -155,8 +155,6 @@ def add_package_files( > if filepath.is_symlink() or not filepath.is_file(): > continue > > -bb.debug(1, "Adding file %s to %s" % (filepath, objset.doc._id)) > - > filename = str(filepath.relative_to(topdir)) > file_purposes = get_purposes(filepath) > > @@ -187,6 +185,8 @@ def add_package_files( > > file_counter += 1 > > +bb.debug(1, "Added %d files to %s" % (len(spdx_files), objset.doc._id)) > + > return spdx_files > > > -- > 2.25.1 > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#206591): https://lists.openembedded.org/g/openembedded-core/message/206591 Mute This Topic: https://lists.openembedded.org/mt/109291177/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 1/6] meta/lib/oe/sbom30.py: correct python list comprehension
LGTM
Reviewed-by: Joshua Watt
On Tue, Oct 29, 2024 at 11:07 PM Hongxu Jia wrote:
>
> The python list comprehension is not right for list:
> >>> license_text_map = {'LicenseRef-FSF-Unlimited':
> >>> 'http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/8d31e22acc4a8979f24dc24042692fb548fc8fc8d85d775ddac406abb122ceea/license-text/FSF-Unlimited'}
> >>> license_text = ((k, license_text_map[k]) for k in
> >>> sorted(license_text_map.keys()))
> >>> print(license_text)
> at 0x7f8575173270>
> >>> [(k,v) for k, v in license_text]
> []
>
> Change the () to [] to make it a list instead of a generator expression.
>
> Signed-off-by: Hongxu Jia
> ---
> meta/lib/oe/sbom30.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
> index 27ab5e45ac..8db90f30fd 100644
> --- a/meta/lib/oe/sbom30.py
> +++ b/meta/lib/oe/sbom30.py
> @@ -577,9 +577,9 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
> re.sub(r"[^a-zA-Z0-9_-]", "_", license_expression),
> ]
>
> -license_text = (
> +license_text = [
> (k, license_text_map[k]) for k in sorted(license_text_map.keys())
> -)
> +]
>
> if not license_text:
> lic = self.find_filter(
> --
> 2.25.1
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206589):
https://lists.openembedded.org/g/openembedded-core/message/206589
Mute This Topic: https://lists.openembedded.org/mt/109291174/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 6/6] oeqa/selftest: Add SPDX 3.0 include source case for work-share
Excellent! Thanks for adding a test case
Reviewed-by: Joshua Watt
On Tue, Oct 29, 2024 at 11:07 PM Hongxu Jia wrote:
>
> Build gcc and check gcc-14.2.0/README in objset is available
>
> $ oe-selftest -r spdx.SPDX30Check.test_gcc_include_source
> ...
> 2024-10-26 01:24:57,063 - oe-selftest - INFO - test_gcc_include_source
> (spdx.SPDX30Check.test_gcc_include_source)
> 2024-10-26 01:28:24,204 - oe-selftest - INFO - The spdxId of
> gcc-14.2.0/README in gcc.spdx.json is
> http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/88d5068ffd41e5ea6b4e0dd390b23bf499bb2b6674a41e09eaf2a887eced16c8/sourcefile/42
> 2024-10-26 01:28:26,369 - oe-selftest - INFO - ... ok
> 2024-10-26 01:28:33,315 - oe-selftest - INFO -
> --
> 2024-10-26 01:28:33,316 - oe-selftest - INFO - Ran 1 test in 216.457s
> 2024-10-26 01:28:33,316 - oe-selftest - INFO - OK
> 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS:
> 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS -
> spdx.SPDX30Check.test_gcc_include_source: PASSED (209.31s)
> 2024-10-26 01:28:45,260 - oe-selftest - INFO - SUMMARY:
> 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest () - Ran 1 test in
> 216.457s
> 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest - OK - All
> required tests passed (successes=1, skipped=0, failures=0, errors=0)
>
> Signed-off-by: Hongxu Jia
> ---
> meta/lib/oeqa/selftest/cases/spdx.py | 29
> 1 file changed, 29 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/cases/spdx.py
> b/meta/lib/oeqa/selftest/cases/spdx.py
> index be595babb3..8384070219 100644
> --- a/meta/lib/oeqa/selftest/cases/spdx.py
> +++ b/meta/lib/oeqa/selftest/cases/spdx.py
> @@ -110,6 +110,7 @@ class SPDX3CheckBase(object):
> "SDKMACHINE",
> "SDK_DEPLOY",
> "SPDX_VERSION",
> +"SSTATE_PKGARCH",
> "TOOLCHAIN_OUTPUTNAME",
> ],
> target_name,
> @@ -136,6 +137,34 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
> "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/base-files.spdx.json",
> )
>
> +
> +def test_gcc_include_source(self):
> +import oe.spdx30
> +
> +objset = self.check_recipe_spdx(
> +"gcc",
> +"{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/gcc.spdx.json",
> +extraconf=textwrap.dedent(
> +"""\
> +SPDX_INCLUDE_SOURCES = "1"
> +"""
> +),
> +)
> +
> +gcc_pv = get_bb_var("PV", "gcc")
> +filename = f'gcc-{gcc_pv}/README'
> +found = False
> +for software_file in objset.foreach_type(oe.spdx30.software_File):
> +if software_file.name == filename:
> +found = True
> +self.logger.info(f"The spdxId of {filename} in gcc.spdx.json
> is {software_file.spdxId}")
> +break
> +
> +self.assertTrue(
> +found,
> +f"Not found source file {filename} in gcc.spdx.json\n"
> +)
> +
> def test_core_image_minimal(self):
> objset = self.check_recipe_spdx(
> "core-image-minimal",
> --
> 2.25.1
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206594):
https://lists.openembedded.org/g/openembedded-core/message/206594
Mute This Topic: https://lists.openembedded.org/mt/109291181/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH V3 5/6] create-spdx-{2.2,3.0}: support SPDX include source for work-share directory
LGTM
Reviewed-by: Joshua Watt
On Tue, Oct 29, 2024 at 11:07 PM Hongxu Jia wrote:
>
> Originally, while SPDX_INCLUDE_SOURCES = "1" [1], there is bug in scan
> for gcc, libgcc in which the sources locates in work-share directory.
> Copy source from ${WORKDIR} to ${SPDXWORK} did not satisfy the situation
> while ${S} was not included in ${WORKDIR}
>
> This commit aim to support SPDX include source for work-share directory
>
> 1. If is_work_shared_spdx, Copy source from ${S} to ${SPDXWORK},
> normally the dest dir in ${SPDXWORK} has the same basename dir of ${S};
> but for kernel source, rename basename dir 'kernel-source' to ${BP}
> (${BPN}-${PV})
>
> 2. For SPDX source copy, do hard link copy to save copy time
>
> 3. Move do_patch to no work shared situation along with do_unpack
>
> 4. Tweak task do_create_spdx dependencies to assure the patched source
> in work share is ready for SPDX source copy
>
> 5. Remove bb.data.inherits_class('kernel', d) from is_work_shared_spdx,
> the kernel source locates in 'work-shared', test kernel.bbclass is not
> necessary
>
> [1]
> https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES
>
> Signed-off-by: Hongxu Jia
> ---
> meta/classes/spdx-common.bbclass | 9 +++
> meta/lib/oe/spdx_common.py | 41
> 2 files changed, 24 insertions(+), 26 deletions(-)
>
> diff --git a/meta/classes/spdx-common.bbclass
> b/meta/classes/spdx-common.bbclass
> index ad02da5cd6..d3cdc8b6ce 100644
> --- a/meta/classes/spdx-common.bbclass
> +++ b/meta/classes/spdx-common.bbclass
> @@ -57,6 +57,15 @@ def create_spdx_source_deps(d):
> if oe.spdx_common.has_task(d, "do_shared_workdir"):
> deps.append("%s:do_shared_workdir" % pn)
>
> +# For gcc-source-${PV} source code
> +if oe.spdx_common.has_task(d, "do_preconfigure"):
> +deps.append("%s:do_preconfigure" % pn)
> +elif oe.spdx_common.has_task(d, "do_patch"):
> +deps.append("%s:do_patch" % pn)
> +# For gcc-cross-x86_64 source code
> +elif oe.spdx_common.has_task(d, "do_configure"):
> +deps.append("%s:do_configure" % pn)
> +
> return " ".join(deps)
>
>
> diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
> index 7a85579f64..ff18d91780 100644
> --- a/meta/lib/oe/spdx_common.py
> +++ b/meta/lib/oe/spdx_common.py
> @@ -38,7 +38,7 @@ def extract_licenses(filename):
>
>
> def is_work_shared_spdx(d):
> -return bb.data.inherits_class("kernel", d) or ("work-shared" in
> d.getVar("WORKDIR"))
> +return '/work-shared/' in d.getVar('S')
>
>
> def load_spdx_license_data(d):
> @@ -74,11 +74,6 @@ def process_sources(d):
> if d.getVar("PN") == "shadow-sysroot":
> return False
>
> -# We just archive gcc-source for all the gcc related recipes
> -if d.getVar("BPN") in ["gcc", "libgcc"]:
> -bb.debug(1, "spdx: There is bug in scan of %s is, do nothing" % pn)
> -return False
> -
> return True
>
>
> @@ -190,34 +185,28 @@ def get_patched_src(d):
> bb.utils.mkdirhier(d.getVar("B"))
>
> bb.build.exec_func("do_unpack", d)
> -# Copy source of kernel to spdx_workdir
> +
> +if d.getVar("SRC_URI") != "":
> +bb.build.exec_func("do_patch", d)
> +
> +# Copy source from work-share to spdx_workdir
> if is_work_shared_spdx(d):
> -share_src = d.getVar("WORKDIR")
> +share_src = d.getVar('S')
> d.setVar("WORKDIR", spdx_workdir)
> d.setVar("STAGING_DIR_NATIVE", spdx_sysroot_native)
> +# Copy source to ${SPDXWORK}, same basename dir of ${S};
> src_dir = (
> spdx_workdir
> + "/"
> -+ d.getVar("PN")
> -+ "-"
> -+ d.getVar("PV")
> -+ "-"
> -+ d.getVar("PR")
> ++ os.path.basename(share_src)
> )
> -bb.utils.mkdirhier(src_dir)
> +# For kernel souce, rename suffix dir 'kernel-source'
> +# to ${BP} (${BPN}-${PV})
> if bb.data.inherits_class("k
Re: [oe-core][PATCH V3 4/6] create-spdx-{2.2,3.0}: fix do_create_spdx dependency while spdx include sources
LGTM
Reviewed-by: Joshua Watt
On Tue, Oct 29, 2024 at 11:07 PM Hongxu Jia wrote:
>
> Call function ${@create_spdx_source_deps(d)} or ${create_spdx_source_deps(d)}
> along with addtask not working, use task do_create_spdx flag 'depends'
> to instead
>
> Move function create_spdx_source_deps to spdx-common.bbclass for both of
> create-spdx-2.2.bbclass and create-spdx-3.0.bbclass
>
> Signed-off-by: Hongxu Jia
> ---
> meta/classes/create-spdx-2.2.bbclass | 5 -
> meta/classes/create-spdx-3.0.bbclass | 19 ---
> meta/classes/spdx-common.bbclass | 21 +
> meta/lib/oe/spdx_common.py | 4
> 4 files changed, 33 insertions(+), 16 deletions(-)
>
> diff --git a/meta/classes/create-spdx-2.2.bbclass
> b/meta/classes/create-spdx-2.2.bbclass
> index cd1d6819bf..27242ecf70 100644
> --- a/meta/classes/create-spdx-2.2.bbclass
> +++ b/meta/classes/create-spdx-2.2.bbclass
> @@ -584,7 +584,10 @@ addtask do_create_spdx_setscene
>
> do_create_spdx[dirs] = "${SPDXWORK}"
> do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
> -do_create_spdx[depends] += "${PATCHDEPENDENCY}"
> +do_create_spdx[depends] += " \
> +${PATCHDEPENDENCY} \
> +${@create_spdx_source_deps(d)} \
> +"
>
> python do_create_runtime_spdx() {
> from datetime import datetime, timezone
> diff --git a/meta/classes/create-spdx-3.0.bbclass
> b/meta/classes/create-spdx-3.0.bbclass
> index 5f0590198f..bc23d2d211 100644
> --- a/meta/classes/create-spdx-3.0.bbclass
> +++ b/meta/classes/create-spdx-3.0.bbclass
> @@ -132,22 +132,8 @@ addtask do_create_spdx after \
> do_collect_spdx_deps \
> do_deploy_source_date_epoch \
> do_populate_sysroot do_package do_packagedata \
> -${create_spdx_source_deps(d)} \
> before do_populate_sdk do_populate_sdk_ext do_build do_rm_work
>
> -def create_spdx_source_deps(d):
> -deps = []
> -if d.getVar("SPDX_INCLUDE_SOURCES") == "1":
> -deps.extend([
> -# do_unpack is a hack for now; we only need it to get the
> -# dependencies do_unpack already has so we can extract the source
> -# ourselves
> -"do_unpack",
> -# For kernel source code
> -"do_shared_workdir",
> -])
> -return " ".join(deps)
> -
> SSTATETASKS += "do_create_spdx"
> do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
> do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
> @@ -159,7 +145,10 @@ addtask do_create_spdx_setscene
>
> do_create_spdx[dirs] = "${SPDXWORK}"
> do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
> -do_create_spdx[depends] += "${PATCHDEPENDENCY}"
> +do_create_spdx[depends] += " \
> +${PATCHDEPENDENCY} \
> +${@create_spdx_source_deps(d)} \
> +"
>
> python do_create_package_spdx() {
> import oe.spdx30_tasks
> diff --git a/meta/classes/spdx-common.bbclass
> b/meta/classes/spdx-common.bbclass
> index cd9cc0db98..ad02da5cd6 100644
> --- a/meta/classes/spdx-common.bbclass
> +++ b/meta/classes/spdx-common.bbclass
> @@ -39,6 +39,27 @@ SPDX_CUSTOM_ANNOTATION_VARS ??= ""
>
> SPDX_MULTILIB_SSTATE_ARCHS ??= "${SSTATE_ARCHS}"
>
> +def create_spdx_source_deps(d):
> +import oe.spdx_common
> +
> +deps = []
> +if d.getVar("SPDX_INCLUDE_SOURCES") == "1":
> +pn = d.getVar('PN')
> +# do_unpack is a hack for now; we only need it to get the
> +# dependencies do_unpack already has so we can extract the source
> +# ourselves
> +if oe.spdx_common.has_task(d, "do_unpack"):
> +deps.append("%s:do_unpack" % pn)
> +
> +if oe.spdx_common.is_work_shared_spdx(d) and \
> + oe.spdx_common.process_sources(d):
> +# For kernel source code
> +if oe.spdx_common.has_task(d, "do_shared_workdir"):
> +deps.append("%s:do_shared_workdir" % pn)
> +
> +return " ".join(deps)
> +
> +
> python do_collect_spdx_deps() {
> # This task calculates the build time dependencies of the recipe, and is
> # required because while a task can deptask on itself, those dependencies
> diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
> index 1ea55419ae..7a85579f64 100644
> --- a/meta/lib/oe/spdx_common.py
> +++ b/meta/lib/oe/spdx_common.py
> @@ -226,6 +226,10 @@ def get_patched_src(d):
> d.setVar("WORKDIR", workdir)
>
>
Re: [oe-core][PATCH 2/6] meta/lib/oe/sbom30.py: correct typo
LGTM, thanks Reviewed-by: Joshua Watt On Tue, Oct 29, 2024 at 11:07 PM Hongxu Jia wrote: > > The isinstance expected 2 arguments > > Signed-off-by: Hongxu Jia > --- > meta/lib/oe/sbom30.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py > index 8db90f30fd..e3a9428668 100644 > --- a/meta/lib/oe/sbom30.py > +++ b/meta/lib/oe/sbom30.py > @@ -631,7 +631,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): > sha256_hash = bb.utils.sha256_file(path) > > for f in self.by_sha256_hash.get(sha256_hash, []): > -if not isinstance(oe.spdx30.software_File): > +if not isinstance(f, oe.spdx30.software_File): > continue > > if purposes: > -- > 2.25.1 > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#206590): https://lists.openembedded.org/g/openembedded-core/message/206590 Mute This Topic: https://lists.openembedded.org/mt/109291175/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH V2 1/7] meta/lib/oe/sbom30.py: correct python list comprehension
LGTM, Thanks
Reviewed-by: Joshua Watt
On Tue, Oct 29, 2024 at 3:17 AM Hongxu Jia wrote:
>
> The python list comprehension is not right for list:
> >>> license_text_map = {'LicenseRef-FSF-Unlimited':
> >>> 'http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/8d31e22acc4a8979f24dc24042692fb548fc8fc8d85d775ddac406abb122ceea/license-text/FSF-Unlimited'}
> >>> license_text = ((k, license_text_map[k]) for k in
> >>> sorted(license_text_map.keys()))
> >>> print(license_text)
> at 0x7f8575173270>
> >>> [(k,v) for k, v in license_text]
> []
>
> Change the () to [] to make it a list instead of a generator expression.
>
> Signed-off-by: Hongxu Jia
> ---
> meta/lib/oe/sbom30.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
> index 27ab5e45ac..8db90f30fd 100644
> --- a/meta/lib/oe/sbom30.py
> +++ b/meta/lib/oe/sbom30.py
> @@ -577,9 +577,9 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
> re.sub(r"[^a-zA-Z0-9_-]", "_", license_expression),
> ]
>
> -license_text = (
> +license_text = [
> (k, license_text_map[k]) for k in sorted(license_text_map.keys())
> -)
> +]
>
> if not license_text:
> lic = self.find_filter(
> --
> 2.25.1
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206503):
https://lists.openembedded.org/g/openembedded-core/message/206503
Mute This Topic: https://lists.openembedded.org/mt/109273901/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH V2 4/7] meta/lib/oe/spdx30_tasks.py: improve debug log in add_package_files
LGTM, thanks Reviewed-by: Joshua Watt On Tue, Oct 29, 2024 at 3:17 AM Hongxu Jia wrote: > > While SPDX_INCLUDE_SOURCES = "1" [1], there are mess of `Adding file' > in debug log > ''' > DEBUG: Adding file > tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/sources-unpack/COPYING > to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native > DEBUG: Adding file > tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/gettext-0.22.5/.tarball-version > to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native > ''' > > Summary the total number other than print for each file. > ''' > DEBUG: Added 7201 files to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/f5e0e04913ac4c595be791fc001d545a77519ed6ee8c743deef721ca0898bc94/document/gettext-minimal-native > ''' > > [1] > https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES > > Signed-off-by: Hongxu Jia > --- > meta/lib/oe/spdx30_tasks.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py > index e0b656d81f..5aeed5cd6f 100644 > --- a/meta/lib/oe/spdx30_tasks.py > +++ b/meta/lib/oe/spdx30_tasks.py > @@ -155,8 +155,6 @@ def add_package_files( > if filepath.is_symlink() or not filepath.is_file(): > continue > > -bb.debug(1, "Adding file %s to %s" % (filepath, objset.doc._id)) > - > filename = str(filepath.relative_to(topdir)) > file_purposes = get_purposes(filepath) > > @@ -187,6 +185,8 @@ def add_package_files( > > file_counter += 1 > > +bb.debug(1, "Added %d files to %s" % (len(spdx_files), objset.doc._id)) > + > return spdx_files > > > -- > 2.25.1 > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#206505): https://lists.openembedded.org/g/openembedded-core/message/206505 Mute This Topic: https://lists.openembedded.org/mt/109273906/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 2/7] meta/lib/oe/sbom30.py: correct typo
LGTM, thanks Reviewed-by: Joshua Watt On Tue, Oct 29, 2024 at 3:17 AM Hongxu Jia wrote: > > The isinstance expected 2 arguments > > Signed-off-by: Hongxu Jia > --- > meta/lib/oe/sbom30.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py > index 8db90f30fd..e3a9428668 100644 > --- a/meta/lib/oe/sbom30.py > +++ b/meta/lib/oe/sbom30.py > @@ -631,7 +631,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): > sha256_hash = bb.utils.sha256_file(path) > > for f in self.by_sha256_hash.get(sha256_hash, []): > -if not isinstance(oe.spdx30.software_File): > +if not isinstance(f, oe.spdx30.software_File): > continue > > if purposes: > -- > 2.25.1 > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#206504): https://lists.openembedded.org/g/openembedded-core/message/206504 Mute This Topic: https://lists.openembedded.org/mt/109273903/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH V2 3/7] meta/lib/oe/sbom30.py: create hasDeclaredLicense relationship if file_licenses is empty
On Tue, Oct 29, 2024 at 3:17 AM Hongxu Jia wrote:
>
> If file_licenses is empty, the hasDeclaredLicense relationship is from
> sourcefile
> to NoneElement. Such as
>
> {
> "type": "Relationship",
> ...
> "from":
> "http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb/sourcefile/3323";,
> "relationshipType": "hasDeclaredLicense",
> "to": [
> "NoneElement"
> ]
> },
>
> According to Specification Version 3.0.1
>
> NoneElement should be used if [1]
>
> the SPDX creator desires to assert that there are NO elements for the
> given context of use.
>
> NoAssertionElement should be used if [2]
>
> the SPDX creator has attempted to but cannot reach a reasonable objective
> determination;
> the SPDX creator has made no attempt to determine this field; or
> the SPDX creator has intentionally provided no information (no meaning
> should be implied by doing so).
>
> If we indicates to look for licenses and didn't find any. It should be
> NoAssertionElement other than NoneElement
>
> [1]
> https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Individuals/NoneElement/
> [2]
> https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Individuals/NoAssertionElement/
>
> Signed-off-by: Hongxu Jia
> ---
> meta/lib/oe/sbom30.py | 5 +
> 1 file changed, 5 insertions(+)
>
> diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
> index e3a9428668..7ae05c42a9 100644
> --- a/meta/lib/oe/sbom30.py
> +++ b/meta/lib/oe/sbom30.py
> @@ -620,6 +620,11 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
> for extracted_lic in oe.spdx_common.extract_licenses(filepath):
> file_licenses.add(self.new_license_expression(extracted_lic,
> license_data))
>
> +# SPDX creator has attempted to but cannot reach a reasonable
> objective determination
> +#
> https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Individuals/NoAssertionElement/
> +if not file_licenses:
> +file_licenses = [oe.spdx30.Element.NoAssertionElement]
This still needs to be NoneElement, because we looked and did not find
any licenses, therefore in our estimation, there are not any to be
found. This is the meaning intended by the spec; we (the SPDX creator)
are explicitly stating that there are NO licenses (elements) defined
in the file (given context).
If we didn't scan the file at all, that means that we were unable (or
made no attempt) to determine if the file had licenses or not; and
thus NoAssertion would be the correct choice since we would want to
explicitly state we chose not to look for any licenses. This is not
what we actually do in practice, except for in the case where
extract_licenses() gets an exception; if you wanted to, you could have
the function return None in that case and then use that to add a
NoAssertion to the relationship, but the empty list must be treated as
NoneElement
> +
> self.new_relationship(
> [spdx_file],
> oe.spdx30.RelationshipType.hasDeclaredLicense,
> --
> 2.25.1
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206501):
https://lists.openembedded.org/g/openembedded-core/message/206501
Mute This Topic: https://lists.openembedded.org/mt/109273905/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 6/6] Add SPDX 3.0 include source case for work-share
LGTM, thanks.
Reviewed-by: Joshua Watt
On Fri, Oct 25, 2024 at 9:25 PM Hongxu Jia wrote:
>
> Build gcc and check gcc-14.2.0/README in objset is available
>
> $ oe-selftest -r spdx.SPDX30Check.test_gcc_include_source
> ...
> 2024-10-26 01:24:57,063 - oe-selftest - INFO - test_gcc_include_source
> (spdx.SPDX30Check.test_gcc_include_source)
> 2024-10-26 01:28:24,204 - oe-selftest - INFO - The spdxId of
> gcc-14.2.0/README in gcc.spdx.json is
> http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/88d5068ffd41e5ea6b4e0dd390b23bf499bb2b6674a41e09eaf2a887eced16c8/sourcefile/42
> 2024-10-26 01:28:26,369 - oe-selftest - INFO - ... ok
> 2024-10-26 01:28:33,315 - oe-selftest - INFO -
> --
> 2024-10-26 01:28:33,316 - oe-selftest - INFO - Ran 1 test in 216.457s
> 2024-10-26 01:28:33,316 - oe-selftest - INFO - OK
> 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS:
> 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS -
> spdx.SPDX30Check.test_gcc_include_source: PASSED (209.31s)
> 2024-10-26 01:28:45,260 - oe-selftest - INFO - SUMMARY:
> 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest () - Ran 1 test in
> 216.457s
> 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest - OK - All
> required tests passed (successes=1, skipped=0, failures=0, errors=0)
>
> Signed-off-by: Hongxu Jia
> ---
> meta/lib/oeqa/selftest/cases/spdx.py | 29
> 1 file changed, 29 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/cases/spdx.py
> b/meta/lib/oeqa/selftest/cases/spdx.py
> index be595babb3..8384070219 100644
> --- a/meta/lib/oeqa/selftest/cases/spdx.py
> +++ b/meta/lib/oeqa/selftest/cases/spdx.py
> @@ -110,6 +110,7 @@ class SPDX3CheckBase(object):
> "SDKMACHINE",
> "SDK_DEPLOY",
> "SPDX_VERSION",
> +"SSTATE_PKGARCH",
> "TOOLCHAIN_OUTPUTNAME",
> ],
> target_name,
> @@ -136,6 +137,34 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
> "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/base-files.spdx.json",
> )
>
> +
> +def test_gcc_include_source(self):
> +import oe.spdx30
> +
> +objset = self.check_recipe_spdx(
> +"gcc",
> +"{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/gcc.spdx.json",
> +extraconf=textwrap.dedent(
> +"""\
> +SPDX_INCLUDE_SOURCES = "1"
> +"""
> +),
> +)
> +
> +gcc_pv = get_bb_var("PV", "gcc")
> +filename = f'gcc-{gcc_pv}/README'
> +found = False
> +for software_file in objset.foreach_type(oe.spdx30.software_File):
> +if software_file.name == filename:
> +found = True
> +self.logger.info(f"The spdxId of {filename} in gcc.spdx.json
> is {software_file.spdxId}")
> +break
> +
> +self.assertTrue(
> +found,
> +f"Not found source file {filename} in gcc.spdx.json\n"
> +)
> +
> def test_core_image_minimal(self):
> objset = self.check_recipe_spdx(
> "core-image-minimal",
> --
> 2.25.1
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206467):
https://lists.openembedded.org/g/openembedded-core/message/206467
Mute This Topic: https://lists.openembedded.org/mt/109220137/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 5/6] create-spdx-{2.2,3.0}: support SPDX include source for work-share directory
On Fri, Oct 25, 2024 at 9:25 PM Hongxu Jia wrote:
>
> Originally, while SPDX_INCLUDE_SOURCES = "1" [1], there is bug in scan
> for gcc, libgcc in which the sources locates in work-share directory.
> Copy source from ${WORKDIR} to ${SPDXWORK} did not satisfy the situation
> while ${S} was not included in ${WORKDIR}
>
> This commit aim to support SPDX include source for work-share directory
>
> 1. If is_work_shared_spdx, Copy source from ${S} to ${SPDXWORK},
> normally the dest dir in ${SPDXWORK} has the same basename dir of ${S};
> but for kernel source, rename basename dir 'kernel-source' to ${BP}
> (${BPN}-${PV})
>
> 2. For SPDX source copy, do hard link copy to save copy time
>
> 3. Move do_patch to no work shared situation along with do_unpack
>
> 4. Tweak task do_create_spdx dependencies to assure the patched source
> in work share is ready for SPDX source copy
>
> 5. Remove bb.data.inherits_class('kernel', d) from is_work_shared_spdx,
> the kernel source locates in 'work-shared', test kernel.bbclass is not
> necessary
>
> [1]
> https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES
>
> Signed-off-by: Hongxu Jia
> ---
> meta/classes/create-spdx-2.2.bbclass | 20 ++
> meta/classes/create-spdx-3.0.bbclass | 19 +
> meta/lib/oe/spdx_common.py | 41 +++-
> 3 files changed, 55 insertions(+), 25 deletions(-)
>
> diff --git a/meta/classes/create-spdx-2.2.bbclass
> b/meta/classes/create-spdx-2.2.bbclass
> index cd1d6819bf..900595a7e9 100644
> --- a/meta/classes/create-spdx-2.2.bbclass
> +++ b/meta/classes/create-spdx-2.2.bbclass
> @@ -947,3 +947,23 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir,
> rootfs_spdxid, packages, spdx
> tar.addfile(info, fileobj=index_str)
>
> combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS
> SPDX_MULTILIB_SSTATE_ARCHS"
> +
> +python () {
> +import oe.spdx_common
> +
> +# Assure the patched source in work share is ready for SPDX source copy
> +if oe.spdx_common.is_work_shared_spdx(d) and \
> + d.getVar("SPDX_INCLUDE_SOURCES") == "1" and \
> + oe.spdx_common.process_sources(d):
> +pn = d.getVar('PN')
> +# There is a corner case with "gcc-source-${PV}" recipes, they use
> "do_preconfigure"
> +# to modify source file after do_patch
> +if oe.spdx_common.has_task(d, "do_preconfigure"):
> +d.appendVarFlag('do_create_spdx', 'depends', '
> %s:do_preconfigure' % pn)
> +elif oe.spdx_common.has_task(d, "do_patch"):
> +d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_patch' % pn)
> +# There is a corner case with "gcc-cross-x86_64" recipes, it has
> "do_configure"
> +# but no do_patch
> +elif oe.spdx_common.has_task(d, "do_configure"):
> +d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_configure'
> % pn)
> +}
For some reason, I think I'd prefer something like:
do_create_spdx[depends] = "${@extra_create_spdx_deps(d)}"
def extra_create_spdx_deps(d):
if BLAH:
return "do_configure"
# ETC.
return ""
But I don't know why. Perhaps Richard can weigh in on if one is better
than the other?
> diff --git a/meta/classes/create-spdx-3.0.bbclass
> b/meta/classes/create-spdx-3.0.bbclass
> index 5f0590198f..284e0ff91c 100644
> --- a/meta/classes/create-spdx-3.0.bbclass
> +++ b/meta/classes/create-spdx-3.0.bbclass
> @@ -190,3 +190,22 @@ python spdx30_build_started_handler () {
> addhandler spdx30_build_started_handler
> spdx30_build_started_handler[eventmask] = "bb.event.BuildStarted"
>
> +python () {
> +import oe.spdx_common
> +
> +# Assure the patched source in work share is ready for SPDX source copy
> +if oe.spdx_common.is_work_shared_spdx(d) and \
> + d.getVar("SPDX_INCLUDE_SOURCES") == "1" and \
> + oe.spdx_common.process_sources(d):
> +pn = d.getVar('PN')
> +# There is a corner case with "gcc-source-${PV}" recipes, they use
> "do_preconfigure"
> +# to modify source file after do_patch
> +if oe.spdx_common.has_task(d, "do_preconfigure"):
> +d.appendVarFlag('do_create_spdx', 'depends', '
> %s:do_preconfigure' % pn)
> +elif oe.spdx_common.has_task(d, "do_patch"):
> +d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_patch' % pn)
> +# There is a corner case with "gcc-cross-x86_64" recipes, it has
> "do_configure"
> +# but no do_patch
> +elif oe.spdx_common.has_task(d, "do_configure"):
> +d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_configure'
> % pn)
> +}
> diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
> index 1ea55419ae..ab037b4088 100644
> --- a/meta/lib/oe/spdx_common.py
> +++ b/meta/lib/oe/spdx_common.py
> @@ -38,7 +38,7 @@ def extract_licenses(filename):
>
>
> def is_work_shared_spdx(d):
> -return bb.data.inherits_class("kernel", d) or ("work-shared" in
> d.getVar("WORK
Re: [oe-core][PATCH 4/6] meta/lib/oe/spdx30_tasks.py: improve debug log in add_package_files
On Fri, Oct 25, 2024 at 9:25 PM Hongxu Jia wrote: > > While SPDX_INCLUDE_SOURCES = "1" [1], there are mess of `Adding file' > in debug log > ''' > DEBUG: Adding file > tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/sources-unpack/COPYING > to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native > DEBUG: Adding file > tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/gettext-0.22.5/.tarball-version > to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native > ''' > > Summary the total number other than print for each file. > ''' > DEBUG: Start adding files to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/f5e0e04913ac4c595be791fc001d545a77519ed6ee8c743deef721ca0898bc94/document/gettext-minimal-native > DEBUG: Added 7201 files to > http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/f5e0e04913ac4c595be791fc001d545a77519ed6ee8c743deef721ca0898bc94/document/gettext-minimal-native > ''' > Correct file_counter to start with 0 > > [1] > https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES > > Signed-off-by: Hongxu Jia > --- > meta/lib/oe/spdx30_tasks.py | 8 +--- > 1 file changed, 5 insertions(+), 3 deletions(-) > > diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py > index e0b656d81f..575b66ea5d 100644 > --- a/meta/lib/oe/spdx30_tasks.py > +++ b/meta/lib/oe/spdx30_tasks.py > @@ -144,19 +144,19 @@ def add_package_files( > > spdx_files = set() > > -file_counter = 1 > +bb.debug(1, "Start adding files to %s" % (objset.doc._id)) This probably has little value, so I'd just omit it? > +file_counter = 0 > for subdir, dirs, files in os.walk(topdir): > dirs[:] = [d for d in dirs if d not in ignore_dirs] > if subdir == str(topdir): > dirs[:] = [d for d in dirs if d not in ignore_top_level_dirs] > > + > for file in files: > filepath = Path(subdir) / file > if filepath.is_symlink() or not filepath.is_file(): > continue > > -bb.debug(1, "Adding file %s to %s" % (filepath, objset.doc._id)) > - > filename = str(filepath.relative_to(topdir)) > file_purposes = get_purposes(filepath) > > @@ -187,6 +187,8 @@ def add_package_files( > > file_counter += 1 > > +bb.debug(1, "Added %d files to %s" % (file_counter, objset.doc._id)) len(spdx_files) is more accurate than file_counter and means we don't have to change it to start at 0 > + > return spdx_files > > > -- > 2.25.1 > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#206462): https://lists.openembedded.org/g/openembedded-core/message/206462 Mute This Topic: https://lists.openembedded.org/mt/109220138/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 3/6] meta/lib/oe/sbom30.py: create hasDeclaredLicense relationship conditionally
On Fri, Oct 25, 2024 at 9:25 PM Hongxu Jia wrote:
>
> If file_licenses is empty, the hasDeclaredLicense relationship from sourcefile
> to NoneElement which makes no sense. Such as
This is intentional, as it indicates that we looked for licenses and
didn't find any. Omitting the relationship means we didn't even look
for any licenses (which can also be done explicitly by making a
relationship to "NoAssertion")
>
> {
> "type": "Relationship",
> ...
> "from":
> "http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb/sourcefile/3323";,
> "relationshipType": "hasDeclaredLicense",
> "to": [
> "NoneElement"
> ]
> },
>
> Create hasDeclaredLicense relationship only if file_licenses is not empty
>
> Signed-off-by: Hongxu Jia
> ---
> meta/lib/oe/sbom30.py | 11 ++-
> 1 file changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
> index f4cc1f49dd..6df5759596 100644
> --- a/meta/lib/oe/sbom30.py
> +++ b/meta/lib/oe/sbom30.py
> @@ -620,11 +620,12 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
> for extracted_lic in oe.spdx_common.extract_licenses(filepath):
> file_licenses.add(self.new_license_expression(extracted_lic,
> license_data))
>
> -self.new_relationship(
> -[spdx_file],
> -oe.spdx30.RelationshipType.hasDeclaredLicense,
> -file_licenses,
> -)
> +if file_licenses:
> +self.new_relationship(
> +[spdx_file],
> +oe.spdx30.RelationshipType.hasDeclaredLicense,
> +file_licenses,
> +)
> spdx_file.extension.append(OELicenseScannedExtension())
>
> def new_file(self, _id, name, path, *, purposes=[]):
> --
> 2.25.1
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206461):
https://lists.openembedded.org/g/openembedded-core/message/206461
Mute This Topic: https://lists.openembedded.org/mt/109220134/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 2/6] meta/lib/oe/sbom30.py: correct typo
LGTM, thanks Reviewed-by: Joshua Watt On Fri, Oct 25, 2024 at 9:25 PM Hongxu Jia wrote: > > The isinstance expected 2 arguments > > Signed-off-by: Hongxu Jia > --- > meta/lib/oe/sbom30.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py > index 73a957a20c..f4cc1f49dd 100644 > --- a/meta/lib/oe/sbom30.py > +++ b/meta/lib/oe/sbom30.py > @@ -631,7 +631,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): > sha256_hash = bb.utils.sha256_file(path) > > for f in self.by_sha256_hash.get(sha256_hash, []): > -if not isinstance(oe.spdx30.software_File): > +if not isinstance(f, oe.spdx30.software_File): > continue > > if purposes: > -- > 2.25.1 > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#206459): https://lists.openembedded.org/g/openembedded-core/message/206459 Mute This Topic: https://lists.openembedded.org/mt/109220136/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH 1/6] meta/lib/oe/sbom30.py: correct python list comprehension
On Fri, Oct 25, 2024 at 9:25 PM Hongxu Jia wrote:
>
> The python list comprehension is not right for list:
> >>> license_text_map = {'LicenseRef-FSF-Unlimited':
> >>> 'http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/8d31e22acc4a8979f24dc24042692fb548fc8fc8d85d775ddac406abb122ceea/license-text/FSF-Unlimited'}
> >>> license_text = ((k, license_text_map[k]) for k in
> >>> sorted(license_text_map.keys()))
> >>> print(license_text)
> at 0x7f8575173270>
> >>> [(k,v) for k, v in license_text]
> []
>
> After apply this commit, as expected
> >>> license_text = []
> >>> for k in sorted(license_text_map.keys()):
> ... license_text.append((k, license_text_map[k]))
> ...
> >>> license_text
> [('LicenseRef-FSF-Unlimited',
> 'http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/8d31e22acc4a8979f24dc24042692fb548fc8fc8d85d775ddac406abb122ceea/license-text/FSF-Unlimited')]
>
> Signed-off-by: Hongxu Jia
> ---
> meta/lib/oe/sbom30.py | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
> index 27ab5e45ac..73a957a20c 100644
> --- a/meta/lib/oe/sbom30.py
> +++ b/meta/lib/oe/sbom30.py
> @@ -577,9 +577,9 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
> re.sub(r"[^a-zA-Z0-9_-]", "_", license_expression),
> ]
>
> -license_text = (
> -(k, license_text_map[k]) for k in sorted(license_text_map.keys())
> -)
Probably just change the () to [] to make it a list instead of a
generator expression (I probably was trying to make it a tuple, but
forgot that it would be a generator instead; it doesn't need to be a
tuple).
> +license_text = []
> +for k in sorted(license_text_map.keys()):
> +license_text.append((k, license_text_map[k]))
>
> if not license_text:
> lic = self.find_filter(
> --
> 2.25.1
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206457):
https://lists.openembedded.org/g/openembedded-core/message/206457
Mute This Topic: https://lists.openembedded.org/mt/109220133/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 1/4] classes-global/license: Move functions to library code
Moves several of the functions in license.bbclass to be library code
New function dependencies were manually verified using bitbake-dumpsigs
to ensure that bitbake identified the same dependencies even though they
are now in library code (although the new function names mean that the
task hashes still change)
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass | 10 +-
meta/classes-global/license.bbclass | 165 --
meta/classes-recipe/license_image.bbclass | 14 +-
meta/lib/oe/license.py| 163 +
4 files changed, 175 insertions(+), 177 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index b6940bbb6ff..88b932fc3f0 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -528,8 +528,8 @@ python () {
bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
if bb.data.inherits_class('license', d):
-check_license_format(d)
-unmatched_license_flags = check_license_flags(d)
+oe.license.check_license_format(d)
+unmatched_license_flags = oe.license.check_license_flags(d)
if unmatched_license_flags:
for unmatched in unmatched_license_flags:
message = "Has a restricted license '%s' which is not listed
in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
check_license = False
if check_license and bad_licenses:
-bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or
"").split()
@@ -599,7 +599,7 @@ python () {
for pkg in pkgs:
remaining_bad_licenses =
oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-incompatible_lic = incompatible_license(d,
remaining_bad_licenses, pkg)
+incompatible_lic = oe.license.incompatible_license(d,
remaining_bad_licenses, pkg)
if incompatible_lic:
skipped_pkgs[pkg] = incompatible_lic
else:
@@ -612,7 +612,7 @@ python () {
for pkg in unskipped_pkgs:
bb.debug(1, "Including the package %s" % pkg)
else:
-incompatible_lic = incompatible_license(d, bad_licenses)
+incompatible_lic = oe.license.incompatible_license(d,
bad_licenses)
for pkg in skipped_pkgs:
incompatible_lic += skipped_pkgs[pkg]
incompatible_lic = sorted(list(set(incompatible_lic)))
diff --git a/meta/classes-global/license.bbclass
b/meta/classes-global/license.bbclass
index 043715fcc36..94dcc7f331c 100644
--- a/meta/classes-global/license.bbclass
+++ b/meta/classes-global/license.bbclass
@@ -255,171 +255,6 @@ def find_license_files(d):
return lic_files_paths
-def return_spdx(d, license):
-"""
-This function returns the spdx mapping of a license if it exists.
- """
-return d.getVarFlag('SPDXLICENSEMAP', license)
-
-def canonical_license(d, license):
-"""
-Return the canonical (SPDX) form of the license if available (so GPLv3
-becomes GPL-3.0-only) or the passed license if there is no canonical form.
-"""
-return d.getVarFlag('SPDXLICENSEMAP', license) or license
-
-def expand_wildcard_licenses(d, wildcard_licenses):
-"""
-There are some common wildcard values users may want to use. Support them
-here.
-"""
-licenses = set(wildcard_licenses)
-mapping = {
-"AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
-"GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
-"LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
-}
-for k in mapping:
-if k in wildcard_licenses:
-licenses.remove(k)
-for item in mapping[k]:
-licenses.add(item)
-
-for l in licenses:
-if l in oe.license.obsolete_license_list():
-bb.fatal("Error, %s is an obsolete license, please use an SPDX
reference in INCOMPATIBLE_LICENSE" % l)
-if "*" in l:
-bb.fatal("Error, %s is an invalid license wildcard entry" % l)
-
-return list(licenses)
-
-def incompatible_license_contains(license, truevalue, falsevalue, d):
-license = canonical_license(d, license)
-bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-return truevalue if lice
[OE-core][PATCH v3 0/4] Incompatible Licenses in Dynamic Packages
I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-*
packages at packaging time like other packages. This meant that if there
were *-locale-* packages with an incompatible license, they would still
be generated, and then the final image would likely fail when the
packages were added to it. This is actually not just a problem with
*-locale-* packages, but in fact any packages generated dynamically.
To fix this for all cases, the code to detect incompatible licenses in
packages is moved to a library function, and instead of using the
'_exclude-incompatible-' variable to transfer this knowledge from
base.bbclass to do_package, the library function is called a second time
in do_package. While slightly less efficient, this means that do_package
is checking the actual final list of packages for license exclusions,
make it accurate even when dynamic packages are added.
Finally, the last patch makes it possible to set the LICENSE field of
the generated *-locale-* packages, by making them match
LICENSE:${PN}-locale, if set. This provides a means for recipe writes to
affect the license of the split locale packages in the event that it
doesn't match the recipe LICENSE. This makes logical sense to me, but
please let me know if I'm missing some important assumption
v3: Fixed several bugs that were accidentally omitted from the previous
patch series, and feedback.
Joshua Watt (4):
classes-global/license: Move functions to library code
lib: license: Move package license skip to library
lib: package: Check incompatible licenses at packaging time
lib: package: Copy locale license
meta/classes-global/base.bbclass | 42 +
meta/classes-global/license.bbclass | 165 --
meta/classes-global/package.bbclass | 4 -
meta/classes-recipe/license_image.bbclass | 14 +-
meta/lib/oe/license.py| 202 ++
meta/lib/oe/package.py| 10 +-
6 files changed, 223 insertions(+), 214 deletions(-)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206303):
https://lists.openembedded.org/g/openembedded-core/message/206303
Mute This Topic: https://lists.openembedded.org/mt/109195765/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 4/4] lib: package: Copy locale license
When creating split locales, copy the license from LICENSE:${PN}-locale
if set, otherwise leave it unspecified (which will result in falling
back to LICENSE)
Signed-off-by: Joshua Watt
---
meta/lib/oe/package.py | 4
1 file changed, 4 insertions(+)
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 104b880b9e1..16359232ecd 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -663,6 +663,8 @@ def split_locales(d):
except ValueError:
locale_index = len(packages)
+lic = d.getVar("LICENSE:" + pn + "-locale")
+
localepaths = []
locales = set()
for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -698,6 +700,8 @@ def split_locales(d):
d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn,
mlprefix, ln))
d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
d.setVar('DESCRIPTION:' + pkg, '%s This package contains language
translation files for the %s locale.' % (description, l))
+if lic:
+d.setVar('LICENSE:' + pkg, lic)
if locale_section:
d.setVar('SECTION:' + pkg, locale_section)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206307):
https://lists.openembedded.org/g/openembedded-core/message/206307
Mute This Topic: https://lists.openembedded.org/mt/109195770/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 3/4] lib: package: Check incompatible licenses at packaging time
Instead of checking for incompatible licenses in the anonymous python
and setting '_exclude_incompatible-', (re)check all the packages in
populate_packages(). This ensures that all packages are processed, even
dynamically generated ones.
The use of the '_exclude-incompatible-' variable set in base.bbclass has
been the mechanism used for per-packages licenses since it was added as
a feature (although with different names for the variable throughout
history). However, since this misses dynamic packages, calling
oe.license.skip_incompatible_package_licenses() a second time on the
actual final package set is a better solution.
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass| 1 -
meta/classes-global/package.bbclass | 4
meta/lib/oe/package.py | 6 +++---
3 files changed, 3 insertions(+), 8 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 5b8663f454d..b81e61fdb72 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -581,7 +581,6 @@ python () {
if unskipped_pkgs:
for pkg in skipped_pkgs:
bb.debug(1, "Skipping the package %s at do_rootfs because
of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-d.setVar('_exclude_incompatible-' + pkg, '
'.join(skipped_pkgs[pkg]))
for pkg in unskipped_pkgs:
bb.debug(1, "Including the package %s" % pkg)
else:
diff --git a/meta/classes-global/package.bbclass
b/meta/classes-global/package.bbclass
index 6cd8c0140f2..9be1d6a5b17 100644
--- a/meta/classes-global/package.bbclass
+++ b/meta/classes-global/package.bbclass
@@ -447,10 +447,6 @@ def gen_packagevar(d, pkgvars="PACKAGEVARS"):
for p in pkgs:
for v in vars:
ret.append(v + ":" + p)
-
-# Ensure that changes to INCOMPATIBLE_LICENSE re-run do_package for
-# affected recipes.
-ret.append('_exclude_incompatible-%s' % p)
return " ".join(ret)
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c213a9a3ca6..104b880b9e1 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1447,10 +1447,10 @@ def populate_packages(d):
# Handle excluding packages with incompatible licenses
package_list = []
+skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, packages)
for pkg in packages:
-licenses = d.getVar('_exclude_incompatible-' + pkg)
-if licenses:
-msg = "Excluding %s from packaging as it has incompatible
license(s): %s" % (pkg, licenses)
+if pkg in skipped_pkgs:
+msg = "Excluding %s from packaging as it has incompatible
license(s): %s" % (pkg, skipped_pkgs[pkg])
oe.qa.handle_error("incompatible-license", msg, d)
else:
package_list.append(pkg)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206306):
https://lists.openembedded.org/g/openembedded-core/message/206306
Mute This Topic: https://lists.openembedded.org/mt/109195769/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 2/4] lib: license: Move package license skip to library
Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass | 35
meta/lib/oe/license.py | 39
2 files changed, 43 insertions(+), 31 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 88b932fc3f0..5b8663f454d 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -573,37 +573,10 @@ python () {
bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-check_license = False if pn.startswith("nativesdk-") else True
-for t in ["-native", "-cross-${TARGET_ARCH}",
"-cross-initial-${TARGET_ARCH}",
- "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
- "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-if pn.endswith(d.expand(t)):
-check_license = False
-if pn.startswith("gcc-source-"):
-check_license = False
-
-if check_license and bad_licenses:
-bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or
"").split()
-
-for lic_exception in exceptions:
-if ":" in lic_exception:
-lic_exception = lic_exception.split(":")[1]
-if lic_exception in oe.license.obsolete_license_list():
-bb.fatal("Obsolete license %s used in
INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-pkgs = d.getVar('PACKAGES').split()
-skipped_pkgs = {}
-unskipped_pkgs = []
-for pkg in pkgs:
-remaining_bad_licenses =
oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-incompatible_lic = oe.license.incompatible_license(d,
remaining_bad_licenses, pkg)
-if incompatible_lic:
-skipped_pkgs[pkg] = incompatible_lic
-else:
-unskipped_pkgs.append(pkg)
+pkgs = d.getVar('PACKAGES').split()
+if pkgs:
+skipped_pkgs = oe.license.skip_incompatible_package_licenses(d,
pkgs)
+unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
if unskipped_pkgs:
for pkg in skipped_pkgs:
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index 7739697c401..32c77fa204d 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -422,3 +422,42 @@ def check_license_format(d):
'%s: LICENSE value "%s" has an invalid separator "%s" that
is not ' \
'in the valid list of separators (%s)' %
(pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+if not pkgs:
+return {}
+
+pn = d.getVar("PN")
+
+check_license = False if pn.startswith("nativesdk-") else True
+for t in ["-native", "-cross-${TARGET_ARCH}",
"-cross-initial-${TARGET_ARCH}",
+"-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+"-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+if pn.endswith(d.expand(t)):
+check_license = False
+if pn.startswith("gcc-source-"):
+check_license = False
+
+bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+if not check_license or not bad_licenses:
+return {}
+
+bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+for lic_exception in exceptions:
+if ":" in lic_exception:
+lic_exception = lic_exception.split(":")[1]
+if lic_exception in obsolete_license_list():
+bb.fatal("Obsolete license %s used in
INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+skipped_pkgs = {}
+for pkg in pkgs:
+remaining_bad_licenses = apply_pkg_license_exception(pkg,
bad_licenses, exceptions)
+
+incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+if incompatible_lic:
+skipped_pkgs[pkg] = incompatible_lic
+
+return skipped_pkgs
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206305):
https://lists.openembedded.org/g/openembedded-core/message/206305
Mute This Topic: https://lists.openembedded.org/mt/109195768/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time
On Thu, Oct 24, 2024 at 4:03 AM Peter Kjellerstedt
wrote:
>
> > -Original Message-
> > From: openembedded-core@lists.openembedded.org
> > On Behalf Of Joshua Watt
> > Sent: den 23 oktober 2024 23:15
> > To: openembedded-core@lists.openembedded.org
> > Cc: Joshua Watt
> > Subject: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses
> > at packaging time
> >
> > Instead of checking for incompatible licenses in the anonymous python
> > and setting '_exclude_incompatible-', (re)check all the packages in
> > populate_packages(). This ensures that all packages are processed, even
> > dynamically generated ones.
> >
> > The use of the '_exclude-incompatible-' variable set in base.bbclass has
> > been the mechanism used for per-packages licenses since its it was added
> > as a feature (although with different names for the variable throughout
> > history). However, since this misses dynamic packages, calling
> > oe.license.skip_incompatible_package_licenses() a second time on the
> > actual final package set is a better solution.
> >
> > Signed-off-by: Joshua Watt
> > ---
> > meta/classes-global/base.bbclass | 1 -
> > meta/lib/oe/package.py | 6 +++---
> > 2 files changed, 3 insertions(+), 4 deletions(-)
> >
> > diff --git a/meta/classes-global/base.bbclass
> > b/meta/classes-global/base.bbclass
> > index 5b8663f454d..b81e61fdb72 100644
> > --- a/meta/classes-global/base.bbclass
> > +++ b/meta/classes-global/base.bbclass
> > @@ -581,7 +581,6 @@ python () {
> > if unskipped_pkgs:
> > for pkg in skipped_pkgs:
> > bb.debug(1, "Skipping the package %s at do_rootfs
> > because of incompatible license(s): %s" % (pkg, '
> > '.join(skipped_pkgs[pkg])))
> > -d.setVar('_exclude_incompatible-' + pkg, '
> > '.join(skipped_pkgs[pkg]))
> > for pkg in unskipped_pkgs:
> > bb.debug(1, "Including the package %s" % pkg)
> > else:
> > diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> > index c213a9a3ca6..480408e41e3 100644
> > --- a/meta/lib/oe/package.py
> > +++ b/meta/lib/oe/package.py
> > @@ -1447,10 +1447,10 @@ def populate_packages(d):
> >
> > # Handle excluding packages with incompatible licenses
> > package_list = []
> > +skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):
>
> Umm, how well did you test this? ;)
> The Python interpreter is not too happy about the trailing colon on the line
> above...
Yep, because I found that, fixed it, and forgot to squash the fix in
before I submitted
I'll send a V3
>
> > for pkg in packages:
> > -licenses = d.getVar('_exclude_incompatible-' + pkg)
> > -if licenses:
> > -msg = "Excluding %s from packaging as it has incompatible
> > license(s): %s" % (pkg, licenses)
> > +if pkg in skipped_pkgs:
> > +msg = "Excluding %s from packaging as it has incompatible
> > license(s): %s" % (pkg, skipped_pkgs[pkg])
> > oe.qa.handle_error("incompatible-license", msg, d)
> > else:
> > package_list.append(pkg)
> > --
> > 2.46.2
>
> //Peter
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206300):
https://lists.openembedded.org/g/openembedded-core/message/206300
Mute This Topic: https://lists.openembedded.org/mt/109179697/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v2 1/4] classes-global/license: Move functions to library code
Moves several of the functions in license.bbclass to be library code
New function dependencies were manually verified using bitbake-dumpsigs
to ensure that bitbake identified the same dependencies even though they
are now in library code (although the new function names mean that the
task hashes still change)
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass | 10 +-
meta/classes-global/license.bbclass | 165 --
meta/classes-recipe/license_image.bbclass | 14 +-
meta/lib/oe/license.py| 163 +
4 files changed, 175 insertions(+), 177 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index b6940bbb6ff..88b932fc3f0 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -528,8 +528,8 @@ python () {
bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
if bb.data.inherits_class('license', d):
-check_license_format(d)
-unmatched_license_flags = check_license_flags(d)
+oe.license.check_license_format(d)
+unmatched_license_flags = oe.license.check_license_flags(d)
if unmatched_license_flags:
for unmatched in unmatched_license_flags:
message = "Has a restricted license '%s' which is not listed
in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
check_license = False
if check_license and bad_licenses:
-bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or
"").split()
@@ -599,7 +599,7 @@ python () {
for pkg in pkgs:
remaining_bad_licenses =
oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-incompatible_lic = incompatible_license(d,
remaining_bad_licenses, pkg)
+incompatible_lic = oe.license.incompatible_license(d,
remaining_bad_licenses, pkg)
if incompatible_lic:
skipped_pkgs[pkg] = incompatible_lic
else:
@@ -612,7 +612,7 @@ python () {
for pkg in unskipped_pkgs:
bb.debug(1, "Including the package %s" % pkg)
else:
-incompatible_lic = incompatible_license(d, bad_licenses)
+incompatible_lic = oe.license.incompatible_license(d,
bad_licenses)
for pkg in skipped_pkgs:
incompatible_lic += skipped_pkgs[pkg]
incompatible_lic = sorted(list(set(incompatible_lic)))
diff --git a/meta/classes-global/license.bbclass
b/meta/classes-global/license.bbclass
index 043715fcc36..94dcc7f331c 100644
--- a/meta/classes-global/license.bbclass
+++ b/meta/classes-global/license.bbclass
@@ -255,171 +255,6 @@ def find_license_files(d):
return lic_files_paths
-def return_spdx(d, license):
-"""
-This function returns the spdx mapping of a license if it exists.
- """
-return d.getVarFlag('SPDXLICENSEMAP', license)
-
-def canonical_license(d, license):
-"""
-Return the canonical (SPDX) form of the license if available (so GPLv3
-becomes GPL-3.0-only) or the passed license if there is no canonical form.
-"""
-return d.getVarFlag('SPDXLICENSEMAP', license) or license
-
-def expand_wildcard_licenses(d, wildcard_licenses):
-"""
-There are some common wildcard values users may want to use. Support them
-here.
-"""
-licenses = set(wildcard_licenses)
-mapping = {
-"AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
-"GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
-"LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
-}
-for k in mapping:
-if k in wildcard_licenses:
-licenses.remove(k)
-for item in mapping[k]:
-licenses.add(item)
-
-for l in licenses:
-if l in oe.license.obsolete_license_list():
-bb.fatal("Error, %s is an obsolete license, please use an SPDX
reference in INCOMPATIBLE_LICENSE" % l)
-if "*" in l:
-bb.fatal("Error, %s is an invalid license wildcard entry" % l)
-
-return list(licenses)
-
-def incompatible_license_contains(license, truevalue, falsevalue, d):
-license = canonical_license(d, license)
-bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-return truevalue if lice
[OE-core][PATCH v2 4/4] lib: package: Copy locale license
When creating split locales, copy the license from LICENSE:${PN}-locale
if set, otherwise leave it unspecified (which will result in falling
back to LICENSE)
Signed-off-by: Joshua Watt
---
meta/lib/oe/package.py | 4
1 file changed, 4 insertions(+)
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 480408e41e3..236a0edbcb2 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -663,6 +663,8 @@ def split_locales(d):
except ValueError:
locale_index = len(packages)
+lic = d.getVar("LICENSE:" + pn + "-locale")
+
localepaths = []
locales = set()
for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -698,6 +700,8 @@ def split_locales(d):
d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn,
mlprefix, ln))
d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
d.setVar('DESCRIPTION:' + pkg, '%s This package contains language
translation files for the %s locale.' % (description, l))
+if lic:
+d.setVar('LICENSE:' + pkg, lic)
if locale_section:
d.setVar('SECTION:' + pkg, locale_section)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206269):
https://lists.openembedded.org/g/openembedded-core/message/206269
Mute This Topic: https://lists.openembedded.org/mt/109179699/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time
Instead of checking for incompatible licenses in the anonymous python
and setting '_exclude_incompatible-', (re)check all the packages in
populate_packages(). This ensures that all packages are processed, even
dynamically generated ones.
The use of the '_exclude-incompatible-' variable set in base.bbclass has
been the mechanism used for per-packages licenses since its it was added
as a feature (although with different names for the variable throughout
history). However, since this misses dynamic packages, calling
oe.license.skip_incompatible_package_licenses() a second time on the
actual final package set is a better solution.
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass | 1 -
meta/lib/oe/package.py | 6 +++---
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 5b8663f454d..b81e61fdb72 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -581,7 +581,6 @@ python () {
if unskipped_pkgs:
for pkg in skipped_pkgs:
bb.debug(1, "Skipping the package %s at do_rootfs because
of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-d.setVar('_exclude_incompatible-' + pkg, '
'.join(skipped_pkgs[pkg]))
for pkg in unskipped_pkgs:
bb.debug(1, "Including the package %s" % pkg)
else:
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c213a9a3ca6..480408e41e3 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1447,10 +1447,10 @@ def populate_packages(d):
# Handle excluding packages with incompatible licenses
package_list = []
+skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):
for pkg in packages:
-licenses = d.getVar('_exclude_incompatible-' + pkg)
-if licenses:
-msg = "Excluding %s from packaging as it has incompatible
license(s): %s" % (pkg, licenses)
+if pkg in skipped_pkgs:
+msg = "Excluding %s from packaging as it has incompatible
license(s): %s" % (pkg, skipped_pkgs[pkg])
oe.qa.handle_error("incompatible-license", msg, d)
else:
package_list.append(pkg)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206268):
https://lists.openembedded.org/g/openembedded-core/message/206268
Mute This Topic: https://lists.openembedded.org/mt/109179697/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v2 2/4] lib: license: Move package license skip to library
Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass | 35
meta/lib/oe/license.py | 39
2 files changed, 43 insertions(+), 31 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 88b932fc3f0..5b8663f454d 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -573,37 +573,10 @@ python () {
bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-check_license = False if pn.startswith("nativesdk-") else True
-for t in ["-native", "-cross-${TARGET_ARCH}",
"-cross-initial-${TARGET_ARCH}",
- "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
- "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-if pn.endswith(d.expand(t)):
-check_license = False
-if pn.startswith("gcc-source-"):
-check_license = False
-
-if check_license and bad_licenses:
-bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or
"").split()
-
-for lic_exception in exceptions:
-if ":" in lic_exception:
-lic_exception = lic_exception.split(":")[1]
-if lic_exception in oe.license.obsolete_license_list():
-bb.fatal("Obsolete license %s used in
INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-pkgs = d.getVar('PACKAGES').split()
-skipped_pkgs = {}
-unskipped_pkgs = []
-for pkg in pkgs:
-remaining_bad_licenses =
oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-incompatible_lic = oe.license.incompatible_license(d,
remaining_bad_licenses, pkg)
-if incompatible_lic:
-skipped_pkgs[pkg] = incompatible_lic
-else:
-unskipped_pkgs.append(pkg)
+pkgs = d.getVar('PACKAGES').split()
+if pkgs:
+skipped_pkgs = oe.license.skip_incompatible_package_licenses(d,
pkgs)
+unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
if unskipped_pkgs:
for pkg in skipped_pkgs:
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index 7739697c401..866a876d7f0 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -422,3 +422,42 @@ def check_license_format(d):
'%s: LICENSE value "%s" has an invalid separator "%s" that
is not ' \
'in the valid list of separators (%s)' %
(pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+if not pkgs:
+return {}
+
+pn = d.getVar("PN")
+bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+check_license = False if pn.startswith("nativesdk-") else True
+for t in ["-native", "-cross-${TARGET_ARCH}",
"-cross-initial-${TARGET_ARCH}",
+"-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+"-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+if pn.endswith(d.expand(t)):
+check_license = False
+if pn.startswith("gcc-source-"):
+check_license = False
+
+if not check_license or not bad_licenses:
+return {}
+
+bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+for lic_exception in exceptions:
+if ":" in lic_exception:
+lic_exception = lic_exception.split(":")[1]
+if lic_exception in obsolete_license_list():
+bb.fatal("Obsolete license %s used in
INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+skipped_pkgs = {}
+for pkg in pkgs:
+remaining_bad_licenses = apply_pkg_license_exception(pkg,
bad_licenses, exceptions)
+
+incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+if incompatible_lic:
+skipped_pkgs[pkg] = incompatible_lic
+
+return skipped_pkgs
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206267):
https://lists.openembedded.org/g/openembedded-core/message/206267
Mute This Topic: https://lists.openembedded.org/mt/109179696/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v2 0/4] Incompatible Licenses in Dynamic Packages
I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-*
packages at packaging time like other packages. This meant that if there
were *-locale-* packages with an incompatible license, they would still
be generated, and then the final image would likely fail when the
packages were added to it. This is actually not just a problem with
*-locale-* packages, but in fact any packages generated dynamically.
To fix this for all cases, the code to detect incompatible licenses in
packages is moved to a library function, and instead of using the
'_exclude-incompatible-' variable to transfer this knowledge from
base.bbclass to do_package, the library function is called a second time
in do_package. While slightly less efficient, this means that do_package
is checking the actual final list of packages for license exclusions,
make it accurate even when dynamic packages are added.
Finally, the last patch makes it possible to set the LICENSE field of
the generated *-locale-* packages, by making them match
LICENSE:${PN}-locale, if set. This provides a means for recipe writes to
affect the license of the split locale packages in the event that it
doesn't match the recipe LICENSE. This makes logical sense to me, but
please let me know if I'm missing some important assumption
Joshua Watt (4):
classes-global/license: Move functions to library code
lib: license: Move package license skip to library
lib: package: Check incompatible licenses at packaging time
lib: package: Copy locale license
meta/classes-global/base.bbclass | 42 +
meta/classes-global/license.bbclass | 165 --
meta/classes-recipe/license_image.bbclass | 14 +-
meta/lib/oe/license.py| 202 ++
meta/lib/oe/package.py| 10 +-
5 files changed, 223 insertions(+), 210 deletions(-)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206265):
https://lists.openembedded.org/g/openembedded-core/message/206265
Mute This Topic: https://lists.openembedded.org/mt/109179692/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 4/4] lib: package: Copy locale license
When creating split locales, copy the license from LICENSE:${PN}-locale
if set, otherwise leave it unspecified (which will result in falling
back to LICENSE)
Signed-off-by: Joshua Watt
---
meta/lib/oe/package.py | 4
1 file changed, 4 insertions(+)
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 37fbfe355cb..be8bffbd37b 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -663,6 +663,8 @@ def split_locales(d):
except ValueError:
locale_index = len(packages)
+lic = d.getVar("LICENSE:" + pn + "-locale")
+
localepaths = []
locales = set()
for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -700,6 +702,8 @@ def split_locales(d):
d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn,
mlprefix, ln))
d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
d.setVar('DESCRIPTION:' + pkg, '%s This package contains language
translation files for the %s locale.' % (description, l))
+if lic:
+d.setVar('LICENSE:' + pkg, lic)
if locale_section:
d.setVar('SECTION:' + pkg, locale_section)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206077):
https://lists.openembedded.org/g/openembedded-core/message/206077
Mute This Topic: https://lists.openembedded.org/mt/109092371/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 3/4] lib: package: Check for incompatible licenses in locale packages
Checks if incompatible licenses are present in the dynamically generated
locale packages
Signed-off-by: Joshua Watt
---
meta/lib/oe/package.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c213a9a3ca6..37fbfe355cb 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -685,9 +685,11 @@ def split_locales(d):
description = d.getVar('DESCRIPTION') or ""
locale_section = d.getVar('LOCALE_SECTION')
mlprefix = d.getVar('MLPREFIX') or ""
+locale_pkgs = []
for l in sorted(locales):
ln = legitimize_package_name(l)
pkg = pn + '-locale-' + ln
+locale_pkgs.append(pkg)
packages.insert(locale_index, pkg)
locale_index += 1
files = []
@@ -702,6 +704,7 @@ def split_locales(d):
d.setVar('SECTION:' + pkg, locale_section)
d.setVar('PACKAGES', ' '.join(packages))
+oe.license.skip_incompatible_package_licenses(d, locale_pkgs)
# Disabled by RP 18/06/07
# Wildcards aren't supported in debian
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206076):
https://lists.openembedded.org/g/openembedded-core/message/206076
Mute This Topic: https://lists.openembedded.org/mt/109092368/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 2/4] lib: license: Move package license skip to library
Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass | 36
meta/lib/oe/license.py | 40
2 files changed, 44 insertions(+), 32 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 88b932fc3f0..b81e61fdb72 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -573,42 +573,14 @@ python () {
bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-check_license = False if pn.startswith("nativesdk-") else True
-for t in ["-native", "-cross-${TARGET_ARCH}",
"-cross-initial-${TARGET_ARCH}",
- "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
- "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-if pn.endswith(d.expand(t)):
-check_license = False
-if pn.startswith("gcc-source-"):
-check_license = False
-
-if check_license and bad_licenses:
-bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or
"").split()
-
-for lic_exception in exceptions:
-if ":" in lic_exception:
-lic_exception = lic_exception.split(":")[1]
-if lic_exception in oe.license.obsolete_license_list():
-bb.fatal("Obsolete license %s used in
INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-pkgs = d.getVar('PACKAGES').split()
-skipped_pkgs = {}
-unskipped_pkgs = []
-for pkg in pkgs:
-remaining_bad_licenses =
oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-incompatible_lic = oe.license.incompatible_license(d,
remaining_bad_licenses, pkg)
-if incompatible_lic:
-skipped_pkgs[pkg] = incompatible_lic
-else:
-unskipped_pkgs.append(pkg)
+pkgs = d.getVar('PACKAGES').split()
+if pkgs:
+skipped_pkgs = oe.license.skip_incompatible_package_licenses(d,
pkgs)
+unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
if unskipped_pkgs:
for pkg in skipped_pkgs:
bb.debug(1, "Skipping the package %s at do_rootfs because
of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-d.setVar('_exclude_incompatible-' + pkg, '
'.join(skipped_pkgs[pkg]))
for pkg in unskipped_pkgs:
bb.debug(1, "Including the package %s" % pkg)
else:
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index 7739697c401..75b1e70ba0e 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -422,3 +422,43 @@ def check_license_format(d):
'%s: LICENSE value "%s" has an invalid separator "%s" that
is not ' \
'in the valid list of separators (%s)' %
(pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+if not pkgs:
+return {}
+
+pn = d.getVar("PN")
+bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+check_license = False if pn.startswith("nativesdk-") else True
+for t in ["-native", "-cross-${TARGET_ARCH}",
"-cross-initial-${TARGET_ARCH}",
+"-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+"-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+if pn.endswith(d.expand(t)):
+check_license = False
+if pn.startswith("gcc-source-"):
+check_license = False
+
+if not check_license or not bad_licenses:
+return {}
+
+bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+for lic_exception in exceptions:
+if ":" in lic_exception:
+lic_exception = lic_exception.split(":")[1]
+if lic_exception in obsolete_license_list():
+bb.fatal("Obsolete license %s used in
INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+skipped_pkgs = {}
+for pkg in pkgs:
+remaining_bad_licenses = apply_pkg_license_exception(pkg,
bad_licenses, exceptions)
+
+incompatible_lic =
[OE-core][PATCH 0/4] Incompatible license handling fixes
I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-*
packages at packaging time like other packages. This meant that if there
were *-locale-* packages with an incompatible license, they would still
be generated, and then the final image would likely fail when the
packages were added to it.
These patches are more less ordered from least controversial to most
controversial :)
I suspect the first two are probably fine optimizations, the third uses
the new library code to make sure INCOMPATIBLE_LICENCE correctly applies
to *-locale-* packages. This same call would likely also need to be
called anywhere else dynamic package generation happens, unless there is
some smart way to detect that.
Finally, the last patch makes it possible to set the LICENSE field of
the generated *-locale-* packages, by making them match
LICENSE:${PN}-locale, if set. This provides a means for recipe writes to
affect the license of the split locale packages in the event that it
doesn't match the recipe LICENSE. This makes logical sense to me, but
please let me know if I'm missing some important assumption
Joshua Watt (4):
classes-global/license: Move functions to library code
lib: license: Move package license skip to library
lib: package: Check for incompatible licenses in locale packages
lib: package: Copy locale license
meta/classes-global/base.bbclass | 42 +
meta/classes-global/license.bbclass | 165 --
meta/classes-recipe/license_image.bbclass | 14 +-
meta/lib/oe/license.py| 203 ++
meta/lib/oe/package.py| 7 +
5 files changed, 224 insertions(+), 207 deletions(-)
--
2.46.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206073):
https://lists.openembedded.org/g/openembedded-core/message/206073
Mute This Topic: https://lists.openembedded.org/mt/109092362/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 1/4] classes-global/license: Move functions to library code
Moves several of the functions in license.bbclass to be library code
Signed-off-by: Joshua Watt
---
meta/classes-global/base.bbclass | 10 +-
meta/classes-global/license.bbclass | 165 --
meta/classes-recipe/license_image.bbclass | 14 +-
meta/lib/oe/license.py| 163 +
4 files changed, 175 insertions(+), 177 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index b6940bbb6ff..88b932fc3f0 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -528,8 +528,8 @@ python () {
bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
if bb.data.inherits_class('license', d):
-check_license_format(d)
-unmatched_license_flags = check_license_flags(d)
+oe.license.check_license_format(d)
+unmatched_license_flags = oe.license.check_license_flags(d)
if unmatched_license_flags:
for unmatched in unmatched_license_flags:
message = "Has a restricted license '%s' which is not listed
in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
check_license = False
if check_license and bad_licenses:
-bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or
"").split()
@@ -599,7 +599,7 @@ python () {
for pkg in pkgs:
remaining_bad_licenses =
oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-incompatible_lic = incompatible_license(d,
remaining_bad_licenses, pkg)
+incompatible_lic = oe.license.incompatible_license(d,
remaining_bad_licenses, pkg)
if incompatible_lic:
skipped_pkgs[pkg] = incompatible_lic
else:
@@ -612,7 +612,7 @@ python () {
for pkg in unskipped_pkgs:
bb.debug(1, "Including the package %s" % pkg)
else:
-incompatible_lic = incompatible_license(d, bad_licenses)
+incompatible_lic = oe.license.incompatible_license(d,
bad_licenses)
for pkg in skipped_pkgs:
incompatible_lic += skipped_pkgs[pkg]
incompatible_lic = sorted(list(set(incompatible_lic)))
diff --git a/meta/classes-global/license.bbclass
b/meta/classes-global/license.bbclass
index 043715fcc36..94dcc7f331c 100644
--- a/meta/classes-global/license.bbclass
+++ b/meta/classes-global/license.bbclass
@@ -255,171 +255,6 @@ def find_license_files(d):
return lic_files_paths
-def return_spdx(d, license):
-"""
-This function returns the spdx mapping of a license if it exists.
- """
-return d.getVarFlag('SPDXLICENSEMAP', license)
-
-def canonical_license(d, license):
-"""
-Return the canonical (SPDX) form of the license if available (so GPLv3
-becomes GPL-3.0-only) or the passed license if there is no canonical form.
-"""
-return d.getVarFlag('SPDXLICENSEMAP', license) or license
-
-def expand_wildcard_licenses(d, wildcard_licenses):
-"""
-There are some common wildcard values users may want to use. Support them
-here.
-"""
-licenses = set(wildcard_licenses)
-mapping = {
-"AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
-"GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
-"LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
-}
-for k in mapping:
-if k in wildcard_licenses:
-licenses.remove(k)
-for item in mapping[k]:
-licenses.add(item)
-
-for l in licenses:
-if l in oe.license.obsolete_license_list():
-bb.fatal("Error, %s is an obsolete license, please use an SPDX
reference in INCOMPATIBLE_LICENSE" % l)
-if "*" in l:
-bb.fatal("Error, %s is an invalid license wildcard entry" % l)
-
-return list(licenses)
-
-def incompatible_license_contains(license, truevalue, falsevalue, d):
-license = canonical_license(d, license)
-bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-return truevalue if license in bad_licenses else falsevalue
-
-def incompatible_pkg_license(d, dont_want_licenses, license):
-# Handles an "or" or two license sets provided by
-# flattened_licenses(), pick one that works if possible.
-def choose_l
[OE-core][PATCH] spdx30: Link license and build by alias
The license information and Build created by do_create_spdx are changed
to be referenced by their link alias instead of the actual SPDX ID. This
fixes a case where do_create_package_spdx would pull these from
mismatching sstate, and then the SPDX IDs would be unresolved when
assembling the final document
Signed-off-by: Joshua Watt
---
meta/lib/oe/sbom30.py | 43 +
meta/lib/oe/spdx30_tasks.py | 10 ++---
2 files changed, 31 insertions(+), 22 deletions(-)
diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
index 7b4f78cc718..27ab5e45ac1 100644
--- a/meta/lib/oe/sbom30.py
+++ b/meta/lib/oe/sbom30.py
@@ -305,24 +305,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
def add_aliases(self):
for o in self.foreach_type(oe.spdx30.Element):
-if not o._id or o._id.startswith("_:"):
-continue
-
-alias_ext = get_alias(o)
-if alias_ext is None:
-unihash = self.d.getVar("BB_UNIHASH")
-namespace = self.get_namespace()
-if unihash not in o._id:
-bb.warn(f"Unihash {unihash} not found in {o._id}")
-elif namespace not in o._id:
-bb.warn(f"Namespace {namespace} not found in {o._id}")
-else:
-alias_ext = set_alias(
-o,
-o._id.replace(unihash, "UNIHASH").replace(
-namespace, self.d.getVar("PN")
-),
-)
+self.set_element_alias(o)
def remove_internal_extensions(self):
def remove(o):
@@ -345,6 +328,26 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
str(uuid.uuid5(namespace_uuid, pn)),
)
+def set_element_alias(self, e):
+if not e._id or e._id.startswith("_:"):
+return
+
+alias_ext = get_alias(e)
+if alias_ext is None:
+unihash = self.d.getVar("BB_UNIHASH")
+namespace = self.get_namespace()
+if unihash not in e._id:
+bb.warn(f"Unihash {unihash} not found in {e._id}")
+elif namespace not in e._id:
+bb.warn(f"Namespace {namespace} not found in {e._id}")
+else:
+alias_ext = set_alias(
+e,
+e._id.replace(unihash, "UNIHASH").replace(
+namespace, self.d.getVar("PN")
+),
+)
+
def new_spdxid(self, *suffix, include_unihash=True):
items = [self.get_namespace()]
if include_unihash:
@@ -557,7 +560,9 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
scope=scope,
)
-def new_license_expression(self, license_expression, license_data,
license_text_map={}):
+def new_license_expression(
+self, license_expression, license_data, license_text_map={}
+):
license_list_version = license_data["licenseListVersion"]
# SPDX 3 requires that the license list version be a semver
# MAJOR.MINOR.MICRO, but the actual license version might be
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index d0dd40877e2..e0b656d81f1 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -119,9 +119,11 @@ def add_license_expression(d, objset, license_expression,
license_data):
)
spdx_license_expression = " ".join(convert(l) for l in lic_split)
-return objset.new_license_expression(
+o = objset.new_license_expression(
spdx_license_expression, license_data, license_text_map
)
+objset.set_element_alias(o)
+return o
def add_package_files(
@@ -462,6 +464,8 @@ def create_spdx(d):
build_objset = oe.sbom30.ObjectSet.new_objset(d, d.getVar("PN"))
build = build_objset.new_task_build("recipe", "recipe")
+build_objset.set_element_alias(build)
+
build_objset.doc.rootElement.append(build)
build_objset.set_is_native(is_native)
@@ -603,7 +607,7 @@ def create_spdx(d):
set_var_field("DESCRIPTION", spdx_package, "description",
package=package)
pkg_objset.new_scoped_relationship(
-[build._id],
+[oe.sbom30.get_element_link_id(build)],
oe.spdx30.RelationshipType.hasOutput,
oe.spdx30.LifecycleScopeType.build,
[spdx_package],
@@ -650,7 +654,7 @@ def create_spdx(d):
pkg_objset.new_relationship(
[spdx_package],
oe.spdx30.RelationshipType.hasConcludedLicense,
-[package_spdx_license._id],
+[oe.sbom30.get_element_link_id(package_spdx_license)],
)
[OE-core][PATCH 4/4] Add script to make SPDX bindings
Adds a script to generate the SPDX code bindings Signed-off-by: Joshua Watt --- scripts/contrib/make-spdx-bindings.sh | 12 1 file changed, 12 insertions(+) create mode 100755 scripts/contrib/make-spdx-bindings.sh diff --git a/scripts/contrib/make-spdx-bindings.sh b/scripts/contrib/make-spdx-bindings.sh new file mode 100755 index 000..31caaf339d3 --- /dev/null +++ b/scripts/contrib/make-spdx-bindings.sh @@ -0,0 +1,12 @@ +#! /bin/sh +# +# SPDX-License-Identifier: MIT + +THIS_DIR="$(dirname "$0")" + +VERSION="3.0.1" + +shacl2code generate --input https://spdx.org/rdf/$VERSION/spdx-model.ttl \ +--input https://spdx.org/rdf/$VERSION/spdx-json-serialize-annotations.ttl \ +--context https://spdx.org/rdf/$VERSION/spdx-context.jsonld \ +python -o $THIS_DIR/../../meta/lib/oe/spdx30.py -- 2.46.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#205034): https://lists.openembedded.org/g/openembedded-core/message/205034 Mute This Topic: https://lists.openembedded.org/mt/108688996/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 0/4] SPDX 3.0 improvements
Various bug fixes and improvements to SPDX 3.0 support, notably upgrading to SPDX 3.0.1 Joshua Watt (4): spdx 3.0: Find local sources when searching for debug sources spdx 3.0: Map gitsm URI to git create-spdx-3.0: Upgrade to SPDX 3.0.1 Add script to make SPDX bindings meta/classes/create-spdx-2.2.bbclass | 11 +- meta/classes/create-spdx-3.0.bbclass |2 +- meta/lib/oe/sbom30.py | 15 +- meta/lib/oe/spdx30.py | 3375 +++-- meta/lib/oe/spdx30_tasks.py | 112 +- meta/lib/oe/spdx_common.py| 20 +- scripts/contrib/make-spdx-bindings.sh | 12 + 7 files changed, 1560 insertions(+), 1987 deletions(-) create mode 100755 scripts/contrib/make-spdx-bindings.sh -- 2.46.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#205030): https://lists.openembedded.org/g/openembedded-core/message/205030 Mute This Topic: https://lists.openembedded.org/mt/108688986/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH 3/4] create-spdx-3.0: Upgrade to SPDX 3.0.1
On Fri, Sep 27, 2024 at 10:02 AM Ross Burton via lists.openembedded.org wrote: > > On 27 Sep 2024, at 16:51, Joshua Watt via lists.openembedded.org > wrote: > > > > Upgrades the SPDX 3.0 implementation from 3.0.0 -> 3.0.1. This version > > introduced some breaking changes. Effectively, 3.0.0 was a pre-release > > version that we do not need to support any longer. > > This sounds… important. > > Should we ram this into styhead for the 5.1 release? Would upgrading to 3.0.1 > in a styhead point release be reasonable if not? Or should this stay in > master and not touch styhead? Per discussion with Richard, we don't want to hold up the release for this, but it will need to be backported to styhead. I don't have a preference if that is before or after the release > > Cheers, > Ross > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#205039): https://lists.openembedded.org/g/openembedded-core/message/205039 Mute This Topic: https://lists.openembedded.org/mt/108688994/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 2/4] spdx 3.0: Map gitsm URI to git
"gitsm" is not a recognized URI protocol (outside of bitbake), so map it
to "git" when writing. This should be OK since we report all of the
submodule source code (if enabled), and it's still possible for 3rd
party analyzers to determine that submodules are in use by looking at
.gitmodules.
The code to do the mapping is moved to a common location so it covers
SPDX 2.2 also
[YOCTO #15582]
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-2.2.bbclass | 11 +--
meta/lib/oe/spdx30_tasks.py | 13 +++--
meta/lib/oe/spdx_common.py | 20 +++-
3 files changed, 23 insertions(+), 21 deletions(-)
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index 795ba1a8826..cd1d6819bf7 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -354,15 +354,6 @@ def add_download_packages(d, doc, recipe):
if f.type == "file":
continue
-uri = f.type
-proto = getattr(f, "proto", None)
-if proto is not None:
-uri = uri + "+" + proto
-uri = uri + "://" + f.host + f.path
-
-if f.method.supports_srcrev():
-uri = uri + "@" + f.revisions[name]
-
if f.method.supports_checksum(f):
for checksum_id in CHECKSUM_LIST:
if checksum_id.upper() not in
oe.spdx.SPDXPackage.ALLOWED_CHECKSUMS:
@@ -377,7 +368,7 @@ def add_download_packages(d, doc, recipe):
c.checksumValue = expected_checksum
package.checksums.append(c)
-package.downloadLocation = uri
+package.downloadLocation = oe.spdx_common.fetch_data_to_uri(f,
name)
doc.packages.append(package)
doc.add_relationship(doc, "DESCRIBES", package)
# In the future, we might be able to do more fancy dependencies,
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 70d1bc7e8ae..1ae13b4af82 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -379,22 +379,15 @@ def add_download_files(d, objset):
inputs.add(file)
else:
-uri = fd.type
-proto = getattr(fd, "proto", None)
-if proto is not None:
-uri = uri + "+" + proto
-uri = uri + "://" + fd.host + fd.path
-
-if fd.method.supports_srcrev():
-uri = uri + "@" + fd.revisions[name]
-
dl = objset.add(
oe.spdx30.software_Package(
_id=objset.new_spdxid("source", str(download_idx + 1)),
creationInfo=objset.doc.creationInfo,
name=file_name,
software_primaryPurpose=primary_purpose,
-software_downloadLocation=uri,
+
software_downloadLocation=oe.spdx_common.fetch_data_to_uri(
+fd, name
+),
)
)
diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
index dfe90f96cf9..1ea55419aeb 100644
--- a/meta/lib/oe/spdx_common.py
+++ b/meta/lib/oe/spdx_common.py
@@ -42,7 +42,6 @@ def is_work_shared_spdx(d):
def load_spdx_license_data(d):
-
with open(d.getVar("SPDX_LICENSES"), "r") as f:
data = json.load(f)
# Transform the license array to a dictionary
@@ -225,3 +224,22 @@ def get_patched_src(d):
bb.utils.mkdirhier(spdx_workdir)
finally:
d.setVar("WORKDIR", workdir)
+
+
+def fetch_data_to_uri(fd, name):
+"""
+Translates a bitbake FetchData to a string URI
+"""
+uri = fd.type
+# Map gitsm to git, since gitsm:// is not a valid URI protocol
+if uri == "gitsm":
+uri = "git"
+proto = getattr(fd, "proto", None)
+if proto is not None:
+uri = uri + "+" + proto
+uri = uri + "://" + fd.host + fd.path
+
+if fd.method.supports_srcrev():
+uri = uri + "@" + fd.revisions[name]
+
+return uri
--
2.46.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#205032):
https://lists.openembedded.org/g/openembedded-core/message/205032
Mute This Topic: https://lists.openembedded.org/mt/108688990/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 1/4] spdx 3.0: Find local sources when searching for debug sources
Include the local files when searching for matching debug sources
Signed-off-by: Joshua Watt
---
meta/lib/oe/spdx30_tasks.py | 59 -
1 file changed, 32 insertions(+), 27 deletions(-)
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 6a2858c6654..70d1bc7e8ae 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -119,7 +119,9 @@ def add_license_expression(d, objset, license_expression,
license_data):
)
spdx_license_expression = " ".join(convert(l) for l in lic_split)
-return objset.new_license_expression(spdx_license_expression,
license_data, license_text_map)
+return objset.new_license_expression(
+spdx_license_expression, license_data, license_text_map
+)
def add_package_files(
@@ -202,6 +204,7 @@ def get_package_sources_from_debug(
return False
debug_search_paths = [
+Path(d.getVar("SPDXWORK")),
Path(d.getVar("PKGD")),
Path(d.getVar("STAGING_DIR_TARGET")),
Path(d.getVar("STAGING_DIR_NATIVE")),
@@ -286,8 +289,24 @@ def collect_dep_objsets(d, build):
return dep_objsets, dep_builds
-def collect_dep_sources(dep_objsets):
-sources = {}
+def index_sources_by_hash(sources, dest):
+for s in sources:
+if not isinstance(s, oe.spdx30.software_File):
+continue
+
+if s.software_primaryPurpose !=
oe.spdx30.software_SoftwarePurpose.source:
+continue
+
+for v in s.verifiedUsing:
+if v.algorithm == oe.spdx30.HashAlgorithm.sha256:
+if not v.hashValue in dest:
+dest[v.hashValue] = s
+break
+else:
+bb.fatal(f"No SHA256 found for {s.name}")
+
+
+def collect_dep_sources(dep_objsets, dest):
for objset in dep_objsets:
# Don't collect sources from native recipes as they
# match non-native sources also.
@@ -307,26 +326,7 @@ def collect_dep_sources(dep_objsets):
if e.relationshipType != oe.spdx30.RelationshipType.hasInputs:
continue
-for to in e.to:
-if not isinstance(to, oe.spdx30.software_File):
-continue
-
-if (
-to.software_primaryPurpose
-!= oe.spdx30.software_SoftwarePurpose.source
-):
-continue
-
-for v in to.verifiedUsing:
-if v.algorithm == oe.spdx30.HashAlgorithm.sha256:
-sources[v.hashValue] = to
-break
-else:
-bb.fatal(
-"No SHA256 found for %s in %s" % (to.name,
objset.doc.name)
-)
-
-return sources
+index_sources_by_hash(e.to, dest)
def add_download_files(d, objset):
@@ -511,18 +511,21 @@ def create_spdx(d):
source_files = add_download_files(d, build_objset)
build_inputs |= source_files
-recipe_spdx_license = add_license_expression(d, build_objset,
d.getVar("LICENSE"), license_data)
+recipe_spdx_license = add_license_expression(
+d, build_objset, d.getVar("LICENSE"), license_data
+)
build_objset.new_relationship(
source_files,
oe.spdx30.RelationshipType.hasConcludedLicense,
[recipe_spdx_license],
)
+dep_sources = {}
if oe.spdx_common.process_sources(d) and include_sources:
bb.debug(1, "Adding source files to SPDX")
oe.spdx_common.get_patched_src(d)
-build_inputs |= add_package_files(
+files = add_package_files(
d,
build_objset,
spdx_workdir,
@@ -535,6 +538,8 @@ def create_spdx(d):
ignore_top_level_dirs=["temp"],
archive=None,
)
+build_inputs |= files
+index_sources_by_hash(files, dep_sources)
dep_objsets, dep_builds = collect_dep_objsets(d, build)
if dep_builds:
@@ -555,7 +560,7 @@ def create_spdx(d):
# TODO: Handle native recipe output
if not is_native:
bb.debug(1, "Collecting Dependency sources files")
-sources = collect_dep_sources(dep_objsets)
+collect_dep_sources(dep_objsets, dep_sources)
bb.build.exec_func("read_subpackage_metadata", d)
@@ -726,7 +731,7 @@ def create_spdx(d):
if include_sources:
debug_sources = get_package_sources_from_debug(
-d, package, package_files, sources, source_hash_cache
+d, package, package_files, dep_sources, source_hash_cache
)
debug_source_ids |= set(
oe.sbom30.get_element_link_id(d) for d in debug_sources
--
2.46.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all
Re: [OE-core] [PATCH v5 1/2] binutils: Fix packaging of bfd-plugins
In general, we don't really support any nativesdk- recipes in MinGW;
it's pretty difficult to get recipes working for MinGW in general, so
we are pretty focused on just providing the cross-compiler utilities
so that the SDK can be used to compile for your target device, and
less so to provide utilities for running on Windows proper. A lot of
recipes are pretty much impossible to compile as nativesdk- (e.g.
Python).
In general, my recommendation would be to use the MinGW SDK to provide
the cross compiler, and if you want additional tools in your Windows
environment, use something like msys2 to provide them (or compile your
own versions). IMHO, this a much more maintainable long term solution,
since I don't think it's going to be particularly easy to allow
arbitrary recipes to be compiled as nativesdk- for MinGW, and
msys2/mingw has already solved this problem anyway.
On Thu, Sep 19, 2024 at 7:41 AM Richard Purdie
wrote:
>
> On Thu, 2024-09-19 at 05:48 -0700, Sadineni, Harish via
> lists.openembedded.org wrote:
> > From: Harish Sadineni
> >
> > Fixes the follwing error while building nativesdk-binutils with
> > i686-mingw32 as SDKMACHINE:
> > ERROR: nativesdk-binutils-2.43.1-r0 do_package: QA Issue:
> > nativesdk-binutils: Files/directories
> > were installed but not shipped in any package:
> >
> > /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-w64-mingw32/usr/lib/bfd-plugins
> >
> > /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-w64-mingw32/usr/lib/bfd-plugins/libdep.dll
> >
> > Signed-off-by: Harish Sadineni
> > ---
> > meta/recipes-devtools/binutils/binutils_2.43.1.bb | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/meta/recipes-devtools/binutils/binutils_2.43.1.bb
> > b/meta/recipes-devtools/binutils/binutils_2.43.1.bb
> > index 2cce40f1ef..eb7c16cbc4 100644
> > --- a/meta/recipes-devtools/binutils/binutils_2.43.1.bb
> > +++ b/meta/recipes-devtools/binutils/binutils_2.43.1.bb
> > @@ -72,5 +72,6 @@ SRC_URI:append:class-nativesdk = "
> > file://0003-binutils-nativesdk-Search-for-al
> >
> > USE_ALTERNATIVES_FOR:class-nativesdk = ""
> > FILES:${PN}:append:class-nativesdk = " ${bindir}"
> > +FILES:${PN}:append:class-nativesdk = " ${libdir}/bfd-plugins"
> >
> > BBCLASSEXTEND = "native nativesdk"
>
> I ended up digging into this a lot more. I was able to reproduce the
> failure locally but that implies the autobuilder isn't testing this
> combination. According to the configuration, it should be so I'm a bit
> puzzled why the autobuilder doesn't see it.
>
> The autobuilder config for this is here:
>
> https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json#n1362
>
> Resulting in a build such as this:
>
> https://valkyrie.yoctoproject.org/#/builders/7/builds/162
>
> I would like to understand why automated testing isn't catching this.
>
> Getting back to the patch, I noticed that bintutils.inc already does:
>
> FILES:${PN} = " \
> ${bindir}/${TARGET_PREFIX}* \
> ${libdir}/lib*.so.* \
> ${libdir}/bfd-plugins/lib*.so \
> ...
>
> and the issue is that it uses .so and the recipe is generating a .dll
> file.
>
> I'd note that bitbake.conf has:
>
> SOLIBS = ".so.*"
> SOLIBS:darwin = ".dylib"
>
> SOLIBSDEV = ".so"
> # Due to the ordering of PACKAGES and the naming of the dev symlinks on
> darwin,
> # we can't make the symlinks end up in the -dev packages easily at this
> point. This hack
> # at least means builds aren't completely broken and symlinks don't take up
> much space.
> SOLIBSDEV:darwin = ".dylibbroken"
>
> with the intent of being able to abstract out dylib/so/dll references.
> I'm wondering if we should add:
>
> SOLIBS:mingw32 = ".dll"
> SOLIBSDEV:mingw32 = ".dll"
>
> which would then let us change the binutils.inc entry to:
>
> ${libdir}/bfd-plugins/lib*${SOLIBSDEV}
>
> In testing this I found it breaks zstd and zlib since those always
> generate .so files even on mingw. I've hardcoded SOLIBS and SOLIBSDEV
> in the bbappends in mingw to counteract that:
>
> https://git.yoctoproject.org/meta-mingw/commit/?id=db1f53f73e95499a5f0b0edc45dd9cc9718d3c8a
>
> I think these changes should combine to fix the underlying problem and
> also make it easier to fix things like this in future.
>
> They probably need wider testing though.
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#204705):
https://lists.openembedded.org/g/openembedded-core/message/204705
Mute This Topic: https://lists.openembedded.org/mt/108538762/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] lib/oe/sbom30.py: Fix build parameters
The property to specify the build parameters is `build_parameters` not just `parameters` Signed-off-by: Joshua Watt --- meta/lib/oe/sbom30.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 2cea56ac3e6..76bfb752ef1 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -782,7 +782,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): if not isinstance(value, str): continue -build.parameters.append( +build.build_parameters.append( oe.spdx30.DictionaryEntry(key=varname, value=value) ) -- 2.46.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#204172): https://lists.openembedded.org/g/openembedded-core/message/204172 Mute This Topic: https://lists.openembedded.org/mt/108247424/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] classes/create-spdx-image-3.0: Fix SSTATE_SKIP_CREATION
Fixes SSTATE_SKIP_CREATION for do_create_image_spdx
Signed-off-by: Joshua Watt
---
meta/classes-recipe/create-spdx-image-3.0.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass
b/meta/classes-recipe/create-spdx-image-3.0.bbclass
index fcae134d66e..18e6cf6dfa2 100644
--- a/meta/classes-recipe/create-spdx-image-3.0.bbclass
+++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass
@@ -48,7 +48,7 @@ python do_create_image_spdx() {
}
addtask do_create_image_spdx after do_image_complete do_create_rootfs_spdx
before do_build
SSTATETASKS += "do_create_image_spdx"
-SSTATE_SKIP_CREATION:task-combine-image-type-spdx = "1"
+SSTATE_SKIP_CREATION:task-create-image-spdx = "1"
do_create_image_spdx[sstate-inputdirs] = "${SPDXIMAGEWORK}"
do_create_image_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
do_create_image_spdx[cleandirs] = "${SPDXIMAGEWORK}"
--
2.46.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#204170):
https://lists.openembedded.org/g/openembedded-core/message/204170
Mute This Topic: https://lists.openembedded.org/mt/108247403/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] lib/spdx30_tasks: Report all missing providers
Instead of failing on the first missing provider, collect all of them
and report them all as it is more convenient for end users trying to fix
problems
Signed-off-by: Joshua Watt
---
meta/lib/oe/spdx30_tasks.py | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 4864d6252a6..4da52da654d 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -953,10 +953,12 @@ def collect_build_package_inputs(d, objset, build,
packages):
providers = oe.spdx_common.collect_package_providers(d)
build_deps = set()
+missing_providers = set()
for name in sorted(packages.keys()):
if name not in providers:
-bb.fatal("Unable to find SPDX provider for '%s'" % name)
+missing_providers.add(name)
+continue
pkg_name, pkg_hashfn = providers[name]
@@ -970,6 +972,11 @@ def collect_build_package_inputs(d, objset, build,
packages):
)
build_deps.add(pkg_spdx._id)
+if missing_providers:
+bb.fatal(
+f"Unable to find SPDX provider(s) for: {',
'.join(sorted(missing_providers))}"
+)
+
if build_deps:
objset.new_scoped_relationship(
[build],
--
2.46.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#204171):
https://lists.openembedded.org/g/openembedded-core/message/204171
Mute This Topic: https://lists.openembedded.org/mt/108247411/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] rpcbind: Fix boot time start failure
On Fri, Jul 26, 2024 at 9:55 AM Böszörményi Zoltán wrote:
>
> 2024. 07. 26. 17:21 keltezéssel, Joshua Watt írta:
> > On Fri, Jul 26, 2024 at 5:24 AM Zoltán Böszörményi
> > wrote:
> >> With commits 90bc1810 ("bitbake.conf: Add runtimedir") and
> >> 561e853e ("rpcbind: Specify state directory under /run") rpcbind
> >> still can fail during startup.
> >>
> >> It has two problems:
> >> * The lockfile is still hardcoded as "/var/run/rpcbind.lock".
> >>It needs to use the same internal define for RPCBIND_STATEDIR
> >>as the paths for rpcbind.xdr and portmap.xdr.
> >> * Using --with-statedir=/run/rpcbind doesn't guarantee that this
> >>directory exists when rpcbind.service starts. Add this guarantee
> >>by running rpcbind.service with After=systemd-tmpfiles-setup.service
> >>and add the tmpfiles.d entry for /run/rpcbind.
> > The original patches have been working for me on nanbield (and
> > kirkstone), so I'm a little confused as to why this is not working for
> > you; I'd like to avoid patching the source code if at all possible as
> > carrying patches like this is a burden
>
> The error I got is:
>
> rpcbind[455]: rpcbind: /var/run/rpcbind.lock: Read-only file system
>
> So, while /var/run is a symlink to /run, the latter is not writable yet.
> Indeed, rpcbind.service was starting very early. These changes
> delay it enough to avoid the problem.
>
> The inconsistency between using /run via --with-statedir=/run
> directly for some the xdr files but not for the lockfile is not nice.
> That must have been an oversight in the source code.
Yes it is, however I suspect this patch isn't _strictly_ necessary to
fix the problem as long as the service file change is done (using a
drop file). That said, I'd rather not carry a patch unnecessarily and
we should attempt to fix it upstream instead of carry the patch.
The systemd service change should also be upstreamed, but thankfully
we can fix it with a drop file instead of a patch so I'm fine with
doing that.
>
> These changes fix the issue for me in nanbield. I can reproduce
> the above problem in scarthgap without these changes.
>
> FWIW, different machines can give very different testing results,
> due to the differences in their number of CPU cores and speed.
> Recently, I had to add a modified rngd.service from dracut
> to rng-tools as an in-house change, because one particular
> machine (slow by today's standards, based on AMD Kabini)
> ended up with this service failing, due to the CPU not supporting
> RDRAND and the JITTER random source driver starting particularly
> slowly on it, so rngd couldn't pick it up when started only from
> initramfs.
>
> >> Signed-off-by: Zoltán Böszörményi
> >> ---
> >> ...ke-the-lockfile-follow-with-statedir.patch | 31 ++
> >> ...ice-after-etc-tmpfiles.d-is-processe.patch | 32 +++
> >> .../rpcbind/rpcbind/rpcbind.tmpfiles | 1 +
> >> .../recipes-extended/rpcbind/rpcbind_1.2.6.bb | 6
> >> 4 files changed, 70 insertions(+)
> >> create mode 100644
> >> meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
> >> create mode 100644
> >> meta/recipes-extended/rpcbind/rpcbind/0002-Run-rpcbind.service-after-etc-tmpfiles.d-is-processe.patch
> >> create mode 100644 meta/recipes-extended/rpcbind/rpcbind/rpcbind.tmpfiles
> >>
> >> diff --git
> >> a/meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
> >>
> >> b/meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
> >> new file mode 100644
> >> index 00..d487312d22
> >> --- /dev/null
> >> +++
> >> b/meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
> >> @@ -0,0 +1,31 @@
> >> +From f10db88174e73c78c028561715f16ed38148ebde Mon Sep 17 00:00:00 2001
> >> +From: =?UTF-8?q?Zolt=C3=A1n=20B=C3=B6sz=C3=B6rm=C3=A9nyi?=
> >> +
> >> +Date: Fri, 26 Jul 2024 12:36:06 +0200
> >> +Subject: [PATCH 1/2] Make the lockfile follow --with-statedir=
> >> +MIME-Version: 1.0
> >> +Content-Type: text/plain; charset=UTF-8
> >> +Content-Transfer-Encoding: 8bit
> >> +
> >> +Upstream-Status: Inactive-Upstream [lastrelease: 2021-05-10]
> >> +Signed-off-by: Zoltán Böszörményi
> >> +---
> >> + src/rpcbind.c | 2 +-
> >> + 1 file changed, 1 insertion(+), 1 delet
Re: [OE-core] [PATCH] rpcbind: Fix boot time start failure
On Fri, Jul 26, 2024 at 5:24 AM Zoltán Böszörményi wrote:
>
> With commits 90bc1810 ("bitbake.conf: Add runtimedir") and
> 561e853e ("rpcbind: Specify state directory under /run") rpcbind
> still can fail during startup.
>
> It has two problems:
> * The lockfile is still hardcoded as "/var/run/rpcbind.lock".
> It needs to use the same internal define for RPCBIND_STATEDIR
> as the paths for rpcbind.xdr and portmap.xdr.
> * Using --with-statedir=/run/rpcbind doesn't guarantee that this
> directory exists when rpcbind.service starts. Add this guarantee
> by running rpcbind.service with After=systemd-tmpfiles-setup.service
> and add the tmpfiles.d entry for /run/rpcbind.
The original patches have been working for me on nanbield (and
kirkstone), so I'm a little confused as to why this is not working for
you; I'd like to avoid patching the source code if at all possible as
carrying patches like this is a burden
>
> Signed-off-by: Zoltán Böszörményi
> ---
> ...ke-the-lockfile-follow-with-statedir.patch | 31 ++
> ...ice-after-etc-tmpfiles.d-is-processe.patch | 32 +++
> .../rpcbind/rpcbind/rpcbind.tmpfiles | 1 +
> .../recipes-extended/rpcbind/rpcbind_1.2.6.bb | 6
> 4 files changed, 70 insertions(+)
> create mode 100644
> meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
> create mode 100644
> meta/recipes-extended/rpcbind/rpcbind/0002-Run-rpcbind.service-after-etc-tmpfiles.d-is-processe.patch
> create mode 100644 meta/recipes-extended/rpcbind/rpcbind/rpcbind.tmpfiles
>
> diff --git
> a/meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
>
> b/meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
> new file mode 100644
> index 00..d487312d22
> --- /dev/null
> +++
> b/meta/recipes-extended/rpcbind/rpcbind/0001-Make-the-lockfile-follow-with-statedir.patch
> @@ -0,0 +1,31 @@
> +From f10db88174e73c78c028561715f16ed38148ebde Mon Sep 17 00:00:00 2001
> +From: =?UTF-8?q?Zolt=C3=A1n=20B=C3=B6sz=C3=B6rm=C3=A9nyi?=
> +
> +Date: Fri, 26 Jul 2024 12:36:06 +0200
> +Subject: [PATCH 1/2] Make the lockfile follow --with-statedir=
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Upstream-Status: Inactive-Upstream [lastrelease: 2021-05-10]
> +Signed-off-by: Zoltán Böszörményi
> +---
> + src/rpcbind.c | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +diff --git a/src/rpcbind.c b/src/rpcbind.c
> +index 36a95b9..948a284 100644
> +--- a/src/rpcbind.c
> b/src/rpcbind.c
> +@@ -105,7 +105,7 @@ char *nss_modules = "files";
> + /* who to suid to if -s is given */
> + #define RUN_AS "daemon"
> +
> +-#define RPCBINDDLOCK "/var/run/rpcbind.lock"
> ++#define RPCBINDDLOCK RPCBIND_STATEDIR "/rpcbind.lock"
/var/run should be a symlink to /run, so I really don't think this
should be necessary (at least with the default value for $runtimedir)
> +
> + int runasdaemon = 0;
> + int insecure = 0;
> +--
> +2.45.2
> +
> diff --git
> a/meta/recipes-extended/rpcbind/rpcbind/0002-Run-rpcbind.service-after-etc-tmpfiles.d-is-processe.patch
>
> b/meta/recipes-extended/rpcbind/rpcbind/0002-Run-rpcbind.service-after-etc-tmpfiles.d-is-processe.patch
> new file mode 100644
> index 00..f454fdbb3b
> --- /dev/null
> +++
> b/meta/recipes-extended/rpcbind/rpcbind/0002-Run-rpcbind.service-after-etc-tmpfiles.d-is-processe.patch
> @@ -0,0 +1,32 @@
> +From 408b18c77c254baaa9111b3e7031ebf12149db38 Mon Sep 17 00:00:00 2001
> +From: =?UTF-8?q?Zolt=C3=A1n=20B=C3=B6sz=C3=B6rm=C3=A9nyi?=
> +
> +Date: Fri, 26 Jul 2024 12:54:38 +0200
> +Subject: [PATCH 2/2] Run rpcbind.service after /etc/tmpfiles.d is processed
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Upstream-Status: Inactive-Upstream [lastrelease: 2021-05-10]
> +Signed-off-by: Zoltán Böszörményi
> +---
> + systemd/rpcbind.service.in | 3 +++
> + 1 file changed, 3 insertions(+)
> +
> +diff --git a/systemd/rpcbind.service.in b/systemd/rpcbind.service.in
> +index f45ee1e..e9d94ce 100644
> +--- a/systemd/rpcbind.service.in
> b/systemd/rpcbind.service.in
> +@@ -9,6 +9,9 @@ RequiresMountsFor=@statedir@
> + Requires=rpcbind.socket
> + Wants=rpcbind.target
> +
> ++# Make sure the runtime directory exists
> ++After=systemd-tmpfiles-setup.service
I'd rather have the recipe provide a drop file for this instead of
patching the source
> ++
> + [Service]
> + Type=notify
> + EnvironmentFile=-@_sysconfdir@/rpcbind.conf
> +--
> +2.45.2
> +
> diff --git a/meta/recipes-extended/rpcbind/rpcbind/rpcbind.tmpfiles
> b/meta/recipes-extended/rpcbind/rpcbind/rpcbind.tmpfiles
> new file mode 100644
> index 00..fecee72c09
> --- /dev/null
> +++ b/meta/recipes-extended/rpcbind/rpcbind/rpcbind.tmpfiles
> @@ -0,0 +1 @@
> +d /run/rpcbind 0755 root root -
> diff --git a/meta/recipes-extended/rpcbind/rpcbi
Re: [OE-core][PATCH v6 00/12] Add SPDX 3.0 support
On Tue, Jul 16, 2024, 7:18 AM Richard Purdie < richard.pur...@linuxfoundation.org> wrote: > On Tue, 2024-07-16 at 00:00 +0100, Richard Purdie via > lists.openembedded.org wrote: > > On Mon, 2024-07-15 at 15:26 -0600, Joshua Watt wrote: > > > On Mon, Jul 15, 2024 at 3:07 PM Richard Purdie > > > wrote: > > > > > > > > On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > > > > > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > > > > > wrote: > > > > > > > > > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > > > > > lists.openembedded.org wrote: > > > > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > > > > default. > > > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 > > > > > > > enabled at > > > > > > > the same time > > > > > > > > > > > > > > v2: Added tests and addressed feedback > > > > > > > v3: Fixed several oe-selftest and build failures > > > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > > > v5: Reworked to make SPDX 3 output reproducible by default. > > > > > > > Variables > > > > > > > that introduce non-reproducible output are documented as > > > > > > > such. > > > > > > > v6: Many changes: > > > > > > > * Fixed bug where building baremetal images would break > > > > > > > SPDX > > > > > > > 2.2 > > > > > > > * Most SPDX code is now in python library files instead of > > > > > > > tasks > > > > > > > * Removed dependency on pacakge_write_* tasks > > > > > > > * Fixed sstate selftest cases to account for SPDX 3.0 task > > > > > > > names > > > > > > > > > > > > This had a lot of failures in testing I'm afraid: > > > > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > > > > > > > > > This appears to be caused because the SPDX tasks are not running > > > > > for > > > > > some recipes (e.g. automake). It looks like this like: > > > > > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > > > do_create_package_spdx" > > > > > > > > > > is not actually strong enough to make sure the SPDX tasks for > > > > > automake > > > > > run for e.g. core-image-sato-sdk, but I don't know why. I'll keep > > > > > looking, but if anyone happens to know off the top of their head > > > > > let > > > > > me know > > > > > > > > Can you be specific about which tasks you mean when you say "make > > > > sure > > > > the SPDX tasks for automake run"? Do you mean do_create_spdx, > > > > do_create_package_spdx or a different one? > > > > > > Specifically, do_create_package_spdx must be run for each package > > > installed in the rootfs before do_create_rootfs_spdx runs. I thought > > > that > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > do_create_package_spdx" > > > > > > would do this (the do_create_spdx is probably not necessary), since > > > AFIACT, this is also how the packages get generated before being > > > installed in the root file system via manipulation of > > > do_rootfs[recrdeptask], but I think I'm missing something? > > > > > > In packagegroups.bbclass there is this being added: > > > > do_create_package_spdx[deptask] = "do_create_spdx" > > do_create_package_spdx[rdeptask] = "" > > > > and > > > > bitbake core-image-sato-sdk -g -c create_rootfs_spdx > > > > lists > > > > "core-image-sato-sdk.do_create_rootfs_spdx" -> > "automake.do_create_package_spdx" > > > > in tasks-depends.dot if I disable it. > > > > I'm not 100% sure what is going on and should sleep but wanted to share > > that before I did. > > I deleted those two lines and ran a build which passed testing. > > I don't think we need them with spdx 3 since it doesn't have the "hash > changing" issue that we had that caused us to add that for spdx2? > Good catch, I missed that was related to packagegroups. No, we don't need that for SPDX 3 (still need it for 2.2 though) > The commit was: > > https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44 > > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#202113): https://lists.openembedded.org/g/openembedded-core/message/202113 Mute This Topic: https://lists.openembedded.org/mt/107185009/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] classes-recipe/multilib_script: Expand before splitting
multilib_script.bbclass was unable to work correctly in the case where
e.g. a PACKAGECONFIG removed the script that it was intended to rename
(as an example, the "trace" PACKAGECONFIG in cairo). The way to fix this
would be to do something like:
MULTILIB_SCRIPTS = "${@bb.utils.contains('PACKAGECONFIG', 'trace',
'${PN}-perf-utils:${bindir}/cairo-trace', '', d)}"
but this is not possible because the variable is not expanded before
being split.
To fix this, change the class to expand the variable before splitting.
There are two cases to be considered that could possibly break:
1) If the RHS of the ":" contains a ":", which is accounted for by
limiting the splitting to 1 split, which will leave the ":" in the
RHS in tact. Of note, this works because ":" isn't valid in a
package name
2) If the RHS of the ":" contained whitespace, however this would have
broken the mv command written to multilibscript_rename(), so this
isn't occurring in practice.
Signed-off-by: Joshua Watt
---
meta/classes-recipe/multilib_script.bbclass | 18 --
1 file changed, 8 insertions(+), 10 deletions(-)
diff --git a/meta/classes-recipe/multilib_script.bbclass
b/meta/classes-recipe/multilib_script.bbclass
index e6f0249529c..a7a08930b7e 100644
--- a/meta/classes-recipe/multilib_script.bbclass
+++ b/meta/classes-recipe/multilib_script.bbclass
@@ -28,14 +28,12 @@ python () {
if bb.data.inherits_class('native', d) or bb.data.inherits_class('cross',
d):
return
-for entry in (d.getVar("MULTILIB_SCRIPTS", False) or "").split():
-pkg, script = entry.split(":")
-epkg = d.expand(pkg)
-escript = d.expand(script)
-scriptname = os.path.basename(escript)
-d.appendVar("ALTERNATIVE:" + epkg, " " + scriptname + " ")
-d.setVarFlag("ALTERNATIVE_LINK_NAME", scriptname, escript)
-d.setVarFlag("ALTERNATIVE_TARGET", scriptname, escript +
"-${MULTILIB_SUFFIX}")
-d.appendVar("multilibscript_rename", "\n mv ${PKGD}" + escript +
" ${PKGD}" + escript + "-${MULTILIB_SUFFIX}")
-d.appendVar("FILES:" + epkg, " " + escript + "-${MULTILIB_SUFFIX}")
+for entry in (d.getVar("MULTILIB_SCRIPTS") or "").split():
+pkg, script = entry.split(":", 1)
+scriptname = os.path.basename(script)
+d.appendVar("ALTERNATIVE:" + pkg, " " + scriptname + " ")
+d.setVarFlag("ALTERNATIVE_LINK_NAME", scriptname, script)
+d.setVarFlag("ALTERNATIVE_TARGET", scriptname, script +
"-${MULTILIB_SUFFIX}")
+d.appendVar("multilibscript_rename", "\n mv ${PKGD}" + script +
" ${PKGD}" + script + "-${MULTILIB_SUFFIX}")
+d.appendVar("FILES:" + pkg, " " + script + "-${MULTILIB_SUFFIX}")
}
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#202072):
https://lists.openembedded.org/g/openembedded-core/message/202072
Mute This Topic: https://lists.openembedded.org/mt/107241398/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH v6 00/12] Add SPDX 3.0 support
On Mon, Jul 15, 2024 at 3:07 PM Richard Purdie wrote: > > On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > > wrote: > > > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > > lists.openembedded.org wrote: > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > default. > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 > > > > enabled at > > > > the same time > > > > > > > > v2: Added tests and addressed feedback > > > > v3: Fixed several oe-selftest and build failures > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > v5: Reworked to make SPDX 3 output reproducible by default. > > > > Variables > > > > that introduce non-reproducible output are documented as > > > > such. > > > > v6: Many changes: > > > > * Fixed bug where building baremetal images would break SPDX > > > > 2.2 > > > > * Most SPDX code is now in python library files instead of > > > > tasks > > > > * Removed dependency on pacakge_write_* tasks > > > > * Fixed sstate selftest cases to account for SPDX 3.0 task > > > > names > > > > > > This had a lot of failures in testing I'm afraid: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > > > This appears to be caused because the SPDX tasks are not running for > > some recipes (e.g. automake). It looks like this like: > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > do_create_package_spdx" > > > > is not actually strong enough to make sure the SPDX tasks for > > automake > > run for e.g. core-image-sato-sdk, but I don't know why. I'll keep > > looking, but if anyone happens to know off the top of their head let > > me know > > Can you be specific about which tasks you mean when you say "make sure > the SPDX tasks for automake run"? Do you mean do_create_spdx, > do_create_package_spdx or a different one? Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought that do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" would do this (the do_create_spdx is probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something? > > Cheers, > > Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#202071): https://lists.openembedded.org/g/openembedded-core/message/202071 Mute This Topic: https://lists.openembedded.org/mt/107185009/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH v6 00/12] Add SPDX 3.0 support
On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie wrote: > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > lists.openembedded.org wrote: > > This patch series add support for SPDX 3.0 and sets it as the > > default. > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > the same time > > > > v2: Added tests and addressed feedback > > v3: Fixed several oe-selftest and build failures > > v4: Fixed silly typo mistake in staging.bbclass > > v5: Reworked to make SPDX 3 output reproducible by default. Variables > > that introduce non-reproducible output are documented as such. > > v6: Many changes: > > * Fixed bug where building baremetal images would break SPDX 2.2 > > * Most SPDX code is now in python library files instead of tasks > > * Removed dependency on pacakge_write_* tasks > > * Fixed sstate selftest cases to account for SPDX 3.0 task names > > This had a lot of failures in testing I'm afraid: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like: do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" is not actually strong enough to make sure the SPDX tasks for automake run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know > > Cheers, > > Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#202069): https://lists.openembedded.org/g/openembedded-core/message/202069 Mute This Topic: https://lists.openembedded.org/mt/107185009/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v6 12/12] Switch default spdx version to 3.0
Changes the default SPDX version to 3.0 Signed-off-by: Joshua Watt --- meta/classes/create-spdx.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/create-spdx.bbclass b/meta/classes/create-spdx.bbclass index 19c6c0ff0b9..b604973ae0a 100644 --- a/meta/classes/create-spdx.bbclass +++ b/meta/classes/create-spdx.bbclass @@ -5,4 +5,4 @@ # # Include this class when you don't care what version of SPDX you get; it will # be updated to the latest stable version that is supported -inherit create-spdx-2.2 +inherit create-spdx-3.0 -- 2.45.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201846): https://lists.openembedded.org/g/openembedded-core/message/201846 Mute This Topic: https://lists.openembedded.org/mt/107185008/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v6 11/12] classes/create-spdx-2.2: Handle empty packages
When combining an SPDX document, the package list might be empty (e.g.
a baremetal image). Handle this case instead of erroring out
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-2.2.bbclass | 83 ++--
1 file changed, 42 insertions(+), 41 deletions(-)
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index 0382e4cc51a..865323d66a6 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -822,52 +822,53 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir,
rootfs_spdxid, packages, spdx
doc.packages.append(image)
-for name in sorted(packages.keys()):
-if name not in providers:
-bb.fatal("Unable to find SPDX provider for '%s'" % name)
+if packages:
+for name in sorted(packages.keys()):
+if name not in providers:
+bb.fatal("Unable to find SPDX provider for '%s'" % name)
-pkg_name, pkg_hashfn = providers[name]
+pkg_name, pkg_hashfn = providers[name]
-pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx,
package_archs, pkg_name, pkg_hashfn)
-if not pkg_spdx_path:
-bb.fatal("No SPDX file found for package %s, %s" % (pkg_name,
pkg_hashfn))
+pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx,
package_archs, pkg_name, pkg_hashfn)
+if not pkg_spdx_path:
+bb.fatal("No SPDX file found for package %s, %s" % (pkg_name,
pkg_hashfn))
-pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
+pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
-for p in pkg_doc.packages:
-if p.name == name:
-pkg_ref = oe.spdx.SPDXExternalDocumentRef()
-pkg_ref.externalDocumentId = "DocumentRef-%s" % pkg_doc.name
-pkg_ref.spdxDocument = pkg_doc.documentNamespace
-pkg_ref.checksum.algorithm = "SHA1"
-pkg_ref.checksum.checksumValue = pkg_doc_sha1
+for p in pkg_doc.packages:
+if p.name == name:
+pkg_ref = oe.spdx.SPDXExternalDocumentRef()
+pkg_ref.externalDocumentId = "DocumentRef-%s" %
pkg_doc.name
+pkg_ref.spdxDocument = pkg_doc.documentNamespace
+pkg_ref.checksum.algorithm = "SHA1"
+pkg_ref.checksum.checksumValue = pkg_doc_sha1
-doc.externalDocumentRefs.append(pkg_ref)
-doc.add_relationship(image, "CONTAINS", "%s:%s" %
(pkg_ref.externalDocumentId, p.SPDXID))
-break
-else:
-bb.fatal("Unable to find package with name '%s' in SPDX file %s" %
(name, pkg_spdx_path))
-
-runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx,
package_archs, "runtime-" + name, pkg_hashfn)
-if not runtime_spdx_path:
-bb.fatal("No runtime SPDX document found for %s, %s" % (name,
pkg_hashfn))
-
-runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path)
-
-runtime_ref = oe.spdx.SPDXExternalDocumentRef()
-runtime_ref.externalDocumentId = "DocumentRef-%s" % runtime_doc.name
-runtime_ref.spdxDocument = runtime_doc.documentNamespace
-runtime_ref.checksum.algorithm = "SHA1"
-runtime_ref.checksum.checksumValue = runtime_doc_sha1
-
-# "OTHER" isn't ideal here, but I can't find a relationship that makes
sense
-doc.externalDocumentRefs.append(runtime_ref)
-doc.add_relationship(
-image,
-"OTHER",
-"%s:%s" % (runtime_ref.externalDocumentId, runtime_doc.SPDXID),
-comment="Runtime dependencies for %s" % name
-)
+doc.externalDocumentRefs.append(pkg_ref)
+doc.add_relationship(image, "CONTAINS", "%s:%s" %
(pkg_ref.externalDocumentId, p.SPDXID))
+break
+else:
+bb.fatal("Unable to find package with name '%s' in SPDX file
%s" % (name, pkg_spdx_path))
+
+runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx,
package_archs, "runtime-" + name, pkg_hashfn)
+if not runtime_spdx_path:
+bb.fatal("No runtime SPDX document found for %s, %s" % (name,
pkg_hashfn))
+
+runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path)
+
+runtime_ref = oe.spdx.SPDXExternalDocumentRef()
+runtime_ref.externalDocumentId = "DocumentRef-%s" %
runtime_doc.name
+runtime_ref.spdxDocument = runtime_doc.documentNamespace
+
[OE-core][PATCH v6 10/12] classes/create-spdx-3.0: Move tasks to library
Move the bulk of the python code in the SPDX 3.0 classes into a library
file
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-3.0.bbclass | 874 +-
meta/classes/create-spdx-image-3.0.bbclass | 307 +
meta/lib/oe/spdx30_tasks.py| 1229
3 files changed, 1256 insertions(+), 1154 deletions(-)
create mode 100644 meta/lib/oe/spdx30_tasks.py
diff --git a/meta/classes/create-spdx-3.0.bbclass
b/meta/classes/create-spdx-3.0.bbclass
index a930ea81152..41840d9d1a3 100644
--- a/meta/classes/create-spdx-3.0.bbclass
+++ b/meta/classes/create-spdx-3.0.bbclass
@@ -116,698 +116,15 @@ SPDX_PACKAGE_SUPPLIER[doc] = "The base variable name to
describe the Agent who \
IMAGE_CLASSES:append = " create-spdx-image-3.0"
-def set_timestamp_now(d, o, prop):
-from datetime import datetime, timezone
+oe.spdx30_tasks.set_timestamp_now[vardepsexclude] = "SPDX_INCLUDE_TIMESTAMPS"
+oe.spdx30_tasks.get_package_sources_from_debug[vardepsexclude] +=
"STAGING_KERNEL_DIR"
+oe.spdx30_tasks.collect_dep_objsets[vardepsexclude] = "SSTATE_ARCHS"
-if d.getVar("SPDX_INCLUDE_TIMESTAMPS") == "1":
-setattr(o, prop, datetime.now(timezone.utc))
-else:
-# Doing this helps to validated that the property actually exists, and
-# also that it is not mandatory
-delattr(o, prop)
-
-set_timestamp_now[vardepsexclude] = "SPDX_INCLUDE_TIMESTAMPS"
-
-def add_license_expression(d, objset, license_expression):
-from pathlib import Path
-import oe.spdx30
-import oe.sbom30
-
-license_data = d.getVar("SPDX_LICENSE_DATA")
-simple_license_text = {}
-license_text_map = {}
-license_ref_idx = 0
-
-def add_license_text(name):
-nonlocal objset
-nonlocal simple_license_text
-
-if name in simple_license_text:
-return simple_license_text[name]
-
-lic = objset.find_filter(
-oe.spdx30.simplelicensing_SimpleLicensingText,
-name=name,
-)
-
-if lic is not None:
-simple_license_text[name] = lic
-return lic
-
-lic = objset.add(oe.spdx30.simplelicensing_SimpleLicensingText(
-_id=objset.new_spdxid("license-text", name),
-creationInfo=objset.doc.creationInfo,
-name=name,
-))
-simple_license_text[name] = lic
-
-if name == "PD":
-lic.simplelicensing_licenseText = "Software released to the public
domain"
-return lic
-
-# Seach for the license in COMMON_LICENSE_DIR and LICENSE_PATH
-for directory in [d.getVar('COMMON_LICENSE_DIR')] +
(d.getVar('LICENSE_PATH') or '').split():
-try:
-with (Path(directory) / name).open(errors="replace") as f:
-lic.simplelicensing_licenseText = f.read()
-return lic
-
-except FileNotFoundError:
-pass
-
-# If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set
-filename = d.getVarFlag('NO_GENERIC_LICENSE', name)
-if filename:
-filename = d.expand("${S}/" + filename)
-with open(filename, errors="replace") as f:
-lic.simplelicensing_licenseText = f.read()
-return lic
-else:
-bb.fatal("Cannot find any text for license %s" % name)
-
-def convert(l):
-nonlocal license_text_map
-nonlocal license_ref_idx
-
-if l == "(" or l == ")":
-return l
-
-if l == "&":
-return "AND"
-
-if l == "|":
-return "OR"
-
-if l == "CLOSED":
-return "NONE"
-
-spdx_license = d.getVarFlag("SPDXLICENSEMAP", l) or l
-if spdx_license in license_data["licenses"]:
-return spdx_license
-
-spdx_license = "LicenseRef-" + l
-if spdx_license not in license_text_map:
-license_text_map[spdx_license] = add_license_text(l)._id
-
-return spdx_license
-
-lic_split = license_expression.replace("(", " ( ").replace(")", " )
").replace("|", " | ").replace("&", " & ").split()
-spdx_license_expression = ' '.join(convert(l) for l in lic_split)
-
-return objset.new_license_expression(spdx_license_expression,
license_text_map)
-
-
-def add_package_files(d, objset, topdir, get_spdxid, get_purposes, *,
archive=None, ignore_dirs=[], ignore_top_level_dirs=[]):
-from pathlib import Path
-import oe.spdx30
-import oe.sbom30
-
-source_date_epoch = d.getVar("SOU
[OE-core][PATCH v6 00/12] Add SPDX 3.0 support
This patch series add support for SPDX 3.0 and sets it as the default. Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time v2: Added tests and addressed feedback v3: Fixed several oe-selftest and build failures v4: Fixed silly typo mistake in staging.bbclass v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such. v6: Many changes: * Fixed bug where building baremetal images would break SPDX 2.2 * Most SPDX code is now in python library files instead of tasks * Removed dependency on pacakge_write_* tasks * Fixed sstate selftest cases to account for SPDX 3.0 task names Joshua Watt (12): classes-recipe/image: Add image file manifest classes-recipe/baremetal-image: Add image file manifest classes/create-spdx-3.0: Add classes classes-global/staging: Exclude do_create_spdx from automatic sysroot extension classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images selftest: spdx: Add SPDX 3.0 test cases classes-recipe: nospdx: Add class selftest: sstatetests: Exclude all SPDX tasks classes/spdx-common: Move to library classes/create-spdx-3.0: Move tasks to library classes/create-spdx-2.2: Handle empty packages Switch default spdx version to 3.0 meta/classes-global/staging.bbclass |9 +- meta/classes-recipe/baremetal-image.bbclass | 32 +- meta/classes-recipe/image.bbclass| 58 + meta/classes-recipe/image_types.bbclass |2 + meta/classes-recipe/image_types_wic.bbclass |1 + meta/classes-recipe/nospdx.bbclass | 13 + meta/classes-recipe/packagegroup.bbclass |2 + meta/classes/create-spdx-2.2.bbclass | 106 +- meta/classes/create-spdx-3.0.bbclass | 191 + meta/classes/create-spdx-image-3.0.bbclass | 145 + meta/classes/create-spdx.bbclass |2 +- meta/classes/spdx-common.bbclass | 195 +- meta/lib/oe/sbom30.py| 1121 meta/lib/oe/spdx30.py| 6020 ++ meta/lib/oe/spdx30_tasks.py | 1229 meta/lib/oe/spdx_common.py | 228 + meta/lib/oeqa/selftest/cases/spdx.py | 133 +- meta/lib/oeqa/selftest/cases/sstatetests.py |3 +- meta/recipes-core/meta/build-sysroots.bb |5 +- meta/recipes-core/meta/meta-world-pkgdata.bb |3 +- 20 files changed, 9243 insertions(+), 255 deletions(-) create mode 100644 meta/classes-recipe/nospdx.bbclass create mode 100644 meta/classes/create-spdx-3.0.bbclass create mode 100644 meta/classes/create-spdx-image-3.0.bbclass create mode 100644 meta/lib/oe/sbom30.py create mode 100644 meta/lib/oe/spdx30.py create mode 100644 meta/lib/oe/spdx30_tasks.py create mode 100644 meta/lib/oe/spdx_common.py -- 2.45.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201847): https://lists.openembedded.org/g/openembedded-core/message/201847 Mute This Topic: https://lists.openembedded.org/mt/107185009/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v6 09/12] classes/spdx-common: Move to library
Moves the bulk of the code in the spdx-common bbclass into library code
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-2.2.bbclass | 23 ++-
meta/classes/create-spdx-3.0.bbclass | 22 +-
meta/classes/create-spdx-image-3.0.bbclass | 3 +-
meta/classes/spdx-common.bbclass | 197 +-
meta/lib/oe/sbom30.py | 21 +-
meta/lib/oe/spdx_common.py | 228 +
6 files changed, 270 insertions(+), 224 deletions(-)
create mode 100644 meta/lib/oe/spdx_common.py
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index 3bcde1acc84..0382e4cc51a 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -38,6 +38,12 @@ def recipe_spdx_is_native(d, recipe):
a.annotator == "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"),
d.getVar("SPDX_TOOL_VERSION")) and
a.comment == "isNative" for a in recipe.annotations)
+def get_json_indent(d):
+if d.getVar("SPDX_PRETTY") == "1":
+return 2
+return None
+
+
def convert_license_to_spdx(lic, document, d, existing={}):
from pathlib import Path
import oe.spdx
@@ -113,6 +119,7 @@ def convert_license_to_spdx(lic, document, d, existing={}):
def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *,
archive=None, ignore_dirs=[], ignore_top_level_dirs=[]):
from pathlib import Path
import oe.spdx
+import oe.spdx_common
import hashlib
source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
@@ -165,7 +172,7 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid,
get_types, *, archiv
))
if "SOURCE" in spdx_file.fileTypes:
-extracted_lics = extract_licenses(filepath)
+extracted_lics = oe.spdx_common.extract_licenses(filepath)
if extracted_lics:
spdx_file.licenseInfoInFiles = extracted_lics
@@ -256,6 +263,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
from pathlib import Path
import oe.sbom
import oe.spdx
+import oe.spdx_common
deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
package_archs = d.getVar("SSTATE_ARCHS").split()
@@ -263,7 +271,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
dep_recipes = []
-deps = get_spdx_deps(d)
+deps = oe.spdx_common.get_spdx_deps(d)
for dep_pn, dep_hashfn, in_taskhash in deps:
# If this dependency is not calculated in the taskhash skip it.
@@ -386,6 +394,7 @@ python do_create_spdx() {
from datetime import datetime, timezone
import oe.sbom
import oe.spdx
+import oe.spdx_common
import uuid
from pathlib import Path
from contextlib import contextmanager
@@ -478,10 +487,10 @@ python do_create_spdx() {
add_download_packages(d, doc, recipe)
-if process_sources(d) and include_sources:
+if oe.spdx_common.process_sources(d) and include_sources:
recipe_archive = deploy_dir_spdx / "recipes" / (doc.name + ".tar.zst")
with optional_tarfile(recipe_archive, archive_sources) as archive:
-spdx_get_src(d)
+oe.spdx_common.get_patched_src(d)
add_package_files(
d,
@@ -588,6 +597,7 @@ python do_create_runtime_spdx() {
from datetime import datetime, timezone
import oe.sbom
import oe.spdx
+import oe.spdx_common
import oe.packagedata
from pathlib import Path
@@ -597,7 +607,7 @@ python do_create_runtime_spdx() {
creation_time =
datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
-providers = collect_package_providers(d)
+providers = oe.spdx_common.collect_package_providers(d)
pkg_arch = d.getVar("SSTATE_PKGARCH")
package_archs = d.getVar("SSTATE_ARCHS").split()
package_archs.reverse()
@@ -778,6 +788,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir,
rootfs_spdxid, packages, spdx
import os
import oe.spdx
import oe.sbom
+import oe.spdx_common
import io
import json
from datetime import timezone, datetime
@@ -785,7 +796,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir,
rootfs_spdxid, packages, spdx
import tarfile
import bb.compress.zstd
-providers = collect_package_providers(d)
+providers = oe.spdx_common.collect_package_providers(d)
package_archs = d.getVar("SSTATE_ARCHS").split()
package_archs.reverse()
diff --git a/meta/classes/create-spdx-3.0.bbclass
b/meta/classes/create-spdx-3.0.bbclass
index 51168e4876c..a930ea81152 100644
--- a/meta/classes/create-spdx-3.0.bbclass
+++ b/meta/classes/create-spdx-3.0.bbclass
@@ -350,20 +350,21 @@ def collect_dep_objsets(d, build):
from pathlib import Path
[OE-core][PATCH v6 06/12] selftest: spdx: Add SPDX 3.0 test cases
Adds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can
also be run even if the default is SPDX 3.0
Signed-off-by: Joshua Watt
---
meta/lib/oeqa/selftest/cases/spdx.py | 133 +--
1 file changed, 124 insertions(+), 9 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/spdx.py
b/meta/lib/oeqa/selftest/cases/spdx.py
index 7685a81e7fb..be595babb35 100644
--- a/meta/lib/oeqa/selftest/cases/spdx.py
+++ b/meta/lib/oeqa/selftest/cases/spdx.py
@@ -6,21 +6,26 @@
import json
import os
+import textwrap
+from pathlib import Path
from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var, runCmd
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd
-class SPDXCheck(OESelftestTestCase):
+class SPDX22Check(OESelftestTestCase):
@classmethod
def setUpClass(cls):
-super(SPDXCheck, cls).setUpClass()
+super().setUpClass()
bitbake("python3-spdx-tools-native")
bitbake("-c addto_recipe_sysroot python3-spdx-tools-native")
def check_recipe_spdx(self, high_level_dir, spdx_file, target_name):
-config = """
-INHERIT += "create-spdx"
-"""
+config = textwrap.dedent(
+"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "create-spdx-2.2"
+"""
+)
self.write_config(config)
deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx"
# qemux86-64 creates the directory qemux86_64
machine_dir = machine_var.replace("-", "_")
-full_file_path = os.path.join(deploy_dir, "spdx", spdx_version,
machine_dir, high_level_dir, spdx_file)
+full_file_path = os.path.join(
+deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir,
spdx_file
+)
try:
os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx"
self.assertNotEqual(report, None)
self.assertNotEqual(report["SPDXID"], None)
-python = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'nativepython3')
-validator = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'pyspdxtools')
+python = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
+"nativepython3",
+)
+validator = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
"pyspdxtools"
+)
result = runCmd("{} {} -i {}".format(python, validator, filename))
self.assertExists(full_file_path)
@@ -53,3 +65,106 @@ INHERIT += "create-spdx"
def test_spdx_base_files(self):
self.check_recipe_spdx("packages", "base-files.spdx.json",
"base-files")
+
+
+class SPDX3CheckBase(object):
+"""
+Base class for checking SPDX 3 based tests
+"""
+
+def check_spdx_file(self, filename):
+import oe.spdx30
+
+self.assertExists(filename)
+
+# Read the file
+objset = oe.spdx30.SHACLObjectSet()
+with open(filename, "r") as f:
+d = oe.spdx30.JSONLDDeserializer()
+d.read(f, objset)
+
+return objset
+
+def check_recipe_spdx(self, target_name, spdx_path, *, task=None,
extraconf=""):
+config = textwrap.dedent(
+f"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "{self.SPDX_CLASS}"
+{extraconf}
+"""
+)
+self.write_config(config)
+
+if task:
+bitbake(f"-c {task} {target_name}")
+else:
+bitbake(target_name)
+
+filename = spdx_path.format(
+**get_bb_vars(
+[
+"DEPLOY_DIR_IMAGE",
+"DEPLOY_DIR_SPDX",
+"MACHINE",
+"MACHINE_ARCH",
+"SDKMACHINE",
+"SDK_DEPLOY",
+"SPDX_VERSION",
+"TOOLCHAIN_OUTPUTNAME",
+],
+target_name,
+)
+)
+
+return self.check_spdx_file(filename)
+
+def check_objset_missing_ids(self, objset):
+if objset.missing_ids:
+self.assertTrue(
+False,
+"The following
[OE-core][PATCH v6 08/12] selftest: sstatetests: Exclude all SPDX tasks
SPDX 3.0 introduces a bunch of new SPDX tasks. Instead of explicitly
enumerating them all, modify the regex to match 'create_.*spdx' which
will cover all of the SPDX 2.0 and SPDX 3.0 tasks
Signed-off-by: Joshua Watt
---
meta/lib/oeqa/selftest/cases/sstatetests.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py
b/meta/lib/oeqa/selftest/cases/sstatetests.py
index 94ad6e38b68..0153ef37cb6 100644
--- a/meta/lib/oeqa/selftest/cases/sstatetests.py
+++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
@@ -933,8 +933,7 @@ class SStateCheckObjectPresence(SStateBase):
# these get influnced by IMAGE_FSTYPES tweaks in
yocto-autobuilder-helper's config.json (on x86-64)
# additionally, they depend on noexec (thus, absent stamps) package,
install, etc. image tasks,
# which makes tracing other changes difficult
-exceptions += ["{}.*create_spdx".format(t) for t in targets.split()]
-exceptions += ["{}.*create_runtime_spdx".format(t) for t in
targets.split()]
+exceptions += ["{}.*create_.*spdx".format(t) for t in targets.split()]
output_l = output.splitlines()
for l in output_l:
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201842):
https://lists.openembedded.org/g/openembedded-core/message/201842
Mute This Topic: https://lists.openembedded.org/mt/107185003/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v6 07/12] classes-recipe: nospdx: Add class
Adds a class that allows recipes to opt out of generating SPDX
Signed-off-by: Joshua Watt
---
meta/classes-recipe/nospdx.bbclass | 13 +
meta/recipes-core/meta/build-sysroots.bb | 5 +
meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +--
3 files changed, 15 insertions(+), 6 deletions(-)
create mode 100644 meta/classes-recipe/nospdx.bbclass
diff --git a/meta/classes-recipe/nospdx.bbclass
b/meta/classes-recipe/nospdx.bbclass
new file mode 100644
index 000..b20e28218be
--- /dev/null
+++ b/meta/classes-recipe/nospdx.bbclass
@@ -0,0 +1,13 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+deltask do_collect_spdx_deps
+deltask do_create_spdx
+deltask do_create_spdx_runtime
+deltask do_create_package_spdx
+deltask do_create_rootfs_spdx
+deltask do_create_image_spdx
+deltask do_create_image_sbom
diff --git a/meta/recipes-core/meta/build-sysroots.bb
b/meta/recipes-core/meta/build-sysroots.bb
index db05c111ab2..b0b8fb3c79a 100644
--- a/meta/recipes-core/meta/build-sysroots.bb
+++ b/meta/recipes-core/meta/build-sysroots.bb
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}"
PACKAGE_ARCH = "${MACHINE_ARCH}"
EXCLUDE_FROM_WORLD = "1"
-inherit nopackages
+inherit nopackages nospdx
deltask fetch
deltask unpack
deltask patch
@@ -17,9 +17,6 @@ deltask configure
deltask compile
deltask install
deltask populate_sysroot
-deltask create_spdx
-deltask collect_spdx_deps
-deltask create_runtime_spdx
deltask recipe_qa
do_build_warn () {
diff --git a/meta/recipes-core/meta/meta-world-pkgdata.bb
b/meta/recipes-core/meta/meta-world-pkgdata.bb
index 0438bf61387..244175ddd44 100644
--- a/meta/recipes-core/meta/meta-world-pkgdata.bb
+++ b/meta/recipes-core/meta/meta-world-pkgdata.bb
@@ -27,14 +27,13 @@ python do_collect_packagedata() {
oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
}
+inherit nospdx
deltask do_fetch
deltask do_unpack
deltask do_patch
deltask do_configure
deltask do_compile
deltask do_install
-deltask do_create_spdx
-deltask do_create_spdx_runtime
do_prepare_recipe_sysroot[deptask] = ""
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201841):
https://lists.openembedded.org/g/openembedded-core/message/201841
Mute This Topic: https://lists.openembedded.org/mt/107185002/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v6 05/12] classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
Adds the variable overrides to set the SPDX image purpose for various
image types
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image_types.bbclass | 2 ++
meta/classes-recipe/image_types_wic.bbclass | 1 +
2 files changed, 3 insertions(+)
diff --git a/meta/classes-recipe/image_types.bbclass
b/meta/classes-recipe/image_types.bbclass
index 2f948ecbf88..506b9934cb7 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}"
IMAGE_CMD_TAR ?= "tar"
# ignore return code 1 "file changed as we read it" as other tasks(e.g.
do_image_wic) may be hardlinking rootfs
IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner
-cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+SPDX_IMAGE_PURPOSE:tar = "archive"
do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () {
fi
fi
}
+SPDX_IMAGE_PURPOSE:cpio = "archive"
UBI_VOLNAME ?= "${MACHINE}-rootfs"
UBI_VOLTYPE ?= "dynamic"
diff --git a/meta/classes-recipe/image_types_wic.bbclass
b/meta/classes-recipe/image_types_wic.bbclass
index cf3be909b30..86f40633ebc 100644
--- a/meta/classes-recipe/image_types_wic.bbclass
+++ b/meta/classes-recipe/image_types_wic.bbclass
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () {
mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic"
}
IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+SPDX_IMAGE_PURPOSE:wic = "diskImage"
do_image_wic[cleandirs] = "${WORKDIR}/build-wic"
PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201839):
https://lists.openembedded.org/g/openembedded-core/message/201839
Mute This Topic: https://lists.openembedded.org/mt/107185000/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v6 04/12] classes-global/staging: Exclude do_create_spdx from automatic sysroot extension
do_create_spdx is a outlier in that it doesn't need the RSS to be
extended just because it depends on do_populate_sysroot. In fact, it
only depends on do_populate_sysroot so it can see the actual recipes
sysroot, and attempting to extend the sysroot can cause problems for
some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")
As such, explicitly exclude do_create_spdx from extending the sysroot
just because it depends on do_populate_sysroot.
Signed-off-by: Joshua Watt
---
meta/classes-global/staging.bbclass | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/meta/classes-global/staging.bbclass
b/meta/classes-global/staging.bbclass
index 3678a1b4415..c2213ffa2b4 100644
--- a/meta/classes-global/staging.bbclass
+++ b/meta/classes-global/staging.bbclass
@@ -652,10 +652,17 @@ python do_prepare_recipe_sysroot () {
addtask do_prepare_recipe_sysroot before do_configure after do_fetch
python staging_taskhandler() {
+EXCLUDED_TASKS = (
+"do_prepare_recipe_sysroot",
+"do_create_spdx",
+)
bbtasks = e.tasklist
for task in bbtasks:
+if task in EXCLUDED_TASKS:
+continue
+
deps = d.getVarFlag(task, "depends")
-if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or
(deps and "populate_sysroot" in deps)):
+if task == "do_configure" or (deps and "populate_sysroot" in deps):
d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ")
}
staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201838):
https://lists.openembedded.org/g/openembedded-core/message/201838
Mute This Topic: https://lists.openembedded.org/mt/107184999/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v6 01/12] classes-recipe/image: Add image file manifest
Downstream tasks may want to know what image files were written by the
do_image family of tasks (e.g. SPDX) so have each task write out a
manifest file that describes the files it produced, then aggregate them
in do_image_complete
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image.bbclass | 58 +++
1 file changed, 58 insertions(+)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c63623..32bafcdbbc3 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -88,6 +88,11 @@ PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}"
IMGDEPLOYDIR = "${WORKDIR}/deploy-${PN}-image-complete"
+IMGMANIFESTDIR = "${WORKDIR}/image-task-manifest"
+
+IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest"
+IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json"
+
# Images are generally built explicitly, do not need to be part of world.
EXCLUDE_FROM_WORLD = "1"
@@ -277,14 +282,28 @@ fakeroot python do_image () {
execute_pre_post_process(d, pre_process_cmds)
}
do_image[dirs] = "${TOPDIR}"
+do_image[cleandirs] += "${IMGMANIFESTDIR}"
addtask do_image after do_rootfs
fakeroot python do_image_complete () {
from oe.utils import execute_pre_post_process
+from pathlib import Path
+import json
post_process_cmds = d.getVar("IMAGE_POSTPROCESS_COMMAND")
execute_pre_post_process(d, post_process_cmds)
+
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+
+data = []
+
+for manifest_path in image_manifest_dir.glob("*.json"):
+with manifest_path.open("r") as f:
+data.extend(json.load(f))
+
+with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f:
+json.dump(data, f)
}
do_image_complete[dirs] = "${TOPDIR}"
SSTATETASKS += "do_image_complete"
@@ -292,6 +311,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
sstate_setscene(d)
@@ -507,12 +528,14 @@ python () {
d.setVar(task, '\n'.join(cmds))
d.setVarFlag(task, 'func', '1')
d.setVarFlag(task, 'fakeroot', '1')
+d.setVarFlag(task, 'imagetype', t)
d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
d.prependVarFlag(task, 'postfuncs', 'create_symlinks ')
d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
d.appendVarFlag(task, 'vardepsexclude', ' DATETIME DATE ' + '
'.join(vardepsexclude))
+d.appendVarFlag(task, 'postfuncs', ' write_image_output_manifest')
bb.debug(2, "Adding task %s before %s, after %s" % (task,
'do_image_complete', after))
bb.build.addtask(task, 'do_image_complete', after, d)
@@ -610,6 +633,41 @@ python create_symlinks() {
bb.note("Skipping symlink, source does not exist: %s -> %s" %
(dst, src))
}
+python write_image_output_manifest() {
+import json
+from pathlib import Path
+
+taskname = d.getVar("BB_CURRENTTASK")
+image_deploy_dir = Path(d.getVar('IMGDEPLOYDIR'))
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+manifest_path = image_manifest_dir / ("do_" + d.getVar("BB_CURRENTTASK") +
".json")
+
+image_name = d.getVar("IMAGE_NAME")
+image_basename = d.getVar("IMAGE_BASENAME")
+machine = d.getVar("MACHINE")
+
+subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or
"").split()
+imagetype = d.getVarFlag("do_" + taskname, 'imagetype', False)
+
+data = {
+"taskname": taskname,
+"imagetype": imagetype,
+"images": []
+}
+
+for type in subimages:
+image_filename = image_name + "." + type
+image_path = image_deploy_dir / image_filename
+if not image_path.exists():
+continue
+data["images"].append({
+"filename":
[OE-core][PATCH v6 02/12] classes-recipe/baremetal-image: Add image file manifest
Downstream tasks may want to know what image files were written so write
out a manifest in do_image_complete. The format of the manifest is the
same as the one in image.bbclass
Signed-off-by: Joshua Watt
---
meta/classes-recipe/baremetal-image.bbclass | 32 +++--
1 file changed, 29 insertions(+), 3 deletions(-)
diff --git a/meta/classes-recipe/baremetal-image.bbclass
b/meta/classes-recipe/baremetal-image.bbclass
index 4e7d413626e..7938c0a83f1 100644
--- a/meta/classes-recipe/baremetal-image.bbclass
+++ b/meta/classes-recipe/baremetal-image.bbclass
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}"
IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}"
IMAGE_NAME_SUFFIX ?= ""
+IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest"
+IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json"
+
do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}"
do_image(){
@@ -37,8 +40,28 @@ do_image(){
install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf
${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf
}
-do_image_complete(){
-:
+python do_image_complete(){
+from pathlib import Path
+import json
+
+data = {
+"taskname": "do_image",
+"imagetype": "baremetal-image",
+"images": []
+}
+
+img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR"))
+
+for child in img_deploy_dir.iterdir():
+if not child.is_file() or child.is_symlink():
+continue
+
+data["images"].append({
+"filename": child.name,
+})
+
+with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f:
+json.dump([data], f)
}
python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){
bb.utils.mkdirhier(sysconfdir)
execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND'))
+execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND"))
}
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
@@ -140,5 +166,5 @@ python(){
else:
deps += " %s:%s" % (dep, task)
return deps
-d.appendVarFlag('do_image', 'depends',
extraimage_getdepends('do_populate_sysroot'))
+d.appendVarFlag('do_image', 'depends',
extraimage_getdepends('do_populate_sysroot'))
}
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201836):
https://lists.openembedded.org/g/openembedded-core/message/201836
Mute This Topic: https://lists.openembedded.org/mt/107184996/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] perf: Remove SPDX_S
This variable hasn't been used for some time, so remove it
Signed-off-by: Joshua Watt
---
meta/recipes-kernel/perf/perf.bb | 1 -
1 file changed, 1 deletion(-)
diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index 058f887b861..fc9f70f9f66 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -66,7 +66,6 @@ include ${@bb.utils.contains('PACKAGECONFIG', 'perl',
'perf-perl.inc', '', d)}
inherit kernelsrc
S = "${WORKDIR}/${BP}"
-SPDX_S = "${S}/tools/perf"
# The LDFLAGS is required or some old kernels fails due missing
# symbols and this is preferred than requiring patches to every old
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201834):
https://lists.openembedded.org/g/openembedded-core/message/201834
Mute This Topic: https://lists.openembedded.org/mt/107184828/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] icu: Remove SPDX_S
This variable hasn't been used for some time so remove it.
Signed-off-by: Joshua Watt
---
meta/recipes-support/icu/icu_75-1.bb | 1 -
1 file changed, 1 deletion(-)
diff --git a/meta/recipes-support/icu/icu_75-1.bb
b/meta/recipes-support/icu/icu_75-1.bb
index 27fe392e4d4..148c393b173 100644
--- a/meta/recipes-support/icu/icu_75-1.bb
+++ b/meta/recipes-support/icu/icu_75-1.bb
@@ -11,7 +11,6 @@ DEPENDS = "icu-native autoconf-archive-native"
CVE_PRODUCT = "international_components_for_unicode"
S = "${WORKDIR}/icu/source"
-SPDX_S = "${WORKDIR}/icu"
STAGING_ICU_DIR_NATIVE = "${STAGING_DATADIR_NATIVE}/${BPN}/${PV}"
ICU_MAJOR_VER = "${@d.getVar('PV').split('-')[0]}"
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201833):
https://lists.openembedded.org/g/openembedded-core/message/201833
Mute This Topic: https://lists.openembedded.org/mt/107184823/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH v5 0/8] Add SPDX 3.0 support
On Wed, Jul 10, 2024 at 6:18 AM Richard Purdie wrote: > > On Fri, 2024-07-05 at 08:17 +0100, Richard Purdie via lists.openembedded.org > wrote: > > > On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via > > > lists.openembedded.org wrote: > > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > > default. > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > > > > the same time > > > > > > > > > > v2: Added tests and addressed feedback > > > > > v3: Fixed several oe-selftest and build failures > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > v5: Reworked to make SPDX 3 output reproducible by default. Variables > > > > > that introduce non-reproducible output are documented as such. > > > > > > Thanks for working on this. The patches generally seem to working well > > > in testing but this does look related: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio > > > > > > Particularly these bits: > > > > > > AssertionError: 10 != 0 : Missing objects in the cache: > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst > > I think I understand what might have happened here. I suspect we've had > test builds on the new test cluster which can write to the hash > equivalence server but not the sstate. I think this means entries were > added for artefacts which don't exist on sstate. > > In theory those should be created on new build runs but in practise I > suspect they're shadowed by other artefacts so newer builds probably > don't create them. That would at least explain how we end up where we > are. There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933 which I beleive is still accurate and probably why the images themselves show up; we'll need to adjust the exclusions to match the new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK? > > If I'm correct, if there are changes to the code (as we discussed on > yesterdays tech call), everything will rerun and assuming we run this > only on the main cluster, things should work out ok. > > I would be curious to understand how these things aren't being > recreated when missing though? I'm not sure. I can't really think of any reason in the SPDX code that it wouldn't recreate a missing sstate object; I'm not doing anything particularly strange with sstate > > I did also notice an interesting issue in that if you run a build with > PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the > do_collect_spdx_deps tasks and
[OE-core][PATCH v5 7/8] classes-recipe: nospdx: Add class
Adds a class that allows recipes to opt out of generating SPDX
Signed-off-by: Joshua Watt
---
meta/classes-recipe/nospdx.bbclass | 13 +
meta/recipes-core/meta/build-sysroots.bb | 5 +
meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +--
3 files changed, 15 insertions(+), 6 deletions(-)
create mode 100644 meta/classes-recipe/nospdx.bbclass
diff --git a/meta/classes-recipe/nospdx.bbclass
b/meta/classes-recipe/nospdx.bbclass
new file mode 100644
index 000..b20e28218be
--- /dev/null
+++ b/meta/classes-recipe/nospdx.bbclass
@@ -0,0 +1,13 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+deltask do_collect_spdx_deps
+deltask do_create_spdx
+deltask do_create_spdx_runtime
+deltask do_create_package_spdx
+deltask do_create_rootfs_spdx
+deltask do_create_image_spdx
+deltask do_create_image_sbom
diff --git a/meta/recipes-core/meta/build-sysroots.bb
b/meta/recipes-core/meta/build-sysroots.bb
index db05c111ab2..b0b8fb3c79a 100644
--- a/meta/recipes-core/meta/build-sysroots.bb
+++ b/meta/recipes-core/meta/build-sysroots.bb
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}"
PACKAGE_ARCH = "${MACHINE_ARCH}"
EXCLUDE_FROM_WORLD = "1"
-inherit nopackages
+inherit nopackages nospdx
deltask fetch
deltask unpack
deltask patch
@@ -17,9 +17,6 @@ deltask configure
deltask compile
deltask install
deltask populate_sysroot
-deltask create_spdx
-deltask collect_spdx_deps
-deltask create_runtime_spdx
deltask recipe_qa
do_build_warn () {
diff --git a/meta/recipes-core/meta/meta-world-pkgdata.bb
b/meta/recipes-core/meta/meta-world-pkgdata.bb
index 0438bf61387..244175ddd44 100644
--- a/meta/recipes-core/meta/meta-world-pkgdata.bb
+++ b/meta/recipes-core/meta/meta-world-pkgdata.bb
@@ -27,14 +27,13 @@ python do_collect_packagedata() {
oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
}
+inherit nospdx
deltask do_fetch
deltask do_unpack
deltask do_patch
deltask do_configure
deltask do_compile
deltask do_install
-deltask do_create_spdx
-deltask do_create_spdx_runtime
do_prepare_recipe_sysroot[deptask] = ""
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201488):
https://lists.openembedded.org/g/openembedded-core/message/201488
Mute This Topic: https://lists.openembedded.org/mt/107019837/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v5 5/8] classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
Adds the variable overrides to set the SPDX image purpose for various
image types
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image_types.bbclass | 2 ++
meta/classes-recipe/image_types_wic.bbclass | 1 +
2 files changed, 3 insertions(+)
diff --git a/meta/classes-recipe/image_types.bbclass
b/meta/classes-recipe/image_types.bbclass
index 2f948ecbf88..506b9934cb7 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}"
IMAGE_CMD_TAR ?= "tar"
# ignore return code 1 "file changed as we read it" as other tasks(e.g.
do_image_wic) may be hardlinking rootfs
IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner
-cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+SPDX_IMAGE_PURPOSE:tar = "archive"
do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () {
fi
fi
}
+SPDX_IMAGE_PURPOSE:cpio = "archive"
UBI_VOLNAME ?= "${MACHINE}-rootfs"
UBI_VOLTYPE ?= "dynamic"
diff --git a/meta/classes-recipe/image_types_wic.bbclass
b/meta/classes-recipe/image_types_wic.bbclass
index cf3be909b30..86f40633ebc 100644
--- a/meta/classes-recipe/image_types_wic.bbclass
+++ b/meta/classes-recipe/image_types_wic.bbclass
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () {
mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic"
}
IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+SPDX_IMAGE_PURPOSE:wic = "diskImage"
do_image_wic[cleandirs] = "${WORKDIR}/build-wic"
PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201486):
https://lists.openembedded.org/g/openembedded-core/message/201486
Mute This Topic: https://lists.openembedded.org/mt/107019832/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v5 8/8] Switch default spdx version to 3.0
Changes the default SPDX version to 3.0 Signed-off-by: Joshua Watt --- meta/classes/create-spdx.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/create-spdx.bbclass b/meta/classes/create-spdx.bbclass index 19c6c0ff0b9..b604973ae0a 100644 --- a/meta/classes/create-spdx.bbclass +++ b/meta/classes/create-spdx.bbclass @@ -5,4 +5,4 @@ # # Include this class when you don't care what version of SPDX you get; it will # be updated to the latest stable version that is supported -inherit create-spdx-2.2 +inherit create-spdx-3.0 -- 2.45.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201489): https://lists.openembedded.org/g/openembedded-core/message/201489 Mute This Topic: https://lists.openembedded.org/mt/107019838/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v5 6/8] selftest: spdx: Add SPDX 3.0 test cases
Adds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can
also be run even if the default is SPDX 3.0
Signed-off-by: Joshua Watt
---
meta/lib/oeqa/selftest/cases/spdx.py | 118 +--
1 file changed, 109 insertions(+), 9 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/spdx.py
b/meta/lib/oeqa/selftest/cases/spdx.py
index 7685a81e7fb..4a507633cfe 100644
--- a/meta/lib/oeqa/selftest/cases/spdx.py
+++ b/meta/lib/oeqa/selftest/cases/spdx.py
@@ -6,21 +6,26 @@
import json
import os
+import textwrap
+from pathlib import Path
from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var, runCmd
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd
-class SPDXCheck(OESelftestTestCase):
+class SPDX22Check(OESelftestTestCase):
@classmethod
def setUpClass(cls):
-super(SPDXCheck, cls).setUpClass()
+super().setUpClass()
bitbake("python3-spdx-tools-native")
bitbake("-c addto_recipe_sysroot python3-spdx-tools-native")
def check_recipe_spdx(self, high_level_dir, spdx_file, target_name):
-config = """
-INHERIT += "create-spdx"
-"""
+config = textwrap.dedent(
+"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "create-spdx-2.2"
+"""
+)
self.write_config(config)
deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx"
# qemux86-64 creates the directory qemux86_64
machine_dir = machine_var.replace("-", "_")
-full_file_path = os.path.join(deploy_dir, "spdx", spdx_version,
machine_dir, high_level_dir, spdx_file)
+full_file_path = os.path.join(
+deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir,
spdx_file
+)
try:
os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx"
self.assertNotEqual(report, None)
self.assertNotEqual(report["SPDXID"], None)
-python = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'nativepython3')
-validator = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'pyspdxtools')
+python = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
+"nativepython3",
+)
+validator = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
"pyspdxtools"
+)
result = runCmd("{} {} -i {}".format(python, validator, filename))
self.assertExists(full_file_path)
@@ -53,3 +65,91 @@ INHERIT += "create-spdx"
def test_spdx_base_files(self):
self.check_recipe_spdx("packages", "base-files.spdx.json",
"base-files")
+
+
+class SPDX3CheckBase(object):
+"""
+Base class for checking SPDX 3 based tests
+"""
+
+def check_spdx_file(self, filename):
+import oe.spdx30
+
+self.assertExists(filename)
+
+# Read the file
+objset = oe.spdx30.SHACLObjectSet()
+with open(filename, "r") as f:
+d = oe.spdx30.JSONLDDeserializer()
+d.read(f, objset)
+
+return objset
+
+def check_recipe_spdx(self, target_name, spdx_path, *, task=None):
+config = textwrap.dedent(
+f"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "{self.SPDX_CLASS}"
+"""
+)
+self.write_config(config)
+
+if task:
+bitbake(f"-c {task} {target_name}")
+else:
+bitbake(target_name)
+
+filename = spdx_path.format(
+**get_bb_vars(
+[
+"DEPLOY_DIR_IMAGE",
+"DEPLOY_DIR_SPDX",
+"MACHINE",
+"MACHINE_ARCH",
+"SDKMACHINE",
+"SDK_DEPLOY",
+"SPDX_VERSION",
+"TOOLCHAIN_OUTPUTNAME",
+],
+target_name,
+)
+)
+
+return self.check_spdx_file(filename)
+
+def check_objset_missing_ids(self, objset):
+if objset.missing_ids:
+self.assertTrue(
+False,
+"The following SPDXIDs are unresolved:\n &
[OE-core][PATCH v5 1/8] classes-recipe/image: Add image file manifest
Downstream tasks may want to know what image files were written by the
do_image family of tasks (e.g. SPDX) so have each task write out a
manifest file that describes the files it produced, then aggregate them
in do_image_complete
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image.bbclass | 58 +++
1 file changed, 58 insertions(+)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c63623..32bafcdbbc3 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -88,6 +88,11 @@ PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}"
IMGDEPLOYDIR = "${WORKDIR}/deploy-${PN}-image-complete"
+IMGMANIFESTDIR = "${WORKDIR}/image-task-manifest"
+
+IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest"
+IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json"
+
# Images are generally built explicitly, do not need to be part of world.
EXCLUDE_FROM_WORLD = "1"
@@ -277,14 +282,28 @@ fakeroot python do_image () {
execute_pre_post_process(d, pre_process_cmds)
}
do_image[dirs] = "${TOPDIR}"
+do_image[cleandirs] += "${IMGMANIFESTDIR}"
addtask do_image after do_rootfs
fakeroot python do_image_complete () {
from oe.utils import execute_pre_post_process
+from pathlib import Path
+import json
post_process_cmds = d.getVar("IMAGE_POSTPROCESS_COMMAND")
execute_pre_post_process(d, post_process_cmds)
+
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+
+data = []
+
+for manifest_path in image_manifest_dir.glob("*.json"):
+with manifest_path.open("r") as f:
+data.extend(json.load(f))
+
+with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f:
+json.dump(data, f)
}
do_image_complete[dirs] = "${TOPDIR}"
SSTATETASKS += "do_image_complete"
@@ -292,6 +311,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
sstate_setscene(d)
@@ -507,12 +528,14 @@ python () {
d.setVar(task, '\n'.join(cmds))
d.setVarFlag(task, 'func', '1')
d.setVarFlag(task, 'fakeroot', '1')
+d.setVarFlag(task, 'imagetype', t)
d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
d.prependVarFlag(task, 'postfuncs', 'create_symlinks ')
d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
d.appendVarFlag(task, 'vardepsexclude', ' DATETIME DATE ' + '
'.join(vardepsexclude))
+d.appendVarFlag(task, 'postfuncs', ' write_image_output_manifest')
bb.debug(2, "Adding task %s before %s, after %s" % (task,
'do_image_complete', after))
bb.build.addtask(task, 'do_image_complete', after, d)
@@ -610,6 +633,41 @@ python create_symlinks() {
bb.note("Skipping symlink, source does not exist: %s -> %s" %
(dst, src))
}
+python write_image_output_manifest() {
+import json
+from pathlib import Path
+
+taskname = d.getVar("BB_CURRENTTASK")
+image_deploy_dir = Path(d.getVar('IMGDEPLOYDIR'))
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+manifest_path = image_manifest_dir / ("do_" + d.getVar("BB_CURRENTTASK") +
".json")
+
+image_name = d.getVar("IMAGE_NAME")
+image_basename = d.getVar("IMAGE_BASENAME")
+machine = d.getVar("MACHINE")
+
+subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or
"").split()
+imagetype = d.getVarFlag("do_" + taskname, 'imagetype', False)
+
+data = {
+"taskname": taskname,
+"imagetype": imagetype,
+"images": []
+}
+
+for type in subimages:
+image_filename = image_name + "." + type
+image_path = image_deploy_dir / image_filename
+if not image_path.exists():
+continue
+data["images"].append({
+"filename":
[OE-core][PATCH v5 4/8] classes-global/staging: Exclude do_create_spdx from automatic sysroot extension
do_create_spdx is a outlier in that it doesn't need the RSS to be
extended just because it depends on do_populate_sysroot. In fact, it
only depends on do_populate_sysroot so it can see the actual recipes
sysroot, and attempting to extend the sysroot can cause problems for
some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")
As such, explicitly exclude do_create_spdx from extending the sysroot
just because it depends on do_populate_sysroot.
Signed-off-by: Joshua Watt
---
meta/classes-global/staging.bbclass | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/meta/classes-global/staging.bbclass
b/meta/classes-global/staging.bbclass
index 3678a1b4415..c2213ffa2b4 100644
--- a/meta/classes-global/staging.bbclass
+++ b/meta/classes-global/staging.bbclass
@@ -652,10 +652,17 @@ python do_prepare_recipe_sysroot () {
addtask do_prepare_recipe_sysroot before do_configure after do_fetch
python staging_taskhandler() {
+EXCLUDED_TASKS = (
+"do_prepare_recipe_sysroot",
+"do_create_spdx",
+)
bbtasks = e.tasklist
for task in bbtasks:
+if task in EXCLUDED_TASKS:
+continue
+
deps = d.getVarFlag(task, "depends")
-if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or
(deps and "populate_sysroot" in deps)):
+if task == "do_configure" or (deps and "populate_sysroot" in deps):
d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ")
}
staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201484):
https://lists.openembedded.org/g/openembedded-core/message/201484
Mute This Topic: https://lists.openembedded.org/mt/107019830/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v5 2/8] classes-recipe/baremetal-image: Add image file manifest
Downstream tasks may want to know what image files were written so write
out a manifest in do_image_complete. The format of the manifest is the
same as the one in image.bbclass
Signed-off-by: Joshua Watt
---
meta/classes-recipe/baremetal-image.bbclass | 32 +++--
1 file changed, 29 insertions(+), 3 deletions(-)
diff --git a/meta/classes-recipe/baremetal-image.bbclass
b/meta/classes-recipe/baremetal-image.bbclass
index 4e7d413626e..7938c0a83f1 100644
--- a/meta/classes-recipe/baremetal-image.bbclass
+++ b/meta/classes-recipe/baremetal-image.bbclass
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}"
IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}"
IMAGE_NAME_SUFFIX ?= ""
+IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest"
+IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json"
+
do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}"
do_image(){
@@ -37,8 +40,28 @@ do_image(){
install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf
${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf
}
-do_image_complete(){
-:
+python do_image_complete(){
+from pathlib import Path
+import json
+
+data = {
+"taskname": "do_image",
+"imagetype": "baremetal-image",
+"images": []
+}
+
+img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR"))
+
+for child in img_deploy_dir.iterdir():
+if not child.is_file() or child.is_symlink():
+continue
+
+data["images"].append({
+"filename": child.name,
+})
+
+with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f:
+json.dump([data], f)
}
python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){
bb.utils.mkdirhier(sysconfdir)
execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND'))
+execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND"))
}
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
@@ -140,5 +166,5 @@ python(){
else:
deps += " %s:%s" % (dep, task)
return deps
-d.appendVarFlag('do_image', 'depends',
extraimage_getdepends('do_populate_sysroot'))
+d.appendVarFlag('do_image', 'depends',
extraimage_getdepends('do_populate_sysroot'))
}
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201482):
https://lists.openembedded.org/g/openembedded-core/message/201482
Mute This Topic: https://lists.openembedded.org/mt/107019825/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v5 0/8] Add SPDX 3.0 support
This patch series add support for SPDX 3.0 and sets it as the default. Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time v2: Added tests and addressed feedback v3: Fixed several oe-selftest and build failures v4: Fixed silly typo mistake in staging.bbclass v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such. Joshua Watt (8): classes-recipe/image: Add image file manifest classes-recipe/baremetal-image: Add image file manifest classes/create-spdx-3.0: Add classes classes-global/staging: Exclude do_create_spdx from automatic sysroot extension classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images selftest: spdx: Add SPDX 3.0 test cases classes-recipe: nospdx: Add class Switch default spdx version to 3.0 meta/classes-global/staging.bbclass |9 +- meta/classes-recipe/baremetal-image.bbclass | 32 +- meta/classes-recipe/image.bbclass| 58 + meta/classes-recipe/image_types.bbclass |2 + meta/classes-recipe/image_types_wic.bbclass |1 + meta/classes-recipe/nospdx.bbclass | 13 + meta/classes-recipe/packagegroup.bbclass |2 + meta/classes/create-spdx-3.0.bbclass | 1231 meta/classes/create-spdx-image-3.0.bbclass | 224 + meta/classes/create-spdx.bbclass |2 +- meta/classes/spdx-common.bbclass |6 +- meta/lib/oe/sbom30.py| 1138 meta/lib/oe/spdx30.py| 5996 ++ meta/lib/oeqa/selftest/cases/spdx.py | 118 +- meta/recipes-core/meta/build-sysroots.bb |5 +- meta/recipes-core/meta/meta-world-pkgdata.bb |3 +- 16 files changed, 8819 insertions(+), 21 deletions(-) create mode 100644 meta/classes-recipe/nospdx.bbclass create mode 100644 meta/classes/create-spdx-3.0.bbclass create mode 100644 meta/classes/create-spdx-image-3.0.bbclass create mode 100644 meta/lib/oe/sbom30.py create mode 100644 meta/lib/oe/spdx30.py -- 2.45.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201480): https://lists.openembedded.org/g/openembedded-core/message/201480 Mute This Topic: https://lists.openembedded.org/mt/107019821/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v2] classes/create-spdx-2.2: Handle SemVer License List Versions
SPDX transitioned the license list to use SemVer visioning, (e.g.
"MAJOR.MINOR.MICRO"), but SPDX 2 only allows "MAJOR.MINOR". For maximum
compatibility, only keep the first two version numbers and discard the
rest which allows it to work with either scheme
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-2.2.bbclass | 13 +
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index 99061320e53..3bcde1acc84 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -376,6 +376,11 @@ def add_download_packages(d, doc, recipe):
# but this should be sufficient for now
doc.add_relationship(package, "BUILD_DEPENDENCY_OF", recipe)
+def get_license_list_version(d):
+# Newer versions of the SPDX license list are SemVer ("MAJOR.MINOR.MICRO"),
+# but SPDX 2 only uses "MAJOR.MINOR".
+return
".".join(d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"].split(".")[:2])
+
python do_create_spdx() {
from datetime import datetime, timezone
@@ -417,7 +422,7 @@ python do_create_spdx() {
doc.documentNamespace = get_namespace(d, doc.name)
doc.creationInfo.created = creation_time
doc.creationInfo.comment = "This document was created by analyzing recipe
files during the build."
-doc.creationInfo.licenseListVersion =
d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+doc.creationInfo.licenseListVersion = get_license_list_version(d)
doc.creationInfo.creators.append("Tool: OpenEmbedded Core
create-spdx.bbclass")
doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
doc.creationInfo.creators.append("Person: N/A ()")
@@ -521,7 +526,7 @@ python do_create_spdx() {
package_doc.documentNamespace = get_namespace(d, package_doc.name)
package_doc.creationInfo.created = creation_time
package_doc.creationInfo.comment = "This document was created by
analyzing packages created during the build."
-package_doc.creationInfo.licenseListVersion =
d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+package_doc.creationInfo.licenseListVersion =
get_license_list_version(d)
package_doc.creationInfo.creators.append("Tool: OpenEmbedded Core
create-spdx.bbclass")
package_doc.creationInfo.creators.append("Organization: %s" %
d.getVar("SPDX_ORG"))
package_doc.creationInfo.creators.append("Person: N/A ()")
@@ -628,7 +633,7 @@ python do_create_runtime_spdx() {
runtime_doc.documentNamespace = get_namespace(localdata,
runtime_doc.name)
runtime_doc.creationInfo.created = creation_time
runtime_doc.creationInfo.comment = "This document was created by
analyzing package runtime dependencies."
-runtime_doc.creationInfo.licenseListVersion =
d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+runtime_doc.creationInfo.licenseListVersion =
get_license_list_version(d)
runtime_doc.creationInfo.creators.append("Tool: OpenEmbedded Core
create-spdx.bbclass")
runtime_doc.creationInfo.creators.append("Organization: %s" %
d.getVar("SPDX_ORG"))
runtime_doc.creationInfo.creators.append("Person: N/A ()")
@@ -793,7 +798,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir,
rootfs_spdxid, packages, spdx
doc.documentNamespace = get_namespace(d, doc.name)
doc.creationInfo.created = creation_time
doc.creationInfo.comment = "This document was created by analyzing the
source of the Yocto recipe during the build."
-doc.creationInfo.licenseListVersion =
d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+doc.creationInfo.licenseListVersion = get_license_list_version(d)
doc.creationInfo.creators.append("Tool: OpenEmbedded Core
create-spdx.bbclass")
doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
doc.creationInfo.creators.append("Person: N/A ()")
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201313):
https://lists.openembedded.org/g/openembedded-core/message/201313
Mute This Topic: https://lists.openembedded.org/mt/106983299/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH] classes/create-spdx-2.2: Handle SemVer License List Versions
Hang on, apparently we call this a lot of places, V2 coming soon
On Mon, Jul 1, 2024 at 8:42 AM Joshua Watt wrote:
>
> SPDX transitioned the license list to use SemVer visioning, (e.g.
> "MAJOR.MINOR.MICRO"), but SPDX 2 only allows "MAJOR.MINOR". For maximum
> compatibility, only keep the first two version numbers and discard the
> rest which allows it to work with either scheme
>
> Signed-off-by: Joshua Watt
> ---
> meta/classes/create-spdx-2.2.bbclass | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/create-spdx-2.2.bbclass
> b/meta/classes/create-spdx-2.2.bbclass
> index 99061320e53..6c568d9dbb1 100644
> --- a/meta/classes/create-spdx-2.2.bbclass
> +++ b/meta/classes/create-spdx-2.2.bbclass
> @@ -417,7 +417,9 @@ python do_create_spdx() {
> doc.documentNamespace = get_namespace(d, doc.name)
> doc.creationInfo.created = creation_time
> doc.creationInfo.comment = "This document was created by analyzing
> recipe files during the build."
> -doc.creationInfo.licenseListVersion =
> d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
> +# Newer versions of the SPDX license list are SemVer
> ("MAJOR.MINOR.MICRO"),
> +# but SPDX 2 only uses "MAJOR.MINOR".
> +doc.creationInfo.licenseListVersion =
> ".".join(d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"].split(".")[:2])
> doc.creationInfo.creators.append("Tool: OpenEmbedded Core
> create-spdx.bbclass")
> doc.creationInfo.creators.append("Organization: %s" %
> d.getVar("SPDX_ORG"))
> doc.creationInfo.creators.append("Person: N/A ()")
> --
> 2.45.2
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201308):
https://lists.openembedded.org/g/openembedded-core/message/201308
Mute This Topic: https://lists.openembedded.org/mt/106980332/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] classes/create-spdx-2.2: Handle SemVer License List Versions
SPDX transitioned the license list to use SemVer visioning, (e.g.
"MAJOR.MINOR.MICRO"), but SPDX 2 only allows "MAJOR.MINOR". For maximum
compatibility, only keep the first two version numbers and discard the
rest which allows it to work with either scheme
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-2.2.bbclass | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index 99061320e53..6c568d9dbb1 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -417,7 +417,9 @@ python do_create_spdx() {
doc.documentNamespace = get_namespace(d, doc.name)
doc.creationInfo.created = creation_time
doc.creationInfo.comment = "This document was created by analyzing recipe
files during the build."
-doc.creationInfo.licenseListVersion =
d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+# Newer versions of the SPDX license list are SemVer ("MAJOR.MINOR.MICRO"),
+# but SPDX 2 only uses "MAJOR.MINOR".
+doc.creationInfo.licenseListVersion =
".".join(d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"].split(".")[:2])
doc.creationInfo.creators.append("Tool: OpenEmbedded Core
create-spdx.bbclass")
doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
doc.creationInfo.creators.append("Person: N/A ()")
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201307):
https://lists.openembedded.org/g/openembedded-core/message/201307
Mute This Topic: https://lists.openembedded.org/mt/106980332/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH 0/2] Update to latest version of SPDX licenses
Ah, right, I should have known that would happen. I'll send the fix
On Mon, Jul 1, 2024 at 12:56 AM Richard Purdie
wrote:
>
> On Thu, 2024-06-27 at 16:16 -0600, Joshua Watt via lists.openembedded.org
> wrote:
> > Adds a script to read the released SPDX license JSON data, and uses it
> > to update the common license files
> >
> > Joshua Watt (2):
> > scripts/pull-spdx-licenses.py: Add script
> > licenses: Update to SPDX license version 3.24.0
>
> Thanks for doing this. It does seem to upset one of the selftests:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6871
>
> 2024-06-30 21:07:48,909 - oe-selftest - INFO -
> ==
> 2024-06-30 21:07:48,966 - oe-selftest - INFO - FAIL:
> spdx.SPDXCheck.test_spdx_base_files (subunit.RemotedTestCase)
> 2024-06-30 21:07:48,966 - oe-selftest - INFO -
> --
> 2024-06-30 21:07:48,980 - oe-selftest - INFO -
> testtools.testresult.real._StringException: Traceback (most recent call last):
> File
> "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/spdx.py",
> line 55, in test_spdx_base_files
> self.check_recipe_spdx("packages", "base-files.spdx.json", "base-files")
> File
> "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/spdx.py",
> line 52, in check_recipe_spdx
> result = check_spdx_json(full_file_path)
> ^^^
> File
> "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/spdx.py",
> line 49, in check_spdx_json
> result = runCmd("{} {} -i {}".format(python, validator, filename))
> ^
> File
> "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py",
> line 212, in runCmd
> raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s"
> % (command, result.status, exc_output))
> AssertionError: Command
> '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-296488/tmp/work/x86_64-linux/python3-spdx-tools-native/0.8.2/recipe-sysroot-native/usr/bin/nativepython3
>
> /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-296488/tmp/work/x86_64-linux/python3-spdx-tools-native/0.8.2/recipe-sysroot-native/usr/bin/pyspdxtools
> -i
> /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-296488/tmp/deploy/spdx/2.2/qemux86_64/packages/base-files.spdx.json'
> returned non-zero exit status 1:
> ERROR:root:There have been issues while parsing the provided document:
> Error while parsing Document: ['Error while parsing version 3.24.0: 3.24.0 is
> not a valid version string']
>
> It is coming from here:
>
> https://github.com/spdx/tools-python/blob/main/src/spdx_tools/spdx/model/version.py
>
> in spdx-tools :/.
>
> Cheers,
>
> Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201306):
https://lists.openembedded.org/g/openembedded-core/message/201306
Mute This Topic: https://lists.openembedded.org/mt/106918424/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 1/2] scripts/pull-spdx-licenses.py: Add script
Adds a script to pull the SPDX license data and update the license list
JSON data, as well as update the license directory.
Signed-off-by: Joshua Watt
---
scripts/pull-sdpx-licenses.py | 101 ++
1 file changed, 101 insertions(+)
create mode 100755 scripts/pull-sdpx-licenses.py
diff --git a/scripts/pull-sdpx-licenses.py b/scripts/pull-sdpx-licenses.py
new file mode 100755
index 000..597a62133fb
--- /dev/null
+++ b/scripts/pull-sdpx-licenses.py
@@ -0,0 +1,101 @@
+#! /usr/bin/env python3
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+
+import argparse
+import json
+import sys
+import urllib.request
+from pathlib import Path
+
+TOP_DIR = Path(__file__).parent.parent
+
+
+def main():
+parser = argparse.ArgumentParser(
+description="Update SPDX License files from upstream"
+)
+parser.add_argument(
+"-v",
+"--version",
+metavar="MAJOR.MINOR[.MICRO]",
+help="Pull specific version of License list instead of latest",
+)
+parser.add_argument(
+"--overwrite",
+action="store_true",
+help="Update existing license file text with upstream text",
+)
+parser.add_argument(
+"--deprecated",
+action="store_true",
+help="Update deprecated licenses",
+)
+parser.add_argument(
+"--dest",
+type=Path,
+default=TOP_DIR / "meta" / "files" / "common-licenses",
+help="Write licenses to directory DEST. Default is %(default)s",
+)
+
+args = parser.parse_args()
+
+if args.version:
+version = f"v{args.version}"
+else:
+# Fetch the latest release
+req = urllib.request.Request(
+
"https://api.github.com/repos/spdx/license-list-data/releases/latest";
+)
+req.add_header("X-GitHub-Api-Version", "2022-11-28")
+req.add_header("Accept", "application/vnd.github+json")
+with urllib.request.urlopen(req) as response:
+data = json.load(response)
+version = data["tag_name"]
+
+print(f"Pulling SPDX license list version {version}")
+req = urllib.request.Request(
+
f"https://raw.githubusercontent.com/spdx/license-list-data/{version}/json/licenses.json";
+)
+with urllib.request.urlopen(req) as response:
+spdx_licenses = json.load(response)
+
+with (TOP_DIR / "meta" / "files" / "spdx-licenses.json").open("w") as f:
+json.dump(spdx_licenses, f, sort_keys=True, indent=2)
+
+total_count = len(spdx_licenses["licenses"])
+updated = 0
+for idx, lic in enumerate(spdx_licenses["licenses"]):
+lic_id = lic["licenseId"]
+
+print(f"[{idx + 1} of {total_count}] ", end="")
+
+dest_license_file = args.dest / lic_id
+if dest_license_file.is_file() and not args.overwrite:
+print(f"Skipping {lic_id} since it already exists")
+continue
+
+print(f"Fetching {lic_id}... ", end="", flush=True)
+
+req = urllib.request.Request(lic["detailsUrl"])
+with urllib.request.urlopen(req) as response:
+lic_data = json.load(response)
+
+if lic_data["isDeprecatedLicenseId"] and not args.deprecated:
+print("Skipping (deprecated)")
+continue
+
+with dest_license_file.open("w") as f:
+f.write(lic_data["licenseText"])
+updated += 1
+print("done")
+
+print(f"Updated {updated} licenses")
+
+return 0
+
+
+if __name__ == "__main__":
+sys.exit(main())
--
2.45.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201213):
https://lists.openembedded.org/g/openembedded-core/message/201213
Mute This Topic: https://lists.openembedded.org/mt/106918425/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH 0/2] Update to latest version of SPDX licenses
Adds a script to read the released SPDX license JSON data, and uses it to update the common license files Joshua Watt (2): scripts/pull-spdx-licenses.py: Add script licenses: Update to SPDX license version 3.24.0 meta/files/common-licenses/3D-Slicer-1.0 | 190 + meta/files/common-licenses/AMD-newlib |11 + meta/files/common-licenses/AML-glslang|41 + .../common-licenses/ASWF-Digital-Assets-1.0 |17 + .../common-licenses/ASWF-Digital-Assets-1.1 |17 + meta/files/common-licenses/AdaCore-doc| 1 + .../common-licenses/Adobe-Display-PostScript |30 + meta/files/common-licenses/Adobe-Utopia |12 + meta/files/common-licenses/App-s2p| 5 + meta/files/common-licenses/Arphic-1999|58 + .../files/common-licenses/BSD-2-Clause-Darwin |28 + .../common-licenses/BSD-2-Clause-first-lines |28 + meta/files/common-licenses/BSD-3-Clause-HP|23 + meta/files/common-licenses/BSD-3-Clause-Sun |29 + .../files/common-licenses/BSD-3-Clause-acpica |26 + meta/files/common-licenses/BSD-3-Clause-flex |42 + meta/files/common-licenses/BSD-4.3RENO| 9 + meta/files/common-licenses/BSD-4.3TAHOE |11 + .../BSD-Advertising-Acknowledgement |37 + .../BSD-Attribution-HPND-disclaimer |37 + .../common-licenses/BSD-Inferno-Nettverk |42 + .../common-licenses/BSD-Source-beginning-file |23 + meta/files/common-licenses/BSD-Systemics |39 + .../common-licenses/BSD-Systemics-W3Works |62 + meta/files/common-licenses/Baekmuk| 9 + meta/files/common-licenses/Bitstream-Charter | 9 + meta/files/common-licenses/Bitstream-Vera |15 + meta/files/common-licenses/Boehm-GC |12 + .../common-licenses/Brian-Gladman-2-Clause|17 + .../common-licenses/Brian-Gladman-3-Clause|26 + meta/files/common-licenses/CC-BY-3.0-AU | 136 + meta/files/common-licenses/CC-BY-3.0-IGO | 101 + meta/files/common-licenses/CC-BY-NC-SA-2.0-DE |85 + meta/files/common-licenses/CC-BY-SA-3.0-IGO | 107 + meta/files/common-licenses/CFITSIO| 7 + meta/files/common-licenses/CMU-Mach |22 + meta/files/common-licenses/CMU-Mach-nodoc |11 + meta/files/common-licenses/COIL-1.0 |30 + .../files/common-licenses/Caldera-no-preamble |35 + meta/files/common-licenses/Catharon | 121 + meta/files/common-licenses/Clips |15 + meta/files/common-licenses/Community-Spec-1.0 | 293 + .../common-licenses/Cornell-Lossless-JPEG |20 + meta/files/common-licenses/Cronyx |11 + meta/files/common-licenses/DEC-3-Clause |28 + meta/files/common-licenses/DL-DE-BY-2.0 |45 + meta/files/common-licenses/DL-DE-ZERO-2.0 |25 + meta/files/common-licenses/DRL-1.1|17 + meta/files/common-licenses/Elastic-2.0|93 + meta/files/common-licenses/FBM| 6 + meta/files/common-licenses/FDK-AAC|79 + .../FSFAP-no-warranty-disclaimer | 5 + meta/files/common-licenses/FSFULLRWD |11 + meta/files/common-licenses/Ferguson-Twofish |15 + meta/files/common-licenses/Furuseth |13 + meta/files/common-licenses/GCR-docs |30 + meta/files/common-licenses/Graphics-Gems | 5 + meta/files/common-licenses/Gutmann| 2 + meta/files/common-licenses/HP-1986|10 + meta/files/common-licenses/HP-1989|16 + meta/files/common-licenses/HPND-DEC |22 + .../common-licenses/HPND-Fenneberg-Livingston |13 + meta/files/common-licenses/HPND-INRIA-IMAG| 9 + meta/files/common-licenses/HPND-Intel |25 + meta/files/common-licenses/HPND-Kevlin-Henney |10 + .../files/common-licenses/HPND-MIT-disclaimer |18 + meta/files/common-licenses/HPND-Markus-Kuhn | 3 + meta/files/common-licenses/HPND-Pbmplus | 8 + meta/files/common-licenses/HPND-UC| 8 + meta/files/common-licenses/HPND-UC-export-US |10 + meta/files/common-licenses/HPND-doc | 8 + meta/files/common-licenses/HPND-doc-sell | 9 + meta/files/common-licenses/HPND-export-US | 5 + .../HPND-export-US-acknowledgement|22 + .../common-licenses/HPND-export-US-modify |24 + meta/files/common-licenses/HPND-export2-US|21 + .../HPND-merchantability-variant | 9 + .../HPND-sell-MIT-disclaimer-xserver |12 + meta/files/common-licenses/HPND-sell-regexpr | 9 + .../HPND-sell-variant-MIT-disclaimer |20 + .../HPND-sell-variant-MIT-disclaimer-rev |15 + .../common-licenses/IEC-Code-Components-EULA |37 + meta/files/common-licenses/IJG-short |35 + meta/files/common-licenses/ISC-Veillard
Re: [OE-core][PATCH v4 03/10] classes/create-spdx-3.0: Add classes
On Thu, Jun 27, 2024 at 10:33 AM Joshua Watt wrote: > > On Tue, Jun 25, 2024 at 12:41 PM Mark Hatle > wrote: > > > > Comments inline below > > > > On 6/24/24 2:10 PM, Joshua Watt wrote: > -- snip -- > > > + > > > +SPDX_BUILD_HOST[doc] = "The base variable name to describe the build > > > host on \ > > > +which a build is running. Must be an SPDX_IMPORTS key" > > > > Is there any sort of documentation or external reference for the variable > > above > > (as well as the SPDX_ below) that explains what the SPDX standard is > > expecting > > to be put in there? > > Not specifically for this, but for the SPDX 3.0 spec in general, the > web docs are pretty comprehensive: > https://spdx.github.io/spdx-spec/v3.0/ . Although, be aware the > navigation sidebar is really annoying ATM, but that's supposed to get > fixed soon. > > For a starter on how SPDX documents are written, see: > https://github.com/spdx/spdx-spec/blob/development/v3.0.1/docs/annexes/getting-started.md > > It's a little tricky to encode the SPDX 3 structured data in bitbake > variables; this is what I could come up with so far but if you have > suggestions on improvements, let me know. > > Specifically for this variable, it's referencing a "key" in > SPDX_IMPORTS. SPDX_IMPORTS in turn is encoding entries in the > "imports" property of > https://spdx.github.io/spdx-spec/v3.0/model/Core/Classes/SpdxDocument/ > . The indirection is necessary so that you can tell users where the > SPDX ID lives, since it's external to this document. It's pretty much > impossible for us to validate that the SPDX ID you put in is real, > short of downloading the referenced document, parsing it, and seeing > if the SPDX ID is present. > > For a walk through of how to cross-link SPDX 3 documents, look at: > https://github.com/spdx/spdx-spec/blob/development/v3.0.1/docs/annexes/cross-reference.md > > > > > I.e. machine name, host type, an arbitrary string that means something to > > the > > agent construction the SPDX, etc. I.e. how would I know what is valid in > > these > > various things? > > There are pretty good tools to validate SPDX documents (offline > even!). > https://github.com/spdx/spdx-3-model/blob/main/serialization/json_ld/validation.md > gives an overview on how to do this. It still won't do the validation > that the external SPDX ID is valid for the same reasons as above, but > it's pretty good otherwise. > > > > > This then leads to a second question, deterministic > > behavior/reproducibility. I > > believe the purpose of this is reproducible builds, but we should have a > > more > > deterministic approach in the Yocto Project where we provide (and/or check > > for > > host capabilities) to help allow this to be a more generic, many different > > host > > process. > > I've attempted to make this process as deterministic as SPDX 3 allows. > As an example, SPDX IDs are generated by hashing deterministic data > (where as random SPDX IDs would be decidedly simpler!). However, there > are parts of SPDX 3 that are simply not deterministic for various > reasons (please read to the end of this section). They generally fall > under a few categories: > > The first category is probably best classified as "not very useful if > deterministic" and omitted by default; SPDX_BUILD_HOST would be one > such examples, since if you are going to set this to the same value > all the time and not reference the _actual_ host, you may as well not > include it at all. For these, I've set no default value (there isn't > one that would make sense anyway), so their omission keeps things > deterministic. However users do actually want this, it will > necessarily result in non-deterministic builds unless you always do > your builds on the same host. I think it might be helpful to annotate > in the doc string which variables will introduce such non-determinism, > so I'll do that. > > The second category is "not very useful if deterministic", but are > included in the output by this patch. Examples of this would be build > timestamps and the bitbake parent build tracking (which basically > tracks the invocation of bitbake itself as the "parent" build, so you > can tell which tasks ran in the same invocation). These are useful > pieces of information, and consumers do actually care about these > things, so if push comes to shove we could add a flag to enable them, > but I'm also leery of having too many configuration options for SPDX. > > The last category
Re: [OE-core][PATCH v4 03/10] classes/create-spdx-3.0: Add classes
On Tue, Jun 25, 2024 at 12:41 PM Mark Hatle wrote: > > Comments inline below > > On 6/24/24 2:10 PM, Joshua Watt wrote: -- snip -- > > + > > +SPDX_BUILD_HOST[doc] = "The base variable name to describe the build host > > on \ > > +which a build is running. Must be an SPDX_IMPORTS key" > > Is there any sort of documentation or external reference for the variable > above > (as well as the SPDX_ below) that explains what the SPDX standard is expecting > to be put in there? Not specifically for this, but for the SPDX 3.0 spec in general, the web docs are pretty comprehensive: https://spdx.github.io/spdx-spec/v3.0/ . Although, be aware the navigation sidebar is really annoying ATM, but that's supposed to get fixed soon. For a starter on how SPDX documents are written, see: https://github.com/spdx/spdx-spec/blob/development/v3.0.1/docs/annexes/getting-started.md It's a little tricky to encode the SPDX 3 structured data in bitbake variables; this is what I could come up with so far but if you have suggestions on improvements, let me know. Specifically for this variable, it's referencing a "key" in SPDX_IMPORTS. SPDX_IMPORTS in turn is encoding entries in the "imports" property of https://spdx.github.io/spdx-spec/v3.0/model/Core/Classes/SpdxDocument/ . The indirection is necessary so that you can tell users where the SPDX ID lives, since it's external to this document. It's pretty much impossible for us to validate that the SPDX ID you put in is real, short of downloading the referenced document, parsing it, and seeing if the SPDX ID is present. For a walk through of how to cross-link SPDX 3 documents, look at: https://github.com/spdx/spdx-spec/blob/development/v3.0.1/docs/annexes/cross-reference.md > > I.e. machine name, host type, an arbitrary string that means something to the > agent construction the SPDX, etc. I.e. how would I know what is valid in > these > various things? There are pretty good tools to validate SPDX documents (offline even!). https://github.com/spdx/spdx-3-model/blob/main/serialization/json_ld/validation.md gives an overview on how to do this. It still won't do the validation that the external SPDX ID is valid for the same reasons as above, but it's pretty good otherwise. > > This then leads to a second question, deterministic behavior/reproducibility. > I > believe the purpose of this is reproducible builds, but we should have a more > deterministic approach in the Yocto Project where we provide (and/or check for > host capabilities) to help allow this to be a more generic, many different > host > process. I've attempted to make this process as deterministic as SPDX 3 allows. As an example, SPDX IDs are generated by hashing deterministic data (where as random SPDX IDs would be decidedly simpler!). However, there are parts of SPDX 3 that are simply not deterministic for various reasons (please read to the end of this section). They generally fall under a few categories: The first category is probably best classified as "not very useful if deterministic" and omitted by default; SPDX_BUILD_HOST would be one such examples, since if you are going to set this to the same value all the time and not reference the _actual_ host, you may as well not include it at all. For these, I've set no default value (there isn't one that would make sense anyway), so their omission keeps things deterministic. However users do actually want this, it will necessarily result in non-deterministic builds unless you always do your builds on the same host. I think it might be helpful to annotate in the doc string which variables will introduce such non-determinism, so I'll do that. The second category is "not very useful if deterministic", but are included in the output by this patch. Examples of this would be build timestamps and the bitbake parent build tracking (which basically tracks the invocation of bitbake itself as the "parent" build, so you can tell which tasks ran in the same invocation). These are useful pieces of information, and consumers do actually care about these things, so if push comes to shove we could add a flag to enable them, but I'm also leery of having too many configuration options for SPDX. The last category are the require non-deterministic fields in SPDX 3. The primary offender here is the SPDX creation info "created" datestamp: https://spdx.github.io/spdx-spec/v3.0/model/Core/Classes/CreationInfo/ . This is a mandatory field that is the timestamp of when the SPDX data itself was created, and every SPDX object you create links to this so you can track exactly when each object in a merged document was created. I did attempt to make the argument to SPDX that it was mandatory non-determinism, but it is very important to the SPDX community (for reasons
[OE-core][PATCH v4 06/10] classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
Adds the variable overrides to set the SPDX image purpose for various
image types
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image_types.bbclass | 2 ++
meta/classes-recipe/image_types_wic.bbclass | 1 +
2 files changed, 3 insertions(+)
diff --git a/meta/classes-recipe/image_types.bbclass
b/meta/classes-recipe/image_types.bbclass
index 2f948ecbf88..506b9934cb7 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}"
IMAGE_CMD_TAR ?= "tar"
# ignore return code 1 "file changed as we read it" as other tasks(e.g.
do_image_wic) may be hardlinking rootfs
IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner
-cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+SPDX_IMAGE_PURPOSE:tar = "archive"
do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () {
fi
fi
}
+SPDX_IMAGE_PURPOSE:cpio = "archive"
UBI_VOLNAME ?= "${MACHINE}-rootfs"
UBI_VOLTYPE ?= "dynamic"
diff --git a/meta/classes-recipe/image_types_wic.bbclass
b/meta/classes-recipe/image_types_wic.bbclass
index cf3be909b30..86f40633ebc 100644
--- a/meta/classes-recipe/image_types_wic.bbclass
+++ b/meta/classes-recipe/image_types_wic.bbclass
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () {
mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic"
}
IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+SPDX_IMAGE_PURPOSE:wic = "diskImage"
do_image_wic[cleandirs] = "${WORKDIR}/build-wic"
PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201108):
https://lists.openembedded.org/g/openembedded-core/message/201108
Mute This Topic: https://lists.openembedded.org/mt/106856881/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v4 10/10] Switch default spdx version to 3.0
Changes the default SPDX version to 3.0 Signed-off-by: Joshua Watt --- meta/classes/create-spdx.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/create-spdx.bbclass b/meta/classes/create-spdx.bbclass index 19c6c0ff0b9..b604973ae0a 100644 --- a/meta/classes/create-spdx.bbclass +++ b/meta/classes/create-spdx.bbclass @@ -5,4 +5,4 @@ # # Include this class when you don't care what version of SPDX you get; it will # be updated to the latest stable version that is supported -inherit create-spdx-2.2 +inherit create-spdx-3.0 -- 2.43.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201112): https://lists.openembedded.org/g/openembedded-core/message/201112 Mute This Topic: https://lists.openembedded.org/mt/106856885/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v4 09/10] classes/spdx-common: Move SPDX_SUPPLIER
Move the SPDX_SUPPLIER variable to create-spdx-2.2 since it's format
only has meaning in SPDX 2.2 (SPDX 3 uses SPDX_PACKAGE_SUPPLIER with a
different format)
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-2.2.bbclass | 8
meta/classes/spdx-common.bbclass | 7 ---
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index 3ebf92b5e12..99061320e53 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -8,6 +8,14 @@ inherit spdx-common
SPDX_VERSION = "2.2"
+SPDX_ORG ??= "OpenEmbedded ()"
+SPDX_SUPPLIER ??= "Organization: ${SPDX_ORG}"
+SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created
from \
+this recipe. For SPDX documents create using this class during the build,
this \
+is the contact information for the person or organization who is doing the
\
+build."
+
+
def get_namespace(d, name):
import uuid
namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS,
d.getVar("SPDX_UUID_NAMESPACE"))
diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass
index 9dee325ba6b..6dfc1fd9e4c 100644
--- a/meta/classes/spdx-common.bbclass
+++ b/meta/classes/spdx-common.bbclass
@@ -37,13 +37,6 @@ SPDX_LICENSES ??= "${COREBASE}/meta/files/spdx-licenses.json"
SPDX_CUSTOM_ANNOTATION_VARS ??= ""
-SPDX_ORG ??= "OpenEmbedded ()"
-SPDX_SUPPLIER ??= "Organization: ${SPDX_ORG}"
-SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created
from \
-this recipe. For SPDX documents create using this class during the build,
this \
-is the contact information for the person or organization who is doing the
\
-build."
-
def extract_licenses(filename):
import re
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20):
https://lists.openembedded.org/g/openembedded-core/message/20
Mute This Topic: https://lists.openembedded.org/mt/106856884/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v4 04/10] classes-global/staging: Exclude do_create_spdx from automatic sysroot extension
do_create_spdx is a outlier in that it doesn't need the RSS to be
extended just because it depends on do_populate_sysroot. In fact, it
only depends on do_populate_sysroot so it can see the actual recipes
sysroot, and attempting to extend the sysroot can cause problems for
some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")
As such, explicitly exclude do_create_spdx from extending the sysroot
just because it depends on do_populate_sysroot.
Signed-off-by: Joshua Watt
---
meta/classes-global/staging.bbclass | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/meta/classes-global/staging.bbclass
b/meta/classes-global/staging.bbclass
index 3678a1b4415..c2213ffa2b4 100644
--- a/meta/classes-global/staging.bbclass
+++ b/meta/classes-global/staging.bbclass
@@ -652,10 +652,17 @@ python do_prepare_recipe_sysroot () {
addtask do_prepare_recipe_sysroot before do_configure after do_fetch
python staging_taskhandler() {
+EXCLUDED_TASKS = (
+"do_prepare_recipe_sysroot",
+"do_create_spdx",
+)
bbtasks = e.tasklist
for task in bbtasks:
+if task in EXCLUDED_TASKS:
+continue
+
deps = d.getVarFlag(task, "depends")
-if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or
(deps and "populate_sysroot" in deps)):
+if task == "do_configure" or (deps and "populate_sysroot" in deps):
d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ")
}
staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201105):
https://lists.openembedded.org/g/openembedded-core/message/201105
Mute This Topic: https://lists.openembedded.org/mt/106856876/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v4 08/10] classes-recipe: nospdx: Add class
Adds a class that allows recipes to opt out of generating SPDX
Signed-off-by: Joshua Watt
---
meta/classes-recipe/nospdx.bbclass | 13 +
meta/recipes-core/meta/build-sysroots.bb | 5 +
meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +--
3 files changed, 15 insertions(+), 6 deletions(-)
create mode 100644 meta/classes-recipe/nospdx.bbclass
diff --git a/meta/classes-recipe/nospdx.bbclass
b/meta/classes-recipe/nospdx.bbclass
new file mode 100644
index 000..b20e28218be
--- /dev/null
+++ b/meta/classes-recipe/nospdx.bbclass
@@ -0,0 +1,13 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+deltask do_collect_spdx_deps
+deltask do_create_spdx
+deltask do_create_spdx_runtime
+deltask do_create_package_spdx
+deltask do_create_rootfs_spdx
+deltask do_create_image_spdx
+deltask do_create_image_sbom
diff --git a/meta/recipes-core/meta/build-sysroots.bb
b/meta/recipes-core/meta/build-sysroots.bb
index db05c111ab2..b0b8fb3c79a 100644
--- a/meta/recipes-core/meta/build-sysroots.bb
+++ b/meta/recipes-core/meta/build-sysroots.bb
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}"
PACKAGE_ARCH = "${MACHINE_ARCH}"
EXCLUDE_FROM_WORLD = "1"
-inherit nopackages
+inherit nopackages nospdx
deltask fetch
deltask unpack
deltask patch
@@ -17,9 +17,6 @@ deltask configure
deltask compile
deltask install
deltask populate_sysroot
-deltask create_spdx
-deltask collect_spdx_deps
-deltask create_runtime_spdx
deltask recipe_qa
do_build_warn () {
diff --git a/meta/recipes-core/meta/meta-world-pkgdata.bb
b/meta/recipes-core/meta/meta-world-pkgdata.bb
index 0438bf61387..244175ddd44 100644
--- a/meta/recipes-core/meta/meta-world-pkgdata.bb
+++ b/meta/recipes-core/meta/meta-world-pkgdata.bb
@@ -27,14 +27,13 @@ python do_collect_packagedata() {
oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
}
+inherit nospdx
deltask do_fetch
deltask do_unpack
deltask do_patch
deltask do_configure
deltask do_compile
deltask do_install
-deltask do_create_spdx
-deltask do_create_spdx_runtime
do_prepare_recipe_sysroot[deptask] = ""
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201110):
https://lists.openembedded.org/g/openembedded-core/message/201110
Mute This Topic: https://lists.openembedded.org/mt/106856883/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v4 07/10] selftest: spdx: Add SPDX 3.0 test cases
Adds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can
also be run even if the default is SPDX 3.0
Signed-off-by: Joshua Watt
---
meta/lib/oeqa/selftest/cases/spdx.py | 119 +--
1 file changed, 110 insertions(+), 9 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/spdx.py
b/meta/lib/oeqa/selftest/cases/spdx.py
index 7685a81e7fb..0ae49aec333 100644
--- a/meta/lib/oeqa/selftest/cases/spdx.py
+++ b/meta/lib/oeqa/selftest/cases/spdx.py
@@ -6,21 +6,26 @@
import json
import os
+import textwrap
+from pathlib import Path
from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var, runCmd
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd
-class SPDXCheck(OESelftestTestCase):
+class SPDX22Check(OESelftestTestCase):
@classmethod
def setUpClass(cls):
-super(SPDXCheck, cls).setUpClass()
+super().setUpClass()
bitbake("python3-spdx-tools-native")
bitbake("-c addto_recipe_sysroot python3-spdx-tools-native")
def check_recipe_spdx(self, high_level_dir, spdx_file, target_name):
-config = """
-INHERIT += "create-spdx"
-"""
+config = textwrap.dedent(
+"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "create-spdx-2.2"
+"""
+)
self.write_config(config)
deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx"
# qemux86-64 creates the directory qemux86_64
machine_dir = machine_var.replace("-", "_")
-full_file_path = os.path.join(deploy_dir, "spdx", spdx_version,
machine_dir, high_level_dir, spdx_file)
+full_file_path = os.path.join(
+deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir,
spdx_file
+)
try:
os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx"
self.assertNotEqual(report, None)
self.assertNotEqual(report["SPDXID"], None)
-python = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'nativepython3')
-validator = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'pyspdxtools')
+python = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
+"nativepython3",
+)
+validator = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
"pyspdxtools"
+)
result = runCmd("{} {} -i {}".format(python, validator, filename))
self.assertExists(full_file_path)
@@ -53,3 +65,92 @@ INHERIT += "create-spdx"
def test_spdx_base_files(self):
self.check_recipe_spdx("packages", "base-files.spdx.json",
"base-files")
+
+
+class SPDX3CheckBase(object):
+"""
+Base class for checking SPDX 3 based tests
+"""
+
+def check_spdx_file(self, filename):
+import oe.spdx30
+
+self.assertExists(filename)
+
+# Read the file
+objset = oe.spdx30.SHACLObjectSet()
+with open(filename, "r") as f:
+d = oe.spdx30.JSONLDDeserializer()
+d.read(f, objset)
+
+return objset
+
+def check_recipe_spdx(self, target_name, spdx_path, *, task=None):
+config = textwrap.dedent(
+f"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "{self.SPDX_CLASS}"
+"""
+)
+self.write_config(config)
+
+if task:
+bitbake(f"-c {task} {target_name}")
+else:
+bitbake(target_name)
+
+filename = spdx_path.format(
+**get_bb_vars(
+[
+"DEPLOY_DIR_IMAGE",
+"DEPLOY_DIR_SPDX",
+"MACHINE",
+"MACHINE_ARCH",
+"SDKMACHINE",
+"SDK_DEPLOY",
+"SPDX_VERSION",
+"TOOLCHAIN_OUTPUTNAME",
+],
+target_name,
+)
+)
+
+return self.check_spdx_file(filename)
+
+def check_objset_missing_ids(self, objset):
+if objset.missing_ids:
+self.assertTrue(
+False,
+"The following SPDXIDs are unresolv
[OE-core][PATCH v4 05/10] binutils-cross-testsuite: Rename to binutils-testsuite
This recipe needs to be renamed because the "-cross-" substring in the
name triggers the cross architecture detection in sstate, but this
recipe is not actually a cross recipe.
Signed-off-by: Joshua Watt
---
meta/classes-global/mirrors.bbclass| 2 +-
meta/conf/distro/include/maintainers.inc | 2 +-
meta/lib/oeqa/selftest/cases/binutils.py | 2 +-
...tils-cross-testsuite_2.42.bb => binutils-testsuite_2.42.bb} | 3 +++
4 files changed, 6 insertions(+), 3 deletions(-)
rename meta/recipes-devtools/binutils/{binutils-cross-testsuite_2.42.bb =>
binutils-testsuite_2.42.bb} (95%)
diff --git a/meta/classes-global/mirrors.bbclass
b/meta/classes-global/mirrors.bbclass
index 862648eec5f..d68d30b0f2a 100644
--- a/meta/classes-global/mirrors.bbclass
+++ b/meta/classes-global/mirrors.bbclass
@@ -88,7 +88,7 @@ git://.*/.*
git://HOST/git/PATH;protocol=https \
BB_GIT_SHALLOW:pn-binutils = "1"
BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
-BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
+BB_GIT_SHALLOW:pn-binutils-testsuite = "1"
BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
BB_GIT_SHALLOW:pn-binutils-native = "1"
BB_GIT_SHALLOW:pn-nativesdk-binutils = "1"
diff --git a/meta/conf/distro/include/maintainers.inc
b/meta/conf/distro/include/maintainers.inc
index 14e0637c838..368d3de1583 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -64,7 +64,7 @@ RECIPE_MAINTAINER:pn-bind = "Unassigned
"
RECIPE_MAINTAINER:pn-binutils = "Khem Raj "
RECIPE_MAINTAINER:pn-binutils-cross-${TARGET_ARCH} = "Khem Raj
"
RECIPE_MAINTAINER:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "Khem
Raj "
-RECIPE_MAINTAINER:pn-binutils-cross-testsuite = "Khem Raj "
+RECIPE_MAINTAINER:pn-binutils-testsuite = "Khem Raj "
RECIPE_MAINTAINER:pn-binutils-crosssdk-${SDK_SYS} = "Khem Raj
"
RECIPE_MAINTAINER:pn-bison = "Chen Qi "
RECIPE_MAINTAINER:pn-blktool = "Yi Zhao "
diff --git a/meta/lib/oeqa/selftest/cases/binutils.py
b/meta/lib/oeqa/selftest/cases/binutils.py
index 1688eabe4ec..5ff263d342a 100644
--- a/meta/lib/oeqa/selftest/cases/binutils.py
+++ b/meta/lib/oeqa/selftest/cases/binutils.py
@@ -33,7 +33,7 @@ class BinutilsCrossSelfTest(OESelftestTestCase,
OEPTestResultTestCase):
features.append('CHECK_TARGETS = "{0}"'.format(suite))
self.write_config("\n".join(features))
-recipe = "binutils-cross-testsuite"
+recipe = "binutils-testsuite"
bb_vars = get_bb_vars(["B", "TARGET_SYS", "T"], recipe)
builddir, target_sys, tdir = bb_vars["B"], bb_vars["TARGET_SYS"],
bb_vars["T"]
diff --git a/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.42.bb
b/meta/recipes-devtools/binutils/binutils-testsuite_2.42.bb
similarity index 95%
rename from meta/recipes-devtools/binutils/binutils-cross-testsuite_2.42.bb
rename to meta/recipes-devtools/binutils/binutils-testsuite_2.42.bb
index 630815c7a3e..f2facd52c3a 100644
--- a/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.42.bb
+++ b/meta/recipes-devtools/binutils/binutils-testsuite_2.42.bb
@@ -1,3 +1,6 @@
+# NOTE: This recipe cannot have -cross- in the file name because it triggers
+# the cross build detection in sstate which causes it to use the wrong
+# architecture
require binutils.inc
require binutils-${PV}.inc
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201107):
https://lists.openembedded.org/g/openembedded-core/message/201107
Mute This Topic: https://lists.openembedded.org/mt/106856879/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v4 02/10] classes-recipe/baremetal-image: Add image file manifest
Downstream tasks may want to know what image files were written so write
out a manifest in do_image_complete. The format of the manifest is the
same as the one in image.bbclass
Signed-off-by: Joshua Watt
---
meta/classes-recipe/baremetal-image.bbclass | 30 +++--
1 file changed, 28 insertions(+), 2 deletions(-)
diff --git a/meta/classes-recipe/baremetal-image.bbclass
b/meta/classes-recipe/baremetal-image.bbclass
index 4e7d413626e..d7f53803e2a 100644
--- a/meta/classes-recipe/baremetal-image.bbclass
+++ b/meta/classes-recipe/baremetal-image.bbclass
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}"
IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}"
IMAGE_NAME_SUFFIX ?= ""
+IMAGE_FILE_MANIFEST_DIR = "${WORKDIR}/deploy-image-file-manifest"
+IMAGE_FILE_MANIFEST = "${IMAGE_FILE_MANIFEST_DIR}/manifest.json"
+
do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}"
do_image(){
@@ -37,8 +40,28 @@ do_image(){
install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf
${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf
}
-do_image_complete(){
-:
+python do_image_complete(){
+from pathlib import Path
+import json
+
+data = {
+"taskname": "do_image",
+"imagetype": "baremetal-image",
+"images": []
+}
+
+img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR"))
+
+for child in img_deploy_dir.iterdir():
+if not child.is_file() or child.is_symlink():
+continue
+
+data["images"].append({
+"filename": child.name,
+})
+
+with open(d.getVar("IMAGE_FILE_MANIFEST"), "w") as f:
+json.dump([data], f)
}
python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){
bb.utils.mkdirhier(sysconfdir)
execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND'))
+execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND"))
}
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_FILE_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_FILE_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201104):
https://lists.openembedded.org/g/openembedded-core/message/201104
Mute This Topic: https://lists.openembedded.org/mt/106856870/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v4 01/10] classes-recipe/image: Add image file manifest
Downstream tasks may want to know what image files were written by the
do_image family of tasks (e.g. SPDX) so have each task write out a
manifest file that describes the files it produced, then aggregate them
in do_image_complete
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image.bbclass | 58 +++
1 file changed, 58 insertions(+)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c63623..fc9b6116650 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -88,6 +88,11 @@ PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}"
IMGDEPLOYDIR = "${WORKDIR}/deploy-${PN}-image-complete"
+IMGMANIFESTDIR = "${WORKDIR}/image-task-manifest"
+
+IMAGE_FILE_MANIFEST_DIR = "${WORKDIR}/deploy-image-file-manifest"
+IMAGE_FILE_MANIFEST = "${IMAGE_FILE_MANIFEST_DIR}/manifest.json"
+
# Images are generally built explicitly, do not need to be part of world.
EXCLUDE_FROM_WORLD = "1"
@@ -277,14 +282,28 @@ fakeroot python do_image () {
execute_pre_post_process(d, pre_process_cmds)
}
do_image[dirs] = "${TOPDIR}"
+do_image[cleandirs] += "${IMGMANIFESTDIR}"
addtask do_image after do_rootfs
fakeroot python do_image_complete () {
from oe.utils import execute_pre_post_process
+from pathlib import Path
+import json
post_process_cmds = d.getVar("IMAGE_POSTPROCESS_COMMAND")
execute_pre_post_process(d, post_process_cmds)
+
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+
+data = []
+
+for manifest_path in image_manifest_dir.glob("*.json"):
+with manifest_path.open("r") as f:
+data.extend(json.load(f))
+
+with open(d.getVar("IMAGE_FILE_MANIFEST"), "w") as f:
+json.dump(data, f)
}
do_image_complete[dirs] = "${TOPDIR}"
SSTATETASKS += "do_image_complete"
@@ -292,6 +311,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_FILE_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_FILE_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
sstate_setscene(d)
@@ -507,12 +528,14 @@ python () {
d.setVar(task, '\n'.join(cmds))
d.setVarFlag(task, 'func', '1')
d.setVarFlag(task, 'fakeroot', '1')
+d.setVarFlag(task, 'imagetype', t)
d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
d.prependVarFlag(task, 'postfuncs', 'create_symlinks ')
d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
d.appendVarFlag(task, 'vardepsexclude', ' DATETIME DATE ' + '
'.join(vardepsexclude))
+d.appendVarFlag(task, 'postfuncs', ' write_image_file_manifest')
bb.debug(2, "Adding task %s before %s, after %s" % (task,
'do_image_complete', after))
bb.build.addtask(task, 'do_image_complete', after, d)
@@ -610,6 +633,41 @@ python create_symlinks() {
bb.note("Skipping symlink, source does not exist: %s -> %s" %
(dst, src))
}
+python write_image_file_manifest() {
+import json
+from pathlib import Path
+
+taskname = d.getVar("BB_CURRENTTASK")
+image_deploy_dir = Path(d.getVar('IMGDEPLOYDIR'))
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+manifest_path = image_manifest_dir / ("do_" + d.getVar("BB_CURRENTTASK") +
".json")
+
+image_name = d.getVar("IMAGE_NAME")
+image_basename = d.getVar("IMAGE_BASENAME")
+machine = d.getVar("MACHINE")
+
+subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or
"").split()
+imagetype = d.getVarFlag("do_" + taskname, 'imagetype', False)
+
+data = {
+"taskname": taskname,
+"imagetype": imagetype,
+"images": []
+}
+
+for type in subimages:
+image_filename = image_name + "." + type
+image_path = image_deploy_dir / image_filename
+if not image_path.exists():
+continue
+data["images"].append({
+"filename": image_filename,
+
[OE-core][PATCH v4 00/10] Add SPDX 3.0 support
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same time
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
Joshua Watt (10):
classes-recipe/image: Add image file manifest
classes-recipe/baremetal-image: Add image file manifest
classes/create-spdx-3.0: Add classes
classes-global/staging: Exclude do_create_spdx from automatic sysroot
extension
binutils-cross-testsuite: Rename to binutils-testsuite
classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
selftest: spdx: Add SPDX 3.0 test cases
classes-recipe: nospdx: Add class
classes/spdx-common: Move SPDX_SUPPLIER
Switch default spdx version to 3.0
meta/classes-global/mirrors.bbclass |2 +-
meta/classes-global/staging.bbclass |9 +-
meta/classes-recipe/baremetal-image.bbclass | 30 +-
meta/classes-recipe/image.bbclass | 58 +
meta/classes-recipe/image_types.bbclass |2 +
meta/classes-recipe/image_types_wic.bbclass |1 +
meta/classes-recipe/nospdx.bbclass| 13 +
meta/classes-recipe/packagegroup.bbclass |2 +
meta/classes/create-spdx-2.2.bbclass |8 +
meta/classes/create-spdx-3.0.bbclass | 1179
meta/classes/create-spdx-image-3.0.bbclass| 224 +
meta/classes/create-spdx.bbclass |2 +-
meta/classes/spdx-common.bbclass | 13 +-
meta/conf/distro/include/maintainers.inc |2 +-
meta/lib/oe/sbom30.py | 1122 +++
meta/lib/oe/spdx30.py | 5996 +
meta/lib/oeqa/selftest/cases/binutils.py |2 +-
meta/lib/oeqa/selftest/cases/spdx.py | 119 +-
meta/recipes-core/meta/build-sysroots.bb |5 +-
meta/recipes-core/meta/meta-world-pkgdata.bb |3 +-
...ite_2.42.bb => binutils-testsuite_2.42.bb} |3 +
21 files changed, 8765 insertions(+), 30 deletions(-)
create mode 100644 meta/classes-recipe/nospdx.bbclass
create mode 100644 meta/classes/create-spdx-3.0.bbclass
create mode 100644 meta/classes/create-spdx-image-3.0.bbclass
create mode 100644 meta/lib/oe/sbom30.py
create mode 100644 meta/lib/oe/spdx30.py
rename meta/recipes-devtools/binutils/{binutils-cross-testsuite_2.42.bb =>
binutils-testsuite_2.42.bb} (95%)
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201102):
https://lists.openembedded.org/g/openembedded-core/message/201102
Mute This Topic: https://lists.openembedded.org/mt/106856868/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 10/10] Switch default spdx version to 3.0
Changes the default SPDX version to 3.0 Signed-off-by: Joshua Watt --- meta/classes/create-spdx.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/create-spdx.bbclass b/meta/classes/create-spdx.bbclass index 19c6c0ff0b9..b604973ae0a 100644 --- a/meta/classes/create-spdx.bbclass +++ b/meta/classes/create-spdx.bbclass @@ -5,4 +5,4 @@ # # Include this class when you don't care what version of SPDX you get; it will # be updated to the latest stable version that is supported -inherit create-spdx-2.2 +inherit create-spdx-3.0 -- 2.43.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201100): https://lists.openembedded.org/g/openembedded-core/message/201100 Mute This Topic: https://lists.openembedded.org/mt/106851847/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 06/10] classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
Adds the variable overrides to set the SPDX image purpose for various
image types
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image_types.bbclass | 2 ++
meta/classes-recipe/image_types_wic.bbclass | 1 +
2 files changed, 3 insertions(+)
diff --git a/meta/classes-recipe/image_types.bbclass
b/meta/classes-recipe/image_types.bbclass
index 2f948ecbf88..506b9934cb7 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}"
IMAGE_CMD_TAR ?= "tar"
# ignore return code 1 "file changed as we read it" as other tasks(e.g.
do_image_wic) may be hardlinking rootfs
IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner
-cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+SPDX_IMAGE_PURPOSE:tar = "archive"
do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () {
fi
fi
}
+SPDX_IMAGE_PURPOSE:cpio = "archive"
UBI_VOLNAME ?= "${MACHINE}-rootfs"
UBI_VOLTYPE ?= "dynamic"
diff --git a/meta/classes-recipe/image_types_wic.bbclass
b/meta/classes-recipe/image_types_wic.bbclass
index cf3be909b30..86f40633ebc 100644
--- a/meta/classes-recipe/image_types_wic.bbclass
+++ b/meta/classes-recipe/image_types_wic.bbclass
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () {
mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic"
}
IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+SPDX_IMAGE_PURPOSE:wic = "diskImage"
do_image_wic[cleandirs] = "${WORKDIR}/build-wic"
PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201096):
https://lists.openembedded.org/g/openembedded-core/message/201096
Mute This Topic: https://lists.openembedded.org/mt/106851840/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 09/10] classes/spdx-common: Move SPDX_SUPPLIER
Move the SPDX_SUPPLIER variable to create-spdx-2.2 since it's format
only has meaning in SPDX 2.2 (SPDX 3 uses SPDX_PACKAGE_SUPPLIER with a
different format)
Signed-off-by: Joshua Watt
---
meta/classes/create-spdx-2.2.bbclass | 8
meta/classes/spdx-common.bbclass | 7 ---
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index 3ebf92b5e12..99061320e53 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -8,6 +8,14 @@ inherit spdx-common
SPDX_VERSION = "2.2"
+SPDX_ORG ??= "OpenEmbedded ()"
+SPDX_SUPPLIER ??= "Organization: ${SPDX_ORG}"
+SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created
from \
+this recipe. For SPDX documents create using this class during the build,
this \
+is the contact information for the person or organization who is doing the
\
+build."
+
+
def get_namespace(d, name):
import uuid
namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS,
d.getVar("SPDX_UUID_NAMESPACE"))
diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass
index 9dee325ba6b..6dfc1fd9e4c 100644
--- a/meta/classes/spdx-common.bbclass
+++ b/meta/classes/spdx-common.bbclass
@@ -37,13 +37,6 @@ SPDX_LICENSES ??= "${COREBASE}/meta/files/spdx-licenses.json"
SPDX_CUSTOM_ANNOTATION_VARS ??= ""
-SPDX_ORG ??= "OpenEmbedded ()"
-SPDX_SUPPLIER ??= "Organization: ${SPDX_ORG}"
-SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created
from \
-this recipe. For SPDX documents create using this class during the build,
this \
-is the contact information for the person or organization who is doing the
\
-build."
-
def extract_licenses(filename):
import re
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201099):
https://lists.openembedded.org/g/openembedded-core/message/201099
Mute This Topic: https://lists.openembedded.org/mt/106851845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 05/10] binutils-cross-testsuite: Rename to binutils-testsuite
This recipe needs to be renamed because the "-cross-" substring in the
name triggers the cross architecture detection in sstate, but this
recipe is not actually a cross recipe.
Signed-off-by: Joshua Watt
---
meta/classes-global/mirrors.bbclass| 2 +-
meta/conf/distro/include/maintainers.inc | 2 +-
meta/lib/oeqa/selftest/cases/binutils.py | 2 +-
...tils-cross-testsuite_2.42.bb => binutils-testsuite_2.42.bb} | 3 +++
4 files changed, 6 insertions(+), 3 deletions(-)
rename meta/recipes-devtools/binutils/{binutils-cross-testsuite_2.42.bb =>
binutils-testsuite_2.42.bb} (95%)
diff --git a/meta/classes-global/mirrors.bbclass
b/meta/classes-global/mirrors.bbclass
index 862648eec5f..d68d30b0f2a 100644
--- a/meta/classes-global/mirrors.bbclass
+++ b/meta/classes-global/mirrors.bbclass
@@ -88,7 +88,7 @@ git://.*/.*
git://HOST/git/PATH;protocol=https \
BB_GIT_SHALLOW:pn-binutils = "1"
BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
-BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
+BB_GIT_SHALLOW:pn-binutils-testsuite = "1"
BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
BB_GIT_SHALLOW:pn-binutils-native = "1"
BB_GIT_SHALLOW:pn-nativesdk-binutils = "1"
diff --git a/meta/conf/distro/include/maintainers.inc
b/meta/conf/distro/include/maintainers.inc
index 14e0637c838..368d3de1583 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -64,7 +64,7 @@ RECIPE_MAINTAINER:pn-bind = "Unassigned
"
RECIPE_MAINTAINER:pn-binutils = "Khem Raj "
RECIPE_MAINTAINER:pn-binutils-cross-${TARGET_ARCH} = "Khem Raj
"
RECIPE_MAINTAINER:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "Khem
Raj "
-RECIPE_MAINTAINER:pn-binutils-cross-testsuite = "Khem Raj "
+RECIPE_MAINTAINER:pn-binutils-testsuite = "Khem Raj "
RECIPE_MAINTAINER:pn-binutils-crosssdk-${SDK_SYS} = "Khem Raj
"
RECIPE_MAINTAINER:pn-bison = "Chen Qi "
RECIPE_MAINTAINER:pn-blktool = "Yi Zhao "
diff --git a/meta/lib/oeqa/selftest/cases/binutils.py
b/meta/lib/oeqa/selftest/cases/binutils.py
index 1688eabe4ec..5ff263d342a 100644
--- a/meta/lib/oeqa/selftest/cases/binutils.py
+++ b/meta/lib/oeqa/selftest/cases/binutils.py
@@ -33,7 +33,7 @@ class BinutilsCrossSelfTest(OESelftestTestCase,
OEPTestResultTestCase):
features.append('CHECK_TARGETS = "{0}"'.format(suite))
self.write_config("\n".join(features))
-recipe = "binutils-cross-testsuite"
+recipe = "binutils-testsuite"
bb_vars = get_bb_vars(["B", "TARGET_SYS", "T"], recipe)
builddir, target_sys, tdir = bb_vars["B"], bb_vars["TARGET_SYS"],
bb_vars["T"]
diff --git a/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.42.bb
b/meta/recipes-devtools/binutils/binutils-testsuite_2.42.bb
similarity index 95%
rename from meta/recipes-devtools/binutils/binutils-cross-testsuite_2.42.bb
rename to meta/recipes-devtools/binutils/binutils-testsuite_2.42.bb
index 630815c7a3e..f2facd52c3a 100644
--- a/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.42.bb
+++ b/meta/recipes-devtools/binutils/binutils-testsuite_2.42.bb
@@ -1,3 +1,6 @@
+# NOTE: This recipe cannot have -cross- in the file name because it triggers
+# the cross build detection in sstate which causes it to use the wrong
+# architecture
require binutils.inc
require binutils-${PV}.inc
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201095):
https://lists.openembedded.org/g/openembedded-core/message/201095
Mute This Topic: https://lists.openembedded.org/mt/106851838/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 08/10] classes-recipe: nospdx: Add class
Adds a class that allows recipes to opt out of generating SPDX
Signed-off-by: Joshua Watt
---
meta/classes-recipe/nospdx.bbclass | 13 +
meta/recipes-core/meta/build-sysroots.bb | 5 +
meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +--
3 files changed, 15 insertions(+), 6 deletions(-)
create mode 100644 meta/classes-recipe/nospdx.bbclass
diff --git a/meta/classes-recipe/nospdx.bbclass
b/meta/classes-recipe/nospdx.bbclass
new file mode 100644
index 000..b20e28218be
--- /dev/null
+++ b/meta/classes-recipe/nospdx.bbclass
@@ -0,0 +1,13 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+deltask do_collect_spdx_deps
+deltask do_create_spdx
+deltask do_create_spdx_runtime
+deltask do_create_package_spdx
+deltask do_create_rootfs_spdx
+deltask do_create_image_spdx
+deltask do_create_image_sbom
diff --git a/meta/recipes-core/meta/build-sysroots.bb
b/meta/recipes-core/meta/build-sysroots.bb
index db05c111ab2..b0b8fb3c79a 100644
--- a/meta/recipes-core/meta/build-sysroots.bb
+++ b/meta/recipes-core/meta/build-sysroots.bb
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}"
PACKAGE_ARCH = "${MACHINE_ARCH}"
EXCLUDE_FROM_WORLD = "1"
-inherit nopackages
+inherit nopackages nospdx
deltask fetch
deltask unpack
deltask patch
@@ -17,9 +17,6 @@ deltask configure
deltask compile
deltask install
deltask populate_sysroot
-deltask create_spdx
-deltask collect_spdx_deps
-deltask create_runtime_spdx
deltask recipe_qa
do_build_warn () {
diff --git a/meta/recipes-core/meta/meta-world-pkgdata.bb
b/meta/recipes-core/meta/meta-world-pkgdata.bb
index 0438bf61387..244175ddd44 100644
--- a/meta/recipes-core/meta/meta-world-pkgdata.bb
+++ b/meta/recipes-core/meta/meta-world-pkgdata.bb
@@ -27,14 +27,13 @@ python do_collect_packagedata() {
oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
}
+inherit nospdx
deltask do_fetch
deltask do_unpack
deltask do_patch
deltask do_configure
deltask do_compile
deltask do_install
-deltask do_create_spdx
-deltask do_create_spdx_runtime
do_prepare_recipe_sysroot[deptask] = ""
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201098):
https://lists.openembedded.org/g/openembedded-core/message/201098
Mute This Topic: https://lists.openembedded.org/mt/106851843/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 07/10] selftest: spdx: Add SPDX 3.0 test cases
Adds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can
also be run even if the default is SPDX 3.0
Signed-off-by: Joshua Watt
---
meta/lib/oeqa/selftest/cases/spdx.py | 119 +--
1 file changed, 110 insertions(+), 9 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/spdx.py
b/meta/lib/oeqa/selftest/cases/spdx.py
index 7685a81e7fb..0ae49aec333 100644
--- a/meta/lib/oeqa/selftest/cases/spdx.py
+++ b/meta/lib/oeqa/selftest/cases/spdx.py
@@ -6,21 +6,26 @@
import json
import os
+import textwrap
+from pathlib import Path
from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var, runCmd
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd
-class SPDXCheck(OESelftestTestCase):
+class SPDX22Check(OESelftestTestCase):
@classmethod
def setUpClass(cls):
-super(SPDXCheck, cls).setUpClass()
+super().setUpClass()
bitbake("python3-spdx-tools-native")
bitbake("-c addto_recipe_sysroot python3-spdx-tools-native")
def check_recipe_spdx(self, high_level_dir, spdx_file, target_name):
-config = """
-INHERIT += "create-spdx"
-"""
+config = textwrap.dedent(
+"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "create-spdx-2.2"
+"""
+)
self.write_config(config)
deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx"
# qemux86-64 creates the directory qemux86_64
machine_dir = machine_var.replace("-", "_")
-full_file_path = os.path.join(deploy_dir, "spdx", spdx_version,
machine_dir, high_level_dir, spdx_file)
+full_file_path = os.path.join(
+deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir,
spdx_file
+)
try:
os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx"
self.assertNotEqual(report, None)
self.assertNotEqual(report["SPDXID"], None)
-python = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'nativepython3')
-validator = os.path.join(get_bb_var('STAGING_BINDIR',
'python3-spdx-tools-native'), 'pyspdxtools')
+python = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
+"nativepython3",
+)
+validator = os.path.join(
+get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
"pyspdxtools"
+)
result = runCmd("{} {} -i {}".format(python, validator, filename))
self.assertExists(full_file_path)
@@ -53,3 +65,92 @@ INHERIT += "create-spdx"
def test_spdx_base_files(self):
self.check_recipe_spdx("packages", "base-files.spdx.json",
"base-files")
+
+
+class SPDX3CheckBase(object):
+"""
+Base class for checking SPDX 3 based tests
+"""
+
+def check_spdx_file(self, filename):
+import oe.spdx30
+
+self.assertExists(filename)
+
+# Read the file
+objset = oe.spdx30.SHACLObjectSet()
+with open(filename, "r") as f:
+d = oe.spdx30.JSONLDDeserializer()
+d.read(f, objset)
+
+return objset
+
+def check_recipe_spdx(self, target_name, spdx_path, *, task=None):
+config = textwrap.dedent(
+f"""\
+INHERIT:remove = "create-spdx"
+INHERIT += "{self.SPDX_CLASS}"
+"""
+)
+self.write_config(config)
+
+if task:
+bitbake(f"-c {task} {target_name}")
+else:
+bitbake(target_name)
+
+filename = spdx_path.format(
+**get_bb_vars(
+[
+"DEPLOY_DIR_IMAGE",
+"DEPLOY_DIR_SPDX",
+"MACHINE",
+"MACHINE_ARCH",
+"SDKMACHINE",
+"SDK_DEPLOY",
+"SPDX_VERSION",
+"TOOLCHAIN_OUTPUTNAME",
+],
+target_name,
+)
+)
+
+return self.check_spdx_file(filename)
+
+def check_objset_missing_ids(self, objset):
+if objset.missing_ids:
+self.assertTrue(
+False,
+"The following SPDXIDs are unresolv
[OE-core][PATCH v3 04/10] classes-global/staging: Exclude do_create_spdx from automatic sysroot extension
do_create_spdx is a outlier in that it doesn't need the RSS to be
extended just because it depends on do_populate_sysroot. In fact, it
only depends on do_populate_sysroot so it can see the actual recipes
sysroot, and attempting to extend the sysroot can cause problems for
some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")
As such, explicitly exclude do_create_spdx from extending the sysroot
just because it depends on do_populate_sysroot.
Signed-off-by: Joshua Watt
---
meta/classes-global/staging.bbclass | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/classes-global/staging.bbclass
b/meta/classes-global/staging.bbclass
index 3678a1b4415..ecce68d532e 100644
--- a/meta/classes-global/staging.bbclass
+++ b/meta/classes-global/staging.bbclass
@@ -652,10 +652,14 @@ python do_prepare_recipe_sysroot () {
addtask do_prepare_recipe_sysroot before do_configure after do_fetch
python staging_taskhandler() {
+EXCLUDED_TASK = (
+"do_prepare_recipe_sysroot",
+"do_create_spdx",
+)
bbtasks = e.tasklist
for task in bbtasks:
deps = d.getVarFlag(task, "depends")
-if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or
(deps and "populate_sysroot" in deps)):
+if task not in EXCLUDED_TASKS and (task == "do_configure" or (deps and
"populate_sysroot" in deps)):
d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ")
}
staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201094):
https://lists.openembedded.org/g/openembedded-core/message/201094
Mute This Topic: https://lists.openembedded.org/mt/106851837/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 02/10] classes-recipe/baremetal-image: Add image file manifest
Downstream tasks may want to know what image files were written so write
out a manifest in do_image_complete. The format of the manifest is the
same as the one in image.bbclass
Signed-off-by: Joshua Watt
---
meta/classes-recipe/baremetal-image.bbclass | 30 +++--
1 file changed, 28 insertions(+), 2 deletions(-)
diff --git a/meta/classes-recipe/baremetal-image.bbclass
b/meta/classes-recipe/baremetal-image.bbclass
index 4e7d413626e..d7f53803e2a 100644
--- a/meta/classes-recipe/baremetal-image.bbclass
+++ b/meta/classes-recipe/baremetal-image.bbclass
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}"
IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}"
IMAGE_NAME_SUFFIX ?= ""
+IMAGE_FILE_MANIFEST_DIR = "${WORKDIR}/deploy-image-file-manifest"
+IMAGE_FILE_MANIFEST = "${IMAGE_FILE_MANIFEST_DIR}/manifest.json"
+
do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}"
do_image(){
@@ -37,8 +40,28 @@ do_image(){
install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf
${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf
}
-do_image_complete(){
-:
+python do_image_complete(){
+from pathlib import Path
+import json
+
+data = {
+"taskname": "do_image",
+"imagetype": "baremetal-image",
+"images": []
+}
+
+img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR"))
+
+for child in img_deploy_dir.iterdir():
+if not child.is_file() or child.is_symlink():
+continue
+
+data["images"].append({
+"filename": child.name,
+})
+
+with open(d.getVar("IMAGE_FILE_MANIFEST"), "w") as f:
+json.dump([data], f)
}
python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){
bb.utils.mkdirhier(sysconfdir)
execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND'))
+execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND"))
}
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_FILE_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_FILE_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201092):
https://lists.openembedded.org/g/openembedded-core/message/201092
Mute This Topic: https://lists.openembedded.org/mt/106851835/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH v3 01/10] classes-recipe/image: Add image file manifest
Downstream tasks may want to know what image files were written by the
do_image family of tasks (e.g. SPDX) so have each task write out a
manifest file that describes the files it produced, then aggregate them
in do_image_complete
Signed-off-by: Joshua Watt
---
meta/classes-recipe/image.bbclass | 58 +++
1 file changed, 58 insertions(+)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c63623..fc9b6116650 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -88,6 +88,11 @@ PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}"
IMGDEPLOYDIR = "${WORKDIR}/deploy-${PN}-image-complete"
+IMGMANIFESTDIR = "${WORKDIR}/image-task-manifest"
+
+IMAGE_FILE_MANIFEST_DIR = "${WORKDIR}/deploy-image-file-manifest"
+IMAGE_FILE_MANIFEST = "${IMAGE_FILE_MANIFEST_DIR}/manifest.json"
+
# Images are generally built explicitly, do not need to be part of world.
EXCLUDE_FROM_WORLD = "1"
@@ -277,14 +282,28 @@ fakeroot python do_image () {
execute_pre_post_process(d, pre_process_cmds)
}
do_image[dirs] = "${TOPDIR}"
+do_image[cleandirs] += "${IMGMANIFESTDIR}"
addtask do_image after do_rootfs
fakeroot python do_image_complete () {
from oe.utils import execute_pre_post_process
+from pathlib import Path
+import json
post_process_cmds = d.getVar("IMAGE_POSTPROCESS_COMMAND")
execute_pre_post_process(d, post_process_cmds)
+
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+
+data = []
+
+for manifest_path in image_manifest_dir.glob("*.json"):
+with manifest_path.open("r") as f:
+data.extend(json.load(f))
+
+with open(d.getVar("IMAGE_FILE_MANIFEST"), "w") as f:
+json.dump(data, f)
}
do_image_complete[dirs] = "${TOPDIR}"
SSTATETASKS += "do_image_complete"
@@ -292,6 +311,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_FILE_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_FILE_MANIFEST_DIR}"
addtask do_image_complete after do_image before do_build
python do_image_complete_setscene () {
sstate_setscene(d)
@@ -507,12 +528,14 @@ python () {
d.setVar(task, '\n'.join(cmds))
d.setVarFlag(task, 'func', '1')
d.setVarFlag(task, 'fakeroot', '1')
+d.setVarFlag(task, 'imagetype', t)
d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
d.prependVarFlag(task, 'postfuncs', 'create_symlinks ')
d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
d.appendVarFlag(task, 'vardepsexclude', ' DATETIME DATE ' + '
'.join(vardepsexclude))
+d.appendVarFlag(task, 'postfuncs', ' write_image_file_manifest')
bb.debug(2, "Adding task %s before %s, after %s" % (task,
'do_image_complete', after))
bb.build.addtask(task, 'do_image_complete', after, d)
@@ -610,6 +633,41 @@ python create_symlinks() {
bb.note("Skipping symlink, source does not exist: %s -> %s" %
(dst, src))
}
+python write_image_file_manifest() {
+import json
+from pathlib import Path
+
+taskname = d.getVar("BB_CURRENTTASK")
+image_deploy_dir = Path(d.getVar('IMGDEPLOYDIR'))
+image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+manifest_path = image_manifest_dir / ("do_" + d.getVar("BB_CURRENTTASK") +
".json")
+
+image_name = d.getVar("IMAGE_NAME")
+image_basename = d.getVar("IMAGE_BASENAME")
+machine = d.getVar("MACHINE")
+
+subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or
"").split()
+imagetype = d.getVarFlag("do_" + taskname, 'imagetype', False)
+
+data = {
+"taskname": taskname,
+"imagetype": imagetype,
+"images": []
+}
+
+for type in subimages:
+image_filename = image_name + "." + type
+image_path = image_deploy_dir / image_filename
+if not image_path.exists():
+continue
+data["images"].append({
+"filename": image_filename,
+
[OE-core][PATCH v3 00/10] Add SPDX 3.0 support
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same time
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
Joshua Watt (10):
classes-recipe/image: Add image file manifest
classes-recipe/baremetal-image: Add image file manifest
classes/create-spdx-3.0: Add classes
classes-global/staging: Exclude do_create_spdx from automatic sysroot
extension
binutils-cross-testsuite: Rename to binutils-testsuite
classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
selftest: spdx: Add SPDX 3.0 test cases
classes-recipe: nospdx: Add class
classes/spdx-common: Move SPDX_SUPPLIER
Switch default spdx version to 3.0
meta/classes-global/mirrors.bbclass |2 +-
meta/classes-global/staging.bbclass |6 +-
meta/classes-recipe/baremetal-image.bbclass | 30 +-
meta/classes-recipe/image.bbclass | 58 +
meta/classes-recipe/image_types.bbclass |2 +
meta/classes-recipe/image_types_wic.bbclass |1 +
meta/classes-recipe/nospdx.bbclass| 13 +
meta/classes-recipe/packagegroup.bbclass |2 +
meta/classes/create-spdx-2.2.bbclass |8 +
meta/classes/create-spdx-3.0.bbclass | 1179
meta/classes/create-spdx-image-3.0.bbclass| 224 +
meta/classes/create-spdx.bbclass |2 +-
meta/classes/spdx-common.bbclass | 13 +-
meta/conf/distro/include/maintainers.inc |2 +-
meta/lib/oe/sbom30.py | 1122 +++
meta/lib/oe/spdx30.py | 5996 +
meta/lib/oeqa/selftest/cases/binutils.py |2 +-
meta/lib/oeqa/selftest/cases/spdx.py | 119 +-
meta/recipes-core/meta/build-sysroots.bb |5 +-
meta/recipes-core/meta/meta-world-pkgdata.bb |3 +-
...ite_2.42.bb => binutils-testsuite_2.42.bb} |3 +
21 files changed, 8762 insertions(+), 30 deletions(-)
create mode 100644 meta/classes-recipe/nospdx.bbclass
create mode 100644 meta/classes/create-spdx-3.0.bbclass
create mode 100644 meta/classes/create-spdx-image-3.0.bbclass
create mode 100644 meta/lib/oe/sbom30.py
create mode 100644 meta/lib/oe/spdx30.py
rename meta/recipes-devtools/binutils/{binutils-cross-testsuite_2.42.bb =>
binutils-testsuite_2.42.bb} (95%)
--
2.43.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201090):
https://lists.openembedded.org/g/openembedded-core/message/201090
Mute This Topic: https://lists.openembedded.org/mt/106851832/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH v2 0/7] Add SPDX 3.0 support
steps/15/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6837/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6881/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6906/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6892/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3501/steps/15/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/9099/steps/23/logs/stdio > > > On 19/06/2024 16:13:22-0600, Joshua Watt wrote: > > This patch series add support for SPDX 3.0 and sets it as the default. > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > the same time; users may choose to specifically use SPDX 3.0 instead of > > 2.2 by doing INHERIT += "create-spdx-3.0" > > > > v2: Added tests and addressed feedback > > > > > > Joshua Watt (7): > > classes-recipe/image: Add image file manifest > > classes/create-spdx-3.0: Add class > > classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images > > selftest: spdx: Add SPDX 3.0 test cases > > classes-recipe: nospdx: Add class > > classes/spdx-common: Move SPDX_SUPPLIER > > Switch default spdx version to 3.0 > > > > meta/classes-recipe/image.bbclass| 58 + > > meta/classes-recipe/image_types.bbclass |2 + > > meta/classes-recipe/image_types_wic.bbclass |1 + > > meta/classes-recipe/nospdx.bbclass | 13 + > > meta/classes-recipe/packagegroup.bbclass |2 + > > meta/classes/create-spdx-2.2.bbclass |8 + > > meta/classes/create-spdx-3.0.bbclass | 1116 > > meta/classes/create-spdx-image-3.0.bbclass | 203 + > > meta/classes/create-spdx.bbclass |2 +- > > meta/classes/spdx-common.bbclass | 32 +- > > meta/lib/oe/sbom30.py| 1118 > > meta/lib/oe/spdx30.py| 5996 ++ > > meta/lib/oeqa/selftest/cases/spdx.py | 119 +- > > meta/recipes-core/meta/build-sysroots.bb |5 +- > > meta/recipes-core/meta/meta-world-pkgdata.bb |3 +- > > 15 files changed, 8654 insertions(+), 24 deletions(-) > > create mode 100644 meta/classes-recipe/nospdx.bbclass > > create mode 100644 meta/classes/create-spdx-3.0.bbclass > > create mode 100644 m: failures:eta/classes/create-spdx-image-3.0.bbclass > > create mode 100644 meta/lib/oe/sbom30.py > > create mode 100644 meta/lib/oe/spdx30.py > > > > -- > > 2.43.2 > > > > > > > > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201032): https://lists.openembedded.org/g/openembedded-core/message/201032 Mute This Topic: https://lists.openembedded.org/mt/106770352/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
