Re: [ptxdist] [PATCH] xz: add extra URL
Hi Bruno. I bumped xz last week. I think it's hosted on github from now on. https://lore.ptxdist.org/ptxdist/20240225143514.2406777-1-christian.me...@t2data.com/T/#m2a3fe2fdc489cf841e388f18b5f9e38c7ccfa408 On 3/4/24 20:40, Bruno Thomsen wrote: > Add an additional github URL that is currently used. > It will then act as fallback so we don't rely on 302 Found Location > from primary URL. > > Suggested-by: Roland Hieber > Signed-off-by: Bruno Thomsen > --- > rules/xz.make | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/rules/xz.make b/rules/xz.make > index f24a2ac03..969eed4bd 100644 > --- a/rules/xz.make > +++ b/rules/xz.make > @@ -18,7 +18,9 @@ XZ_VERSION := 5.4.4 > XZ_MD5 := fbb849a27e266964aefe26bad508144f > XZ := xz-$(XZ_VERSION) > XZ_SUFFIX:= tar.bz2 > -XZ_URL := https://tukaani.org/xz/$(XZ).$(XZ_SUFFIX) > +XZ_URL := \ > + https://tukaani.org/xz/$(XZ).$(XZ_SUFFIX) \ > + > https://github.com/tukaani-project/xz/releases/download/v$(XZ_VERSION)/$(XZ).$(XZ_SUFFIX) > XZ_SOURCE:= $(SRCDIR)/$(XZ).$(XZ_SUFFIX) > XZ_DIR := $(BUILDDIR)/$(XZ) > XZ_LICENSE := public_domain AND LGPL-2.1-or-later AND GPL-2.0-or-later AND > GPL-3.0-or-later > > base-commit: 6bf08f784b8d41b63ba1d322e79145ad670b09e2
[ptxdist] [PATCH] xz: add extra URL
Add an additional github URL that is currently used. It will then act as fallback so we don't rely on 302 Found Location from primary URL. Suggested-by: Roland Hieber Signed-off-by: Bruno Thomsen --- rules/xz.make | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rules/xz.make b/rules/xz.make index f24a2ac03..969eed4bd 100644 --- a/rules/xz.make +++ b/rules/xz.make @@ -18,7 +18,9 @@ XZ_VERSION:= 5.4.4 XZ_MD5 := fbb849a27e266964aefe26bad508144f XZ := xz-$(XZ_VERSION) XZ_SUFFIX := tar.bz2 -XZ_URL := https://tukaani.org/xz/$(XZ).$(XZ_SUFFIX) +XZ_URL := \ + https://tukaani.org/xz/$(XZ).$(XZ_SUFFIX) \ + https://github.com/tukaani-project/xz/releases/download/v$(XZ_VERSION)/$(XZ).$(XZ_SUFFIX) XZ_SOURCE := $(SRCDIR)/$(XZ).$(XZ_SUFFIX) XZ_DIR := $(BUILDDIR)/$(XZ) XZ_LICENSE := public_domain AND LGPL-2.1-or-later AND GPL-2.0-or-later AND GPL-3.0-or-later base-commit: 6bf08f784b8d41b63ba1d322e79145ad670b09e2 -- 2.43.2
Re: [ptxdist] [PATCH 3/3] RFC: sbom_report: Add support
On Mon, Feb 19, 2024 at 05:56:17PM +0100, Simon Falsig wrote:
> This provides support for building SBOMs in CycloneDX format.
>
> A target is added alongside the other reports, that (based on the
> fast-bsp-report) extracts name, version, cpe and license of each target
> package, and puts these into a final sbom-report in CycloneDX/JSON
> format.
>
> This requires a working Python3 setup with the cyclonedx-python-lib
> package installed.
> ---
> bin/ptxdist | 3 +-
> rules/host-system-python3.in | 3 ++
> rules/host-system-python3.make | 6
> rules/post/ptxd_make_report.make | 16 +++--
> rules/sbom-report.in | 9 +
> scripts/lib/ptxd_make_report.sh | 16 +
> scripts/lib/ptxd_make_sbom_report.py | 52
> 7 files changed, 102 insertions(+), 3 deletions(-)
> create mode 100644 rules/sbom-report.in
> create mode 100644 scripts/lib/ptxd_make_sbom_report.py
>
> diff --git a/bin/ptxdist b/bin/ptxdist
> index 77cad673b..9e62d2bc0 100755
> --- a/bin/ptxdist
> +++ b/bin/ptxdist
> @@ -792,6 +792,7 @@ Misc:
>full-bsp-reportgenerate a yaml file that describes the BSP and
> all packages. More data but will build all
> packages if necessary.
> + sbom-reportgenerate a CycloneDX json SBOM
So I'm not so sure if I want a separate command for this. I think it would
be better to build this with the images.
>print print the contents of a variable, in
> the way
> it is known by "make"
>printnext assumes that the contents of is another
> @@ -1841,7 +1842,7 @@ EOF
> ptxd_make_log export_src EXPORTDIR="${1}"
> exit
> ;;
> - fast-bsp-report|full-bsp-report)
> + fast-bsp-report|full-bsp-report|sbom-report)
> check_premake_compiler &&
> ptxd_make_log "${cmd}"
> exit
> diff --git a/rules/host-system-python3.in b/rules/host-system-python3.in
> index 25201e93f..c158a295f 100644
> --- a/rules/host-system-python3.in
> +++ b/rules/host-system-python3.in
> @@ -35,4 +35,7 @@ config HOST_SYSTEM_PYTHON3_PYELFTOOLS
> config HOST_SYSTEM_PYTHON3_PYYAML
> bool
>
> +config HOST_SYSTEM_PYTHON3_CYCLONEDX
> + bool
> +
> endif
> diff --git a/rules/host-system-python3.make b/rules/host-system-python3.make
> index 3688cf09d..f4979a453 100644
> --- a/rules/host-system-python3.make
> +++ b/rules/host-system-python3.make
> @@ -88,6 +88,12 @@ ifdef PTXCONF_HOST_SYSTEM_PYTHON3_PYYAML
> ptxd_bailout "Python pyyaml module not found! \
> Please install python3-yaml (debian)";
> endif
> +ifdef PTXCONF_HOST_SYSTEM_PYTHON3_CYCLONEDX
> + @echo "Checking for Python cyclonedx-python-lib ..."
> + @$(SYSTEMPYTHON3) -c 'import cyclonedx.factory' 2>/dev/null || \
> + ptxd_bailout "Python cyclonedx-python-lib module not found! \
> + Please install cyclonedx-python-lib (pip)";
> +endif
>
> @$(call touch)
>
This part (or whatever replaces this) should be a separate patch.
> diff --git a/rules/post/ptxd_make_report.make
> b/rules/post/ptxd_make_report.make
> index eecd2a577..529b1c78e 100644
> --- a/rules/post/ptxd_make_report.make
> +++ b/rules/post/ptxd_make_report.make
> @@ -10,7 +10,10 @@ ptx/report-env = \
> $(image/env) \
> ptx_report_target="$(strip $(1))" \
> ptx_packages_selected="$(filter-out
> $(IMAGE_PACKAGES),$(PTX_PACKAGES_SELECTED))" \
> - ptx_image_packages="$(IMAGE_PACKAGES)"
> + ptx_image_packages="$(IMAGE_PACKAGES)" \
> + ptx_target_packages="$(PACKAGES)" \
This should not be needed. See below.
> + ptx_system_python3="$(SYSTEMPYTHON3)"
> +
Just the normal she-bang should to the right thing.
>
> PHONY += full-bsp-report
> full-bsp-report: $(RELEASEDIR)/full-bsp-report.yaml
> @@ -26,13 +29,22 @@ $(RELEASEDIR)/full-bsp-report.yaml: \
> @$(call ptx/report-env, $@) ptxd_make_full_bsp_report
> @$(call finish)
>
> +
> PHONY += fast-bsp-report
> fast-bsp-report: $(RELEASEDIR)/fast-bsp-report.yaml
>
> -
Unrelated changes.
> $(RELEASEDIR)/fast-bsp-report.yaml: $(addprefix $(STATEDIR)/,$(addsuffix
> .fast-report,$(PTX_PACKAGES_SELECTED)))
> @$(call targetinfo)
> @$(call ptx/report-env, $@) ptxd_make_fast_bsp_report
> @$(call finish)
>
> +
> +PHONY += sbom-report
> +sbom-report: $(RELEASEDIR)/sbom-report.json
> +
> +$(RELEASEDIR)/sbom-report.json: $(STATEDIR)/host-system-python3.install
> $(addprefix $(STATEDIR)/,$(addsuffix .fast-report,$(PACKAGES)))
> + @$(call targetinfo)
> + @$(call ptx/report-env, $@) ptxd_make_sbom_report
> + @$(call finish)
> +
So I don't think this should be one target like this. The old license
documents work like this. But
Re: [ptxdist] [PATCH 1/3] RFC: ptxd_make_world: Extract CPE for packages
On Mon, Feb 19, 2024 at 05:56:15PM +0100, Simon Falsig wrote:
> If a package specifies a CPE or CPE_VENDOR and CPE_PRODUCT, this is
> extracted into the fast report for that package. If no CPE is
> specified, or not both of CPE_VENDOR and CPE_PRODUCT, then no value is
> added.
>
> By default, the existing VERSION is used, but can be overridden with
> CPE_VERSION.
>
> Constructed CPEs are validated against the official CPE regex.
>
> The CPE (Common Platform Enumerator) allows matching CVEs to specific
> packages, and see if these apply to a specific deployment.
So we need to change the plan on how to do this a bit. I've looked into
this stuff some more. Specifically what yocto is doing here. This is a bit
of a mess. Take a look at this[1]:
[...]
CVE_PRODUCT = "haxx:curl haxx:libcurl curl:curl curl:libcurl libcurl:libcurl
daniel_stenberg:curl"
[...]
There are packages that need multiple vendor/product combinations :-/. So
the product is a list. And if an entry contains a ':' then it's
:. Otherwise '*' is used for the vendor.
And please use _CVE_PRODUCT (instead of _CPE_PRODUCT) so it's called the
same.
[1]
https://git.openembedded.org/openembedded-core/tree/meta/recipes-support/curl/curl_8.6.0.bb?id=efebd6a8824769137a21674e2bfe1c059a41758a#n20
> ---
> rules/post/ptxd_make_world_common.make | 4
> scripts/lib/ptxd_make_world_report.sh | 9 +
> 2 files changed, 13 insertions(+)
>
> diff --git a/rules/post/ptxd_make_world_common.make
> b/rules/post/ptxd_make_world_common.make
> index 4b6f691b6..189bc4ec9 100644
> --- a/rules/post/ptxd_make_world_common.make
> +++ b/rules/post/ptxd_make_world_common.make
> @@ -80,6 +80,10 @@ world/env/impl = \
> pkg_PKG="$(call ptx/escape,$(1))"
> \
> pkg_pkg="$(call ptx/escape,$($(1)))"
> \
> pkg_version="$(call ptx/escape,$($(1)_VERSION))"
> \
> + pkg_cpe_vendor="$(call ptx/escape,$($(1)_CPE_VENDOR))"
> \
> + pkg_cpe_product="$(call ptx/escape,$($(1)_CPE_PRODUCT))"
> \
> + pkg_cpe_version="$(call ptx/escape,$($(1)_CPE_VERSION))"
> \
> + pkg_cpe="$(call ptx/escape,$($(1)_CPE))"
> \
> pkg_config="$(call ptx/escape,$($(1)_CONFIG))"
> \
> pkg_ref_config="$(call ptx/escape,$($(1)_REF_CONFIG))"
> \
> pkg_cargo_lock="$(call ptx/escape,$($(1)_CARGO_LOCK))"
> \
> diff --git a/scripts/lib/ptxd_make_world_report.sh
> b/scripts/lib/ptxd_make_world_report.sh
> index 2c02e81f7..37fa2b89e 100644
> --- a/scripts/lib/ptxd_make_world_report.sh
> +++ b/scripts/lib/ptxd_make_world_report.sh
> @@ -72,6 +72,15 @@ ptxd_make_world_report_yaml() {
> do_list "rundeps:" "${pkg_run_deps}"
> do_echo "config:" "${pkg_config}"
> do_echo "version:" "${pkg_version}"
> +if [ ! -n "${pkg_cpe_version}" -a ! -n "${pkg_cpe}" ]; then
> + # Default to using pkg_version for the CPE string, unless _CPE_VERSION
> or _CPE are explicitly
> + # specified. In the case of the latter, there's no need to keep track
> of the version separately.
> + pkg_cpe_version="${pkg_version}"
> +fi
don't do this here. This fallback should be handled in the script.
> +do_echo "cpe:" "${pkg_cpe}"
This should be a list in case we need more than one.
> +do_echo "cpe_vendor:" "${pkg_cpe_vendor}"
Same here as described above.
Michael
> +do_echo "cpe_product:" "${pkg_cpe_product}"
> +do_echo "cpe_version:" "${pkg_cpe_version}"
> do_list "url:" "${pkg_url}"
> do_echo "md5:" "${pkg_md5}"
> do_echo "source:" "${pkg_src}"
> --
> 2.25.1
>
>
>
--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0|
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
Re: [ptxdist] SBOM support
Hi, On Mon, Feb 19, 2024 at 04:54:16PM +, Simon Falsig wrote: > > I'd be happy to get a bit of initial feedback on the approach. I'll have a > > look at putting up some initial patches in the coming days too. > > > > Thanks in advance and best regards, > > Sorry for the silence around this, but I've been busy with other things in > the last months. No problem. That's just the way it works. I've been looking into SBOMs as well but my current focus is spdx. > Finally managed to get something working, that integrates with the existing > host-system-python3 handling. I'll be sending some patches right after this. > > Main open questions would be: > - Currently HOST_SYSTEM_PYTHON3 and (the new) HOST_SYSTEM_PYTHON3_CYCLONEDX > packages need to be enabled manually through the "enable sbom report > generation" option in PTXdist options. Not sure if that is the right place > for it, or if there is a nicer way of handling it? (for instance, if it's > not enabled, 'ptxdist sbom-report' will just fail with a not-so-helpful > error message...) Hmm, maybe we can do something with a lazy package. That would require a real package, that actually builds the stuff, but I prefer that anyways. Otherwise we'll just add an option next to PROJECT_GENERATE_REPORTS that selects this. I'm not sure yet, let me worry about that part. > - It looks a bit like a local venv is being set up (in sysroot-host), but > I can't really figure out how to use it. A nice change could be to install > whatever host-system packages are needed in that venv automatically? Right > now the functionality requires users to manually install the required > python library with pip. So this stuff is not packaged anywhere and as far as I can tell it's not on pypi either, so I really prefer to package this locally. And yes, PTXdist now installs this stuff into a venv. Use host-meson as an example. It is installed into the venv. You may not need all the dependencies, it depends a bit on the package. The installation into the venv should happen automatically. I'll reply to the patches for more stuff. Michael -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
