Re: [OE-core] Removing Github release SRC_URIs from oe-core recipes?
Ah nice, that is definitely something we would be interested in! Alberto, would you be so kind to share some details? On Mon, 2024-01-15 at 15:12 +0100, Alexander Kanavin wrote: > On Mon, 15 Jan 2024 at 15:05, Alexander Kanavin via > lists.openembedded.org > > wrote: > > > > On Mon, 15 Jan 2024 at 15:03, Jasper Orschulko > > wrote: > > > > > > > Sadly it wouldn't. Auto-generated github archives are known to > > > > be > > > > non-deterministic, and we even have a qa check to ensure no > > > > recipe is > > > > using them. I didn't raise this point because my objections are > > > > on > > > > the > > > > principle of using release tarballs, not this technicality. > > > > > > Ah ok, that is interesting. Could you please elaborate on the > > > non- > > > deterministic nature of the github archives? This information > > > could be > > > vital for the Osselot project as well, as they currently use said > > > archives as the data basis. > > > > https://lwn.net/Articles/921787/ covers the subject well, I think. > > Oh, and before I forget: Alberto (in cc) has recently done some work > in bitbake to enable file-level source tracing, presumably for exact > same purpose as you are, so I'd suggest you discuss what kind of > approach he envisions (or maybe has implemented). > > Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193684): https://lists.openembedded.org/g/openembedded-core/message/193684 Mute This Topic: https://lists.openembedded.org/mt/103730186/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Removing Github release SRC_URIs from oe-core recipes?
Thanks, good to know! Well, that buries that idea then I guess... we will have to somehow deal with the release tarballs then I guess. On Mon, 2024-01-15 at 15:05 +0100, Alexander Kanavin wrote: > On Mon, 15 Jan 2024 at 15:03, Jasper Orschulko > wrote: > > > > > Sadly it wouldn't. Auto-generated github archives are known to be > > > non-deterministic, and we even have a qa check to ensure no > > > recipe is > > > using them. I didn't raise this point because my objections are > > > on > > > the > > > principle of using release tarballs, not this technicality. > > > > Ah ok, that is interesting. Could you please elaborate on the non- > > deterministic nature of the github archives? This information could > > be > > vital for the Osselot project as well, as they currently use said > > archives as the data basis. > > https://lwn.net/Articles/921787/ covers the subject well, I think. > > Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193683): https://lists.openembedded.org/g/openembedded-core/message/193683 Mute This Topic: https://lists.openembedded.org/mt/103730186/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Removing Github release SRC_URIs from oe-core recipes?
> Sadly it wouldn't. Auto-generated github archives are known to be > non-deterministic, and we even have a qa check to ensure no recipe is > using them. I didn't raise this point because my objections are on > the > principle of using release tarballs, not this technicality. Ah ok, that is interesting. Could you please elaborate on the non- deterministic nature of the github archives? This information could be vital for the Osselot project as well, as they currently use said archives as the data basis. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193678): https://lists.openembedded.org/g/openembedded-core/message/193678 Mute This Topic: https://lists.openembedded.org/mt/103730186/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Removing Github release SRC_URIs from oe-core recipes?
Hi Etienne, GitHub also provides tarballs for the unmodified source code, e.g.:https://github.com/libexpat/libexpat/archive/refs/tags/R_2_5_0.tar.gz This corresponds to the "Source Code" asset that Github automatically adds to releases, see: https://github.com/libexpat/libexpat/releases/tag/R_2_5_0 So if download speeds are a concern, using the source archive tars would be a valid alternative. Best, Jasper On Mon, 2024-01-15 at 10:15 +0100, Etienne Cordonnier wrote: > Hi Jasper, > one thing to keep in mind is that for some repositories, downloading > a release tarball is significantly faster than cloning the whole > repository (since the tarball is much smaller than the git repository > containing the entire git history). > > Etienne > > On Mon, Jan 15, 2024 at 7:38 AM Alexander Kanavin > wrote: > > On Mon, 15 Jan 2024 at 07:26, Alexander Kanavin via > > lists.openembedded.org > > > > wrote: > > > > > I'm also curious about what osselot outputs that can't be done > > > with a > > > oe-core class directly? Is something missing in existing create- > > > spdx > > > classes? Here's for example what osselot provides for busybox, > > > but I > > > can't really make sense of it: > > > > > > https://github.com/Open-Source-Compliance/package-analysis/tree/main/analysed-packages/busybox/version-1.36.1 > > > > Okay, I've read the README file in that repo, and if i understood > > it > > right, the process is: > > - run fossology > > - have a human inspect the output, and correct it on a file by file > > basis (tremendous waste of time and limited developer resources > > even > > when done the 'open source way' if you ask me but whatevs) > > - place the corrected output into the above repository > > > > Do you really really need the 'human corrected' part of all this? > > It > > will never possibly cover all of the packages you need to ship and > > match all their versions. If not, then maybe using and improving > > this > > layer is a better way out? > > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_meta-2Dspdxscanner_=DwIBaQ=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp=TRpMNG5GBFN9HHhbGEuDcivc9L3M4YJoBiOCfnMpvgE= > > > > Alex > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193676): https://lists.openembedded.org/g/openembedded-core/message/193676 Mute This Topic: https://lists.openembedded.org/mt/103730186/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Removing Github release SRC_URIs from oe-core recipes?
Hi Alex, > Okay, I've read the README file in that repo, and if i understood it > right, the process is: > - run fossology > - have a human inspect the output, and correct it on a file by file > basis (tremendous waste of time and limited developer resources even > when done the 'open source way' if you ask me but whatevs) > - place the corrected output into the above repository Correct. > Do you really really need the 'human corrected' part of all this? Unfortunately we do need the "human corrected" part (I wish we wouldn't). We have industry customers (and in turn our legal department) that demand a license compliance report and clearing on a "per-file" basis. Currently available scanning tools are unfortunately way to unreliable for usage without any human interaction (believe me, we tried). Will this be stupid amount of work? Yes. Is this compliance obligations gone mad? IMO, absolutely yes. But unfortunately this seems to be the way the industry is heading. We as a supplier company are getting more and more requests of this sort from our clients (big players in the public transport and automotive industry) who won't "play" with us unless these obligations are fulfilled. I've also heared similar stories from other companies. (That's what you get when you let company lawyers go wild ️) So I'm not happy with the situation but doing the best I can, given the requirements. In our case that means sharing our license clearings (which we have to do in any case) with the open source community, in the hope that other companies have similar challenges and that we can get some crowd-sourcing going. > It will never possibly cover all of the packages you need to ship and > match all their versions. So what we are currently striving for is covering all target relevant packages (aka without any special suffixes) of a "basic linux build" (aka image-core-minimal) for LTS releases (starting with kirkstone). Additionally, meta-ossselot will have logic for re-use of other package versions (so does osselot as a whole), limiting curation to the diff. I also have some ideas on dealing with openembedded patches added to point releases (currently another pain point). Best, Jasper -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193674): https://lists.openembedded.org/g/openembedded-core/message/193674 Mute This Topic: https://lists.openembedded.org/mt/103730186/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Removing Github release SRC_URIs from oe-core recipes?
Hi all,
The TL;DR:
If no one objects, I would like to put into motion that we gradually
move all oe-core recipes SRC_URIs from Github releases to git source
code (archives) for better source code traceability.
Reasoning follows:
---
Currently, there are (at least¹) 35 recipes in poky that use Github
release packages, rather than using the unmodified git source code
(archives).
In my humble opinion, this is not a good idea, as the GitHub releases
do not necessarily align with the content of the accompanying git
repository, obscurifying the actual source code that goes into a yocto
package.
Case in point, I am currently working on a (MIT licensed) bitbake layer
for integrating Osselot (https://www.osselot.org/), a software license
curation database into the bitbake build process, in the hope of
collectively lessening the reoccuring work (and pain) of companies with
strict open source software clearance requirements.
Osselot uses the original source code for point-releases (e.g. GitHub
Source Code archive) for curating package license information on file-
level and exporting the license information in SPDX format.
My bitbake integration would then attempt to match code in the S
directory against license data within the SPDX file, verifying file
equivalence by hash.
This is when I noticed a reoccurring pattern within multiple oe-core
packages that rely on GitHub releases:
For example, the release download for libexpat
(https://github.com/libexpat/libexpat/releases/download/R_2_5_0/expat-2.5.0.tar.gz
) contains these additional files when compared to the corresponding
tag in the git repository:
"expat_config.h.in",
"configure",
"Makefile.in",
"aclocal.m4",
"expat_config.h",
"m4/ltversion.m4",
"m4/ltoptions.m4",
"m4/libtool.m4",
"m4/lt~obsolete.m4",
"m4/ltsugar.m4",
"conftools/config.guess",
"conftools/config.sub",
"conftools/missing",
"conftools/install-sh",
"conftools/ar-lib",
"conftools/ltmain.sh",
"conftools/depcomp",
"conftools/compile",
"conftools/test-driver",
"xmlwf/Makefile.in",
"lib/Makefile.in"
In this case the files are the result of pre-run autoconf (which from a
openembedded perspective do not belong here in the first place, that's
the "do_configure" tasks job), however theoretically the GitHub project
maintainer could upload anything as a "GitHub release" that does not
necessarily need to align with the actually git source code.
So not only does this make things complicated for me and/or the Osselot
curators (either I am stuck with X amount of uncurated data, or
curation efforts will have to be expanded to also include GitHub
release archives) in my specific use-case, but it also reduces the
confidence in the reproducibility. Who is to say that a build from the
git sources produces the same output of a "magic" release package?
I believe that with the amount of work oe has already put into
reproducible builds from scratch, it would be a shame if we wouldn't
expand this effort make (at least) oe-core recipes build reproducible
from git source.
What are your thoughts on this?
With best regards,
Jasper
---
¹: grep -rE "inherit.*github-releases" poky/**/*.bb | wc -l
Currently, the GITHUB_BASE_URI variable used in this class defaults to
"https://github.com/${BPN}/${BPN}/releases/;.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193628):
https://lists.openembedded.org/g/openembedded-core/message/193628
Mute This Topic: https://lists.openembedded.org/mt/103730186/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Providing detailed CPE information for CVE matching capabilities
Hello again,
I have kept been pondering on this topic for quite some time and have
after quite some thought come to the conclusion that the mismatch in
the CPE Name Matching actually is due to a wrong interpretion of the
specification on the side of the dependencytrack maintainers and by
extension myself
(see https://github.com/DependencyTrack/dependency-track/issues/2988
for more details). So all-clear on that frontline. Sorry for the fuzz.
However, I still stand by the idea of providing more detailed CPE
information (e.g. target architecture) to avoid false positives, as
well as making openembedded-specific CVE reports possible.
Also, I believe it should be possible to gather the missing pieces of
information in an automated fashion by using a migration script for the
layers by leverage of the NIST CPE API. Using the cpeMatchString
parameter and the information we already have (product name and
version), we can attempt to guess the correct Part and Vendor
parameters:
// This might fail, if there is no CPE for current version (yet).
// However, might provide more accurate results.
if API(cpeMatchString="cpe:2.3:*:*:product_name:product_version:..."):
gather_facts_from_first_result
// If no matching CPE found:
// Reduce the product_name, which might result in less accurate results
else if API(cpeMatchString="cpe:2.3:*:*:product_name:*:..."):
gather_facts_from_first_result
// CVE unknown: Either name mismatch or no entry exists for this
product.
// Manual check required.
else:
CVE_*=unknown
This will reduce manual labor to feasibility checks, rather than
information gathering.
Leveraging the same mechanism, we could also attempt to guess the CPE_*
information when invoking the recipetool for the creation of new
recipes.
This would allow for a quick adoption and flatten out the path for
making these variables required by default, in turn improving reporting
and security for openembedded products.
Cheers, Jasper
On Fri, 2023-08-25 at 09:57 -1000, Steve Sakoman wrote:
> On Fri, Aug 25, 2023 at 9:18 AM Jasper Orschulko via
> lists.openembedded.org
> wrote:
> >
> > Hi Richard,
> > hi all,
> >
> > I want to address a flaw in the current CPE generation
> > functionality in
> > openembedded, which renders the CPEs unusable in regards to the
> > minimal
> > requirements of the NIST CPE Name Matching Specification standard
> > (https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7696.pdf).
> >
> > This makes working with the generated CPEs in thirdparty software
> > close
> > to impossible, as it would require the thirdparty software to
> > implement
> > matching logic beyond the requirements defined in the NIST
> > standard.
> >
> >
> > Status quo:
> >
> > Currently, the CPE is generated from the optional CVE_PRODUCT
> > (optionally including the vendor) and CVE_VERSION variables. If
> > neither
> > are provided, CPE generation will fall back on the BPN and PV of
> > the
> > recipe.
> >
> > As a result, the generated CPE will usually look as follows:
> >
> > cpe:2.3:*:*:${BPN}:${PV}:*:*:*:*:*:*:*:*
> >
> > Or, if the CVE_PRODUCT includes a vendor:
> >
> > cpe:2.3:*:${vendor}:${product}:${PV}:*:*:*:*:*:*:*:*
> >
> >
> >
> > The Issue (The TL;DR):
> >
> > This will return zero¹ matches when doing CPE matching against CVE
> > CPEs
> > in accordance with the minimal requirements of the NIST Name
> > Matching
> > Specification. For reliable matches the components CPE should be as
> > precise as possible, i.e. not containing any "*" (ANY) values for
> > vital
> > attributes, such as: Part, Vendor, Product, Version, Target_HW.
> >
> >
> >
> > The Issue (In detail):
> >
> > To understand the issue in full detail, in depth knowledge of the
> > NIST
> > specification is required. However, I will try to provide a
> > comprehensive summary below:
> >
> > CPE matching is done using a source CPE and target CPE. Due to
> > limitations (see Table 6-2, cases 4 & 11), using the CVE CPE as a
> > source and the component CPE as a target is the only sensible way
> > (otherwise a CVE CPE containing wildcards would lead to undefined
> > behavior).
> >
> > Each attribute of the CVE CPE, will then be matched against the
> > component CPE, setting the relationship of each source attribute to
> > the
> > corresponding target attribute to either superset (⊃), equal (=),
> > subset (⊂) or disjoint (≠) (see Table 6-1, example: Table 6-3).
> >
> > The NIST standard then defines 4 required CPE Name Comparison
> > Relati
Re: [OE-core] Providing detailed CPE information for CVE matching capabilities
Hi Steve,
I don't think it makes much of a difference at this point. When looking
at the recipes in openembedded core (master), only a handful of recipes
provide the CVE vendor:
➜ meta git:(master) pwd
/home/jasper/git/openembedded-core/meta
➜ meta git:(master) grep -rP 'CVE_PRODUCT\s*=\s*"\w+:\w+'
recipes-support/curl/curl_8.2.1.bb:CVE_PRODUCT = "haxx:curl
haxx:libcurl curl:curl curl:libcurl libcurl:libcurl
daniel_stenberg:curl"
recipes-support/boost/boost.inc:CVE_PRODUCT = "boost:boost"
recipes-extended/ed/ed_1.19.bb:CVE_PRODUCT = "gnu:ed"
recipes-extended/tar/tar_1.35.bb:CVE_PRODUCT = "gnu:tar"
recipes-devtools/flex/flex_2.6.4.bb:CVE_PRODUCT = "flex_project:flex"
recipes-devtools/subversion/subversion_1.14.2.bb:CVE_PRODUCT =
"apache:subversion"
recipes-connectivity/openssl/openssl_3.1.2.bb:CVE_PRODUCT =
"openssl:openssl"
Without the patch these few recipes will indeed match _some_ CVEs.
Everything other recipe is missing a vendor attribute and will
therefore not match according to the NIST specification regardless of
the provided "a" or "*".
But yes, for what it's worth it probably does not make sense to
backport the patch at the moment, since it will just replace one
erroneous behaviour with another. I only realized this after submitting
the patch, as in my use-case (integration with dependencytrack) there
were indeed more "matches" (1 CVE on Kernel vs. 300+), but after closer
inspection it turned out that was only due to an optional fuzzy
matching feature in dependencytrack and not because the results
actually improved. Sorry for the trouble.
Cheers, Jasper
On Fri, 2023-08-25 at 09:57 -1000, Steve Sakoman wrote:
> On Fri, Aug 25, 2023 at 9:18 AM Jasper Orschulko via
> lists.openembedded.org
> wrote:
> >
> > Hi Richard,
> > hi all,
> >
> > I want to address a flaw in the current CPE generation
> > functionality in
> > openembedded, which renders the CPEs unusable in regards to the
> > minimal
> > requirements of the NIST CPE Name Matching Specification standard
> > (https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7696.pdf).
> >
> > This makes working with the generated CPEs in thirdparty software
> > close
> > to impossible, as it would require the thirdparty software to
> > implement
> > matching logic beyond the requirements defined in the NIST
> > standard.
> >
> >
> > Status quo:
> >
> > Currently, the CPE is generated from the optional CVE_PRODUCT
> > (optionally including the vendor) and CVE_VERSION variables. If
> > neither
> > are provided, CPE generation will fall back on the BPN and PV of
> > the
> > recipe.
> >
> > As a result, the generated CPE will usually look as follows:
> >
> > cpe:2.3:*:*:${BPN}:${PV}:*:*:*:*:*:*:*:*
> >
> > Or, if the CVE_PRODUCT includes a vendor:
> >
> > cpe:2.3:*:${vendor}:${product}:${PV}:*:*:*:*:*:*:*:*
> >
> >
> >
> > The Issue (The TL;DR):
> >
> > This will return zero¹ matches when doing CPE matching against CVE
> > CPEs
> > in accordance with the minimal requirements of the NIST Name
> > Matching
> > Specification. For reliable matches the components CPE should be as
> > precise as possible, i.e. not containing any "*" (ANY) values for
> > vital
> > attributes, such as: Part, Vendor, Product, Version, Target_HW.
> >
> >
> >
> > The Issue (In detail):
> >
> > To understand the issue in full detail, in depth knowledge of the
> > NIST
> > specification is required. However, I will try to provide a
> > comprehensive summary below:
> >
> > CPE matching is done using a source CPE and target CPE. Due to
> > limitations (see Table 6-2, cases 4 & 11), using the CVE CPE as a
> > source and the component CPE as a target is the only sensible way
> > (otherwise a CVE CPE containing wildcards would lead to undefined
> > behavior).
> >
> > Each attribute of the CVE CPE, will then be matched against the
> > component CPE, setting the relationship of each source attribute to
> > the
> > corresponding target attribute to either superset (⊃), equal (=),
> > subset (⊂) or disjoint (≠) (see Table 6-1, example: Table 6-3).
> >
> > The NIST standard then defines 4 required CPE Name Comparison
> > Relations
> > (see Table 6-4), with two of them relevant for what we would
> > consider a
> > "match" between a CVE CPE and a component CPE:
> >
> > If all attribute relations are EQUAL (=) -> Then CPE name relation
> > is
[OE-core] Providing detailed CPE information for CVE matching capabilities
to give some insights on why I believe that improvements to the openembedded CPE generation process, as well as to information provided by the recipes (even though a diligent task) are vital. I am looking forward to your feedback on this. Cheers, Jasper ¹: This is also partly due to my commit https://git.openembedded.org/openembedded-core/commit/?id=e7c1def3c3c3a72249802ef6fb64292277a7a53e which, while technically correct, makes a bad situation even worse. Still, only a handful of recipes which happen to be of type "application" and explicitly set the "vendor" attribute, had the chance to successfully match some (but not all) CVE CPEs. False negatives would still lie at > 99%. -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Ostendstraße 1-14 | 12459 Berlin https://iris-sensing.com/ -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186724): https://lists.openembedded.org/g/openembedded-core/message/186724 Mute This Topic: https://lists.openembedded.org/mt/100962508/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [mickledore][PATCH] cve_check: Fix cpe_id generation
From: Jasper Orschulko
Use "*" (wildcard) instead of "a" (application)in cpe_id generation,
as the product is not necessarily of type application, e.g.
linux_kernel, which is of type "o" (operating system).
(From OE-Core rev: cae9528b002c06143bf048b991b9d7e93968cb6b)
Signed-off-by: Jasper Orschulko
Signed-off-by: Luca Ceresoli
Signed-off-by: Richard Purdie
---
meta/lib/oe/cve_check.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py
index dbaa0b373a..22d7e7c205 100644
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -149,7 +149,7 @@ def get_cpe_ids(cve_product, version):
else:
vendor = "*"
-cpe_id = 'cpe:2.3:a:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
+cpe_id = 'cpe:2.3:*:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
cpe_ids.append(cpe_id)
return cpe_ids
--
2.41.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186521):
https://lists.openembedded.org/g/openembedded-core/message/186521
Mute This Topic: https://lists.openembedded.org/mt/100899248/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [dunfell][PATCH] cve_check: Fix cpe_id generation
From: Jasper Orschulko
Use "*" (wildcard) instead of "a" (application)in cpe_id generation,
as the product is not necessarily of type application, e.g.
linux_kernel, which is of type "o" (operating system).
(From OE-Core rev: cae9528b002c06143bf048b991b9d7e93968cb6b)
Signed-off-by: Jasper Orschulko
Signed-off-by: Luca Ceresoli
Signed-off-by: Richard Purdie
---
meta/lib/oe/cve_check.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py
index c508865738..aada53bf38 100644
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -168,7 +168,7 @@ def get_cpe_ids(cve_product, version):
else:
vendor = "*"
-cpe_id = 'cpe:2.3:a:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
+cpe_id = 'cpe:2.3:*:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
cpe_ids.append(cpe_id)
return cpe_ids
--
2.41.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186520):
https://lists.openembedded.org/g/openembedded-core/message/186520
Mute This Topic: https://lists.openembedded.org/mt/100899227/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [kirkstone][PATCH] cve_check: Fix cpe_id generation
From: Jasper Orschulko
Use "*" (wildcard) instead of "a" (application)in cpe_id generation,
as the product is not necessarily of type application, e.g.
linux_kernel, which is of type "o" (operating system).
(From OE-Core rev: cae9528b002c06143bf048b991b9d7e93968cb6b)
Signed-off-by: Jasper Orschulko
Signed-off-by: Luca Ceresoli
Signed-off-by: Richard Purdie
---
meta/lib/oe/cve_check.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py
index 42a77872e9..06d3c6dbda 100644
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -143,7 +143,7 @@ def get_cpe_ids(cve_product, version):
else:
vendor = "*"
-cpe_id = 'cpe:2.3:a:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
+cpe_id = 'cpe:2.3:*:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
cpe_ids.append(cpe_id)
return cpe_ids
--
2.41.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186519):
https://lists.openembedded.org/g/openembedded-core/message/186519
Mute This Topic: https://lists.openembedded.org/mt/100899194/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] cve_check: Fix cpe_id generation
Hi Luca, thanks for the heads-up. That's curious, I assumed this would not be a problem as my signing email address is the same as the from address? (mbox) Adding cc: Jasper Orschulko from line 'From: Jasper Orschulko ' (body) Adding cc: Jasper Orschulko from line 'Signed-off-by: Jasper Orschulko ' Oh well, I added an explicit sendemail.from configuration to my gitconfig. I will send backport patches once this is merged in master, so we'll see if that had the desired effect. Jasper On 2023-08-21 14:10, Luca Ceresoli wrote: Hello Jasper, On Mon, 21 Aug 2023 14:02:30 +0200 "Jasper Orschulko via lists.openembedded.org" wrote: As you can see your sender address has been mangled, and as a result the patch is rejected by the the openembedded git server. This is not your fault, but we need you to modify your git configuration to prevent this from happening in the future. Have a look at the wiki for more info and how to solve that: https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded#Fixing_your_From_identity I'm taking your patch for testing on the autobuilders fixing it manually so you don't need to resend your patch this time. Luca -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186439): https://lists.openembedded.org/g/openembedded-core/message/186439 Mute This Topic: https://lists.openembedded.org/mt/100871306/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] cve_check: Fix cpe_id generation
Use "*" (wildcard) instead of "a" (application)in cpe_id generation,
as the product is not necessarily of type application, e.g.
linux_kernel, which is of type "o" (operating system).
Signed-off-by: Jasper Orschulko
---
meta/lib/oe/cve_check.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py
index 5bf3caac47..3979d521d1 100644
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -156,7 +156,7 @@ def get_cpe_ids(cve_product, version):
else:
vendor = "*"
-cpe_id = 'cpe:2.3:a:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
+cpe_id = 'cpe:2.3:*:{}:{}:{}:*:*:*:*:*:*:*'.format(vendor, product,
version)
cpe_ids.append(cpe_id)
return cpe_ids
--
2.41.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186433):
https://lists.openembedded.org/g/openembedded-core/message/186433
Mute This Topic: https://lists.openembedded.org/mt/100871306/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] repo: upgrade 2.22 -> 2.23
Signed-off-by: Jasper Orschulko
---
meta/recipes-devtools/repo/{repo_2.22.bb => repo_2.23.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-devtools/repo/{repo_2.22.bb => repo_2.23.bb} (95%)
diff --git a/meta/recipes-devtools/repo/repo_2.22.bb
b/meta/recipes-devtools/repo/repo_2.23.bb
similarity index 95%
rename from meta/recipes-devtools/repo/repo_2.22.bb
rename to meta/recipes-devtools/repo/repo_2.23.bb
index a99d5d7070..0b4e9743a6 100644
--- a/meta/recipes-devtools/repo/repo_2.22.bb
+++ b/meta/recipes-devtools/repo/repo_2.23.bb
@@ -12,7 +12,7 @@ LIC_FILES_CHKSUM =
"file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main \
file://0001-python3-shebang.patch \
"
-SRCREV = "cc879a97c3e2614d19b15b4661c3cab4d33139c9"
+SRCREV = "d56e2eb4216827284220fcc35af42e60b4faaea6"
MIRRORS += "git://gerrit.googlesource.com/git-repo.git
git://github.com/GerritCodeReview/git-repo.git"
--
2.35.3
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164577):
https://lists.openembedded.org/g/openembedded-core/message/164577
Mute This Topic: https://lists.openembedded.org/mt/90526311/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][dunfell 16/18] cmake: FindGTest: Add target for gmock library
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi Eero,
my first reflex was to say, that I don't believe this to be necessary.
As you already said, these variables aren't documented and thus I'd
argue that you don't have to consider them for maintaining backwards
compatibility during backports. I personally see this as a bug on our
part and as a coincidence that our setup worked pre-patch.
However, I'm not sure WHY those two variables where actually used by us
or where they originated from. Going as far back as cmake 3.0, these
variables where never documented, however doing a quick search for
those reveals that they actually are used in other (older?) projects.
So I'm not quite sure how many other users might be affected by this.
Maybe we actually SHOULD keep supporting them for the LTS branch, just
in case?
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Mon, 2021-12-13 at 15:03 +0200, Eero Aaltonen wrote:
> On Fri, 2021-12-10 at 18:49 +, Jasper Orschulko wrote:
> > Hello everyone,
> >
> > after some digging we identified the issue to be on our part.
> > We have been using "GTEST_LIBRARY" and "GTEST_MAIN_LIBRARY" in our
> > CMake scripts instead of "GTEST_LIBRARIES" and
> > "GTEST_MAIN_LIBRARIES",
> > as described in the cmake docs:
> >
> https://cmake.org/cmake/help/v3.16/module/FindGTest.html#result-variables
>
> Thank you for providing more details.
>
> The gmock target can be backported to dunfell in an alternative way,
> by
> leaving out the
> find_package(GTest QUIET NO_MODULE)
>
> call and subsequent lines and letting the FindModule create all the
> definitions. This would leave also the undocumented variables intact,
> if that is desired.
>
> This should be as simple as dropping the lines
> --
> # first specifically look for the CMake version of GTest
> find_package(GTest QUIET NO_MODULE)
>
> # if we found the GTest cmake package then we are done, and
> # can print what we found and return.
> if(GTest_FOUND)
> set(GTEST_FOUND ${GTest_FOUND})
> FIND_PACKAGE_HANDLE_STANDARD_ARGS(GTest HANDLE_COMPONENTS
> CONFIG_MODE)
>
> set(GTEST_LIBRARIES GTest::gtest)
> set(GTEST_MAIN_LIBRARIES GTest::gtest_main)
>
> __gtest_define_backwards_compatible_library_targets()
>
> return()
> endif()
> --
>
-BEGIN PGP SIGNATURE-
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmG3hbsACgkQYgqew07V
MNW6Gwf8DcYj0KlqNSLpB3tlsplI4Wac9mfxAd/g2BoKcyekRK2IArMJKzNDEhi6
GqKar0z3QgRZQrF54X63/MnhaqIypHef+8YyUPVVYhq+HUZSXc+6c3SjIdVBBJn8
G04/pxmoLybBvqMmFcP67qhFlojo7yW2viK8stdVZ49VwfE3XIq3fSxrqE6xQq1V
nOoZw5DRLqgCGpeB41OebYXE1bOIX3WOgk5Pzkuf4Iq9bLDoWg5zXSxw0djYCMZS
2E6kHIwJavEVKhmQoC2IpiPrZ4cJQ57JZvIR95HsFkG/C/N6sKXlJu2VLId+5xdb
IwUexGr3vUCXGqrUTWOZE6kxhdo74Q==
=/Myf
-END PGP SIGNATURE-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159656):
https://lists.openembedded.org/g/openembedded-core/message/159656
Mute This Topic: https://lists.openembedded.org/mt/87483225/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][dunfell 16/18] cmake: FindGTest: Add target for gmock library
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hello everyone, after some digging we identified the issue to be on our part. We have been using "GTEST_LIBRARY" and "GTEST_MAIN_LIBRARY" in our CMake scripts instead of "GTEST_LIBRARIES" and "GTEST_MAIN_LIBRARIES", as described in the cmake docs: https://cmake.org/cmake/help/v3.16/module/FindGTest.html#result-variables So all good here! :) - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Thu, 2021-12-09 at 09:32 -1000, Steve Sakoman wrote: > On Thu, Dec 9, 2021 at 7:38 AM Jasper Orschulko > wrote: > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > Hi, > > > > I can't provide any details yet, but I can say with certainty that > > this > > patch breaks our build (using the parent commit > > 746b301d37f9b7333f3d93e6fb7ea2808665df41 as refspec during > > the build worksasexpected): > > > > [...] undefined reference to `testing::* > > > > Can someone confirm this issue? Feel free to reach out for further > > details. > > I've not seen any issues in either local or autobuilder testing, so I > think we need more details! > > Steve > > > On Fri, 2021-12-03 at 08:19 -1000, Steve Sakoman wrote: > > > From: Eero Aaltonen > > > > > > `googlemock` has been absorbed into the > > > [googletest](https://github.com/google/googletest) project and is > > > built > > > and installed from the same source tree. > > > > > > `googletest` has provided a CMake Config-file Package starting > > > with > > > GTest 1.8.1. `find_package(GTest ...)` by default dispatches > > > first to > > > CMake Find Module. Starting with CMake commit > > > 2327b4330cce157d616ff8b611b3e77568d00351 in CMake v3.20.0 the > > > module > > > dispatches onward to the Config-file Package so that the same > > > targets > > > are available. In pre v3.20.0 versions of CMake however the Find > > > Module > > > masks the targets provided by the upstream `GTest` package. > > > > > > Update `Modules/FindGTest.cmake` to provide the same targets as > > > the > > > CMake Config-file Package and backwards compatible targets and > > > result > > > variables. > > > > > > Signed-off-by: Eero Aaltonen > > > Signed-off-by: Steve Sakoman > > > --- > > > .../cmake/cmake-native_3.16.5.bb | 1 + > > > ...ndGTest-Add-target-for-gmock-library.patch | 255 > > > ++ > > > 2 files changed, 256 insertions(+) > > > create mode 100644 meta/recipes-devtools/cmake/cmake/0006-cmake- > > > FindGTest-Add-target-for-gmock-library.patch > > > > > > diff --git a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > > > b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > > > index b2952ee5f5..96a7be6770 100644 > > > --- a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > > > +++ b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > > > @@ -7,6 +7,7 @@ SRC_URI += "file://OEToolchainConfig.cmake \ > > > file://environment.d-cmake.sh \ > > > > > > file://0001-CMakeDetermineSystem-use-oe-environment-vars-to-load.patch > > > \ > > > > > > file://0005-Disable-use-of-ext2fs-ext2_fs.h-by-cmake-s-internal-.patch > > > \ > > > + > > > file://0006-cmake-FindGTest-Add-target-for-gmock-library.patch \ > > > " > > > > > > > > > diff --git a/meta/recipes-devtools/cmake/cmake/0006-cmake- > > > FindGTest- > > > Add-target-for-gmock-library.patch b/meta/recipes- > > > devtools/cmake/cmake/0006-cmake-FindGTest-Add-target-for-gmock- > > > library.patch > > > new file mode 100644 > > > index 00..267f586a71 > > > --- /dev/null > > > +++ b/meta/recipes-devtools/cmake/cmake/0006-cmake-FindGTest-Add- > > > target-for-gmock-library.patch > > > @@ -0,0 +1,255 @@ > > > +From 39eae0d6c1b398f18761abac7f55944f0290f8a1 Mon Sep 17 > > > 00:00:00 > > > 2001 > > > +From: Eero Aaltonen > > > +Date: Sun, 17 Oct 2021 17:13:07 +0300 > > > +Subject: [PATCH] FindGTest: Add target for gmock library >
Re: [OE-core][dunfell 16/18] cmake: FindGTest: Add target for gmock library
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, I can't provide any details yet, but I can say with certainty that this patch breaks our build (using the parent commit 746b301d37f9b7333f3d93e6fb7ea2808665df41 as refspec during the build worksasexpected): [...] undefined reference to `testing::* Can someone confirm this issue? Feel free to reach out for further details. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Fri, 2021-12-03 at 08:19 -1000, Steve Sakoman wrote: > From: Eero Aaltonen > > `googlemock` has been absorbed into the > [googletest](https://github.com/google/googletest) project and is > built > and installed from the same source tree. > > `googletest` has provided a CMake Config-file Package starting with > GTest 1.8.1. `find_package(GTest ...)` by default dispatches first to > CMake Find Module. Starting with CMake commit > 2327b4330cce157d616ff8b611b3e77568d00351 in CMake v3.20.0 the module > dispatches onward to the Config-file Package so that the same targets > are available. In pre v3.20.0 versions of CMake however the Find > Module > masks the targets provided by the upstream `GTest` package. > > Update `Modules/FindGTest.cmake` to provide the same targets as the > CMake Config-file Package and backwards compatible targets and result > variables. > > Signed-off-by: Eero Aaltonen > Signed-off-by: Steve Sakoman > --- > .../cmake/cmake-native_3.16.5.bb | 1 + > ...ndGTest-Add-target-for-gmock-library.patch | 255 > ++ > 2 files changed, 256 insertions(+) > create mode 100644 meta/recipes-devtools/cmake/cmake/0006-cmake- > FindGTest-Add-target-for-gmock-library.patch > > diff --git a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > index b2952ee5f5..96a7be6770 100644 > --- a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > +++ b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb > @@ -7,6 +7,7 @@ SRC_URI += "file://OEToolchainConfig.cmake \ > file://environment.d-cmake.sh \ > > file://0001-CMakeDetermineSystem-use-oe-environment-vars-to-load.patch > \ > > file://0005-Disable-use-of-ext2fs-ext2_fs.h-by-cmake-s-internal-.patch > \ > + > file://0006-cmake-FindGTest-Add-target-for-gmock-library.patch \ > " > > > diff --git a/meta/recipes-devtools/cmake/cmake/0006-cmake-FindGTest- > Add-target-for-gmock-library.patch b/meta/recipes- > devtools/cmake/cmake/0006-cmake-FindGTest-Add-target-for-gmock- > library.patch > new file mode 100644 > index 00..267f586a71 > --- /dev/null > +++ b/meta/recipes-devtools/cmake/cmake/0006-cmake-FindGTest-Add- > target-for-gmock-library.patch > @@ -0,0 +1,255 @@ > +From 39eae0d6c1b398f18761abac7f55944f0290f8a1 Mon Sep 17 00:00:00 > 2001 > +From: Eero Aaltonen > +Date: Sun, 17 Oct 2021 17:13:07 +0300 > +Subject: [PATCH] FindGTest: Add target for gmock library > + > +`googlemock` has been absorbed into the > +[googletest](https://github.com/google/googletest) project and is > built > +and installed from the same source tree. > + > +As GTest may be built with or without GMock, skip GMock if it is not > +present. > + > +Do not provide result variables for GMock. They are not provided by > +upstream GTest's CMake Package Configuration File. > + > +Also update the test case to cover linking to `GTest::gmock`. > + > +The patch was imported from the Kitware git server > +(g...@gitlab.kitware.com:cmake/cmake.git) as of commit id > +50bf457a0dd857cf976b22c5be7d333493233d1e > + > +Patch was modified to support upper case variable `GTEST_FOUND`. > + > +Upstream-Status: Accepted > [https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6632] > +Milestone: 3.23.0 > + > +Signed-off-by: Eero Aaltonen > +--- > + .../dev/FindGTest-target-for-gmock.rst | 4 + > + Modules/FindGTest.cmake | 133 > +++--- > + Tests/FindGTest/Test/CMakeLists.txt | 4 + > + 3 files changed, 121 insertions(+), 20 deletions(-) > + create mode 100644 Help/release/dev/FindGTest-target-for-gmock.rst > + > +diff --git a/Help/release/dev/FindGTest-target-for-gmock.rst > b/Help/release/dev/FindGTest-target-for-gmock.rst > +new file mode 100644 > +index 00..f78242c80e > +--- /dev/null > b/Help/release/dev/FindGTest-target-for-gmock.rst > +@@ -0,0 +1,4 @@ > ++FindGTe
[oe-core][PATCH 1/2] repo: Use separate task for patching repo rev
From: Jasper Orschulko
Using a task instead of a version specific patch for setting the repo
revision within the source code. This drastically decreases the
maintenance burden and easier usage of the OE update helper.
Signed-off-by: Jasper Orschulko
---
.../0001-Set-REPO_REV-to-v2.17.3.patch| 35 ---
meta/recipes-devtools/repo/repo_2.17.3.bb | 9 +++--
2 files changed, 7 insertions(+), 37 deletions(-)
delete mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
deleted file mode 100644
index 60f1086b32..00
--- a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
+++ /dev/null
@@ -1,35 +0,0 @@
-From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
-From: Jasper Orschulko
-Date: Thu, 4 Nov 2021 16:55:12 +0100
-Subject: [PATCH] Set REPO_REV to v2.17.3
-
-repo is an unusual tool because it downloads all of its own Python modules
-using GPG-signed git tags, and stores those files as part of the project
-that it is working with.
-
-So in order to have a reproducible repo installation within the project
-folders, we hardcode the default REPO_REV to a SHA1 that corresponds to
-the version of the recipe. REPO_REV can still be overwriten by the user,
-by specifying the REPO_REV environment variable.
-
-Upstream-Status: Inappropriate [configuration]
-Signed-off-by: Jasper Orschulko
- repo | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/repo b/repo
-index 4cddbf1..5c3551f 100755
a/repo
-+++ b/repo
-@@ -144,7 +144,7 @@ if not REPO_URL:
- REPO_URL = 'https://gerrit.googlesource.com/git-repo'
- REPO_REV = os.environ.get('REPO_REV')
- if not REPO_REV:
-- REPO_REV = 'stable'
-+ REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
- # URL to file bug reports for repo tool issues.
- BUG_URL =
'https://bugs.chromium.org/p/gerrit/issues/entry?template=Repo+tool+issue'
-
---
-2.34.0
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
index f7bbb22964..aeaec13dd7 100644
--- a/meta/recipes-devtools/repo/repo_2.17.3.bb
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -12,13 +12,18 @@ LIC_FILES_CHKSUM =
"file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
-SRC_URI += "file://0001-python3-shebang.patch \
-file://0001-Set-REPO_REV-to-v2.17.3.patch"
+SRC_URI += "file://0001-python3-shebang.patch"
MIRRORS += "git://gerrit.googlesource.com/git-repo.git
git://github.com/GerritCodeReview/git-repo.git"
S = "${WORKDIR}/git"
+do_set_fixed_rev() {
+sed -Ei "s/REPO_REV\s*=\s*('|\")stable('|\")/REPO_REV = '${SRCREV}'/g"
${S}/repo
+}
+
+do_patch[postfuncs] += "do_set_fixed_rev"
+
do_install() {
install -D ${WORKDIR}/git/repo ${D}${bindir}/repo
}
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159266):
https://lists.openembedded.org/g/openembedded-core/message/159266
Mute This Topic: https://lists.openembedded.org/mt/87546668/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH 2/2] repo: upgrade 2.17.3 -> 2.18
From: Jasper Orschulko
Signed-off-by: Jasper Orschulko
---
meta/recipes-devtools/repo/{repo_2.17.3.bb => repo_2.18.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-devtools/repo/{repo_2.17.3.bb => repo_2.18.bb} (95%)
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.18.bb
similarity index 95%
rename from meta/recipes-devtools/repo/repo_2.17.3.bb
rename to meta/recipes-devtools/repo/repo_2.18.bb
index aeaec13dd7..bf5e8d0429 100644
--- a/meta/recipes-devtools/repo/repo_2.17.3.bb
+++ b/meta/recipes-devtools/repo/repo_2.18.bb
@@ -10,7 +10,7 @@ LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
-SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+SRCREV = "4a478edb443864561089b2699c9e65c85fc5e036"
SRC_URI += "file://0001-python3-shebang.patch"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159265):
https://lists.openembedded.org/g/openembedded-core/message/159265
Mute This Topic: https://lists.openembedded.org/mt/87546667/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH v5 1/2] repo: Add recipe for 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
In regards to automatic update patch generation this actually is pretty
useful. I think I'll throw an extra patch on top to integrate this
suggestion.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Thu, 2021-11-11 at 11:00 +, Jose Quaresma wrote:
>
> Jasper Orschulko via lists.openembedded.org
> escreveu no dia
> quinta, 11/11/2021 à(s) 10:21:
> > From: Jasper Orschulko
> >
> > Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
> >
> > Signed-off-by: Jasper Orschulko
> > ---
> > meta/conf/distro/include/maintainers.inc | 1 +
> > .../0001-Set-REPO_REV-to-v2.17.3.patch | 35
> > +++
> > .../repo/repo/0001-python3-shebang.patch | 26 ++
> > meta/recipes-devtools/repo/repo_2.17.3.bb | 28 +++
> > 4 files changed, 90 insertions(+)
> > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-REPO_REV-to-v2.17.3.patch
> > create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> >
> > diff --git a/meta/conf/distro/include/maintainers.inc
> > b/meta/conf/distro/include/maintainers.inc
> > index f3e0a75d56..58a0a9615f 100644
> > --- a/meta/conf/distro/include/maintainers.inc
> > +++ b/meta/conf/distro/include/maintainers.inc
> > @@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert
> > Yang "
> > RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
> > RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
> > RECIPE_MAINTAINER:pn-readline = "Hongxu Jia
> > "
> > +RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
> > "
> > RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi
> > "
> > RECIPE_MAINTAINER:pn-rgb = "Unassigned
> > "
> > RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia
> > "
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 00..285b1d3129
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,35 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the default REPO_REV to a SHA1 that
> > corresponds to
> > +the version of the recipe. REPO_REV can still be overwriten by the
> > user,
> > +by specifying the REPO_REV environment variable.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> >
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index b13e34c..31130e9 100755
> > +--- a/repo
> > b/repo
> > +@@ -130,7 +130,7 @@ if not REPO_URL:
> > + REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > + REPO_REV = os.environ.get('REPO_REV')
> > + if not REPO_REV:
> > +- REPO_REV = 'stable'
> > ++ REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
> > +
> > + # increment this whenever we make important changes to this
> > script
> > + VERSION = (2, 8)
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 00..d3888c8bb2
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
> > @@ -0,0 +1,26 @@
> > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Tue, 14 Sep 2021 16:46:51 +0200
>
Re: [OE-core] [bitbake-devel] [eo-core][PATCH v6 1/2] repo: Add recipe for 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Sorry about that. Seems I got sloppy during one of the patch revisions and accidently formatted one of the patches from the wrong repo revision. Should now work with v7 on the ML. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Wed, 2021-11-24 at 23:07 +, Richard Purdie wrote: > On Wed, 2021-11-24 at 19:35 +0100, Jasper Orschulko via > lists.openembedded.org > wrote: > > From: Jasper Orschulko > > > > Add a recipe for repo 2.17.3, prerequisite for the repo fetcher. > > > > Signed-off-by: Jasper Orschulko > > --- > > meta/conf/distro/include/maintainers.inc | 1 + > > .../0001-Set-REPO_REV-to-v2.17.3.patch | 35 > > +++ > > .../repo/repo/0001-python3-shebang.patch | 26 ++ > > meta/recipes-devtools/repo/repo_2.17.3.bb | 28 +++ > > 4 files changed, 90 insertions(+) > > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001- > > Set-REPO_REV-to-v2.17.3.patch > > create mode 100644 meta/recipes-devtools/repo/repo/0001-python3- > > shebang.patch > > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb > > > > > > I thought I'd try testing this. Unfortunately it doesn't build: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/1004/steps/13/logs/stdio > > Cheers, > > Richard > > -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGezXcACgkQYgqew07V MNVN0ggAhgCUYWOdvRt1k57dHP2aS2EGxLL9vt9LZ6Bf2OMXADinYPgfWAm2gZ2S aSdo3I0Vjlkh/rxB1uOmNpWPFL0Yitzs696vM87Jcxll9Q2xXkHBDa5Lj3Z4sUaZ jChzXVeuKmrALFThMsXbPcJO/KXYZJRh61Pl9vLw4FWpdwezcQ0mXJ4uKGCSVhvR yi/AlEaoh0TC/jsiU/VxaqHUMaviI/1rku/IdzkMWfsKy/Rt3ilSNQ0P6CbUI6kY mbKd3aYzMMsYBw9pph0yHjK3QVu6Lr+GnHH15FQ4JFOhs8yXCK1Fe1Wza9RNvE32 hGmed1mOwHHwPG4CoUV2TbwD0KD/3A== =0m9v -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158772): https://lists.openembedded.org/g/openembedded-core/message/158772 Mute This Topic: https://lists.openembedded.org/mt/87292395/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH v7 1/2] repo: Add recipe for 2.17.3
From: Jasper Orschulko
Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/conf/distro/include/maintainers.inc | 1 +
.../0001-Set-REPO_REV-to-v2.17.3.patch| 35 +++
.../repo/repo/0001-python3-shebang.patch | 26 ++
meta/recipes-devtools/repo/repo_2.17.3.bb | 28 +++
4 files changed, 90 insertions(+)
create mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
diff --git a/meta/conf/distro/include/maintainers.inc
b/meta/conf/distro/include/maintainers.inc
index f3e0a75d56..58a0a9615f 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert Yang
"
RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
RECIPE_MAINTAINER:pn-readline = "Hongxu Jia "
+RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
"
RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi "
RECIPE_MAINTAINER:pn-rgb = "Unassigned "
RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia "
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 00..60f1086b32
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,35 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the default REPO_REV to a SHA1 that corresponds to
+the version of the recipe. REPO_REV can still be overwriten by the user,
+by specifying the REPO_REV environment variable.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index 4cddbf1..5c3551f 100755
+--- a/repo
b/repo
+@@ -144,7 +144,7 @@ if not REPO_URL:
+ REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+ REPO_REV = os.environ.get('REPO_REV')
+ if not REPO_REV:
+- REPO_REV = 'stable'
++ REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
+ # URL to file bug reports for repo tool issues.
+ BUG_URL =
'https://bugs.chromium.org/p/gerrit/issues/entry?template=Repo+tool+issue'
+
+--
+2.34.0
diff --git a/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
new file mode 100644
index 00..d3888c8bb2
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
@@ -0,0 +1,26 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+Yocto does not symlink from python to python3, thus change the shebang from
+python to python3.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 00..f7bbb22964
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git
repositories, does the uploads to revision control systems, and automates parts
of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo;
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
+
+SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+
+SRC_URI += "file://0001-python3-shebang.patch \
+file://0001-Set-REPO_REV-to-v2.17.3.patch"
+
+MIRRORS += "gi
[oe-core][PATCH v7 2/2] base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko
Add repo-native as prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/classes/base.bbclass | 4
1 file changed, 4 insertions(+)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..b709777f24 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,10 @@ python () {
elif uri.scheme == "npm":
d.appendVarFlag('do_fetch', 'depends', '
nodejs-native:do_populate_sysroot')
+elif uri.scheme == "repo":
+needsrcrev = True
+d.appendVarFlag('do_fetch', 'depends', '
repo-native:do_populate_sysroot')
+
# *.lz4 should DEPEND on lz4-native for unpacking
if path.endswith('.lz4'):
d.appendVarFlag('do_unpack', 'depends', '
lz4-native:do_populate_sysroot')
--
2.34.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158770):
https://lists.openembedded.org/g/openembedded-core/message/158770
Mute This Topic: https://lists.openembedded.org/mt/87292789/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [eo-core][PATCH v6 2/2] base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko
Add repo-native as prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/classes/base.bbclass | 4
1 file changed, 4 insertions(+)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..b709777f24 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,10 @@ python () {
elif uri.scheme == "npm":
d.appendVarFlag('do_fetch', 'depends', '
nodejs-native:do_populate_sysroot')
+elif uri.scheme == "repo":
+needsrcrev = True
+d.appendVarFlag('do_fetch', 'depends', '
repo-native:do_populate_sysroot')
+
# *.lz4 should DEPEND on lz4-native for unpacking
if path.endswith('.lz4'):
d.appendVarFlag('do_unpack', 'depends', '
lz4-native:do_populate_sysroot')
--
2.34.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158736):
https://lists.openembedded.org/g/openembedded-core/message/158736
Mute This Topic: https://lists.openembedded.org/mt/87287411/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [eo-core][PATCH v6 1/2] repo: Add recipe for 2.17.3
From: Jasper Orschulko
Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/conf/distro/include/maintainers.inc | 1 +
.../0001-Set-REPO_REV-to-v2.17.3.patch| 35 +++
.../repo/repo/0001-python3-shebang.patch | 26 ++
meta/recipes-devtools/repo/repo_2.17.3.bb | 28 +++
4 files changed, 90 insertions(+)
create mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
diff --git a/meta/conf/distro/include/maintainers.inc
b/meta/conf/distro/include/maintainers.inc
index f3e0a75d56..58a0a9615f 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert Yang
"
RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
RECIPE_MAINTAINER:pn-readline = "Hongxu Jia "
+RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
"
RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi "
RECIPE_MAINTAINER:pn-rgb = "Unassigned "
RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia "
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 00..285b1d3129
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,35 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the default REPO_REV to a SHA1 that corresponds to
+the version of the recipe. REPO_REV can still be overwriten by the user,
+by specifying the REPO_REV environment variable.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..31130e9 100755
+--- a/repo
b/repo
+@@ -130,7 +130,7 @@ if not REPO_URL:
+ REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+ REPO_REV = os.environ.get('REPO_REV')
+ if not REPO_REV:
+- REPO_REV = 'stable'
++ REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
+
+ # increment this whenever we make important changes to this script
+ VERSION = (2, 8)
+--
+2.33.1
diff --git a/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
new file mode 100644
index 00..d3888c8bb2
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
@@ -0,0 +1,26 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+Yocto does not symlink from python to python3, thus change the shebang from
+python to python3.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 00..f7bbb22964
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git
repositories, does the uploads to revision control systems, and automates parts
of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo;
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
+
+SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+
+SRC_URI += "file://0001-python3-shebang.patch \
+file://0001-Set-REPO_REV-to-v2.17.3.patch"
+
+MIRRORS += "git://gerrit.googlesource.com/git-repo.git
git://git
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Richard and folks, > The only other way I could imagine this to work would require us to > keep the manifest file within the meta layer and do a somewhat > "hacky" > workaround: > > It is possible to run `repo init` on a local git folder. > As such, something like this could work: > > 1) create temp folder (e.g. /tmp/repo-X) > 2) run git init in temp folder > 3) install manifest file into temp folder > 4) add & commit > 5) run repo init -u file://$temp-folder today I discovered that repo also supports the --standalone-manifest flag, which makes the workaround described above obsolete. Thus, I believe supporting repo manifests from the layer repo only is the cleanest and easiest way to implement this and we wouldn't have to rely on cloning manifest repos during recipe parsing. However, this would be a breaking change to the current repo fetcher implementation, as it supports remote manifest repos. I am pretty confident however that no one uses the arguably already broken and unusable current repo fetcher implementation in a production workload so this shouldn't be a major issue. What are your thoughts on this? - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Thu, 2021-11-11 at 12:42 +0100, Jasper Orschulko wrote: > Hi Richard, > > > Even ten recipes using this will show a degradation in parsing > > speed > > and I do > > get a lot of complaints when parsing slows down for any reason. The > > user doesn't > > expect this and it won't be visible what bitbake is doing (sitting > > at > > 99% parsed > > for a period). > > > > Also, the "tend to be small" implies someone will create a huge one > > at some > > point even if that is a bad idea for whatever reasons, I just know > > how these > > things end up going :(. > > I believe this is something that could be addressed in the > documentation of the repo fetcher? Something along the lines of > "Using > the repo fetcher can increase the time spent in the parsing phase, as > we need to clone and inspect the manifest repository. To avoid this, > keep the size of your manifest repository at a minimum." > > When this is done, the dalay is barely noticable. We have a testsetup > with 24 recipes and reasonable parsing speeds. > > > * Does it only clone a repo in the AUTOREV case? > > No, it also clones with fixed refspecs, as we want to ensure that > each > of the git repos referenced in the manifest also uses fixed refspecs. > Otherwise, the sources in the background might change, while the > recipe > still (correctly) references the same revision of the manifest repo. > > > * Could it only obtain the manifest file somehow without a clone of > > the repo? > > Unfortunately not really. git does not offer a way to list the > content > of a file without cloning it beforehand. > > The only other way I could imagine this to work would require us to > keep the manifest file within the meta layer and do a somewhat > "hacky" > workaround: > > It is possible to run `repo init` on a local git folder. > As such, something like this could work: > > 1) create temp folder (e.g. /tmp/repo-X) > 2) run git init in temp folder > 3) install manifest file into temp folder > 4) add & commit > 5) run repo init -u file://$temp-folder > > The repo manifest could even still be used outside of yocto, by using > the metalayer repo as basis for the repo init: `repo init -u > $metalayer-repo -m $path-to-manifest-file`. Not pretty, but it works. > > So if cloning within the parser is a dealbreaker for you, this option > might be worth looking into. > -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGeYnwACgkQYgqew07V MNWXFAgAkZ7+CRZadUItM+Ay8uePffysuQhlwxHaAgzx1H+4Pk/HKK9QpEy3jFWz kODr3DPOWlf6eS+AcqpJXVYdMsTucvE34eLIWb8reFkrPQKMtxp0N6U5gqz7e+Bm Wf4lS04tqGZzC/Ld2DOs/zEnGKEwXBRHMX/9j6YvVMYxwKjxghW2LRloWpzghduH j0FTAp1aGKLhCQjqeEuNZRMT+rn0i8k6u8Lxel1wRHi2LNhyT3VD3NThjVL64acH FF83jhaxQ0JxpsSn9ouvJD3wMJHi72rdfGzznuQHmFT0lOxO5AjlcIOtK3KisiAI KQ/jEV/44WN0JO7OpmHrVyOSfPeVBA== =H0FZ -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158723): https://lists.openembedded.org/g/openembedded-core/message/158723 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Will do! :)
This only affects the other patch series though, the recipe itself
should be fine now.
It seems the only question left open is this:
> > I think we can replace the patch 0001-Set-REPO_REV-to-v2.17.3.patch
> > with a post function
> > and with it we can reuse the SRCREV of the recipe. Something like:
> >
> > do_fix_rev(){
> > sed -i "s/REPO_REV = 'stable'/REPO_REV = '${SRCREV}'" ${S}/repo
> > }
> >
> > do_patch[postfuncs] += "do_fix_rev"
>
> Yeah... was thinking about that as well. Wasn't sure though if this
> would be considered bad practise. What do the others think about
> this?
Some feedback on this would be appreciated. Thanks!
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Mon, 2021-11-15 at 14:05 +0100, Alexander Kanavin wrote:
> Thanks, maybe you should write a test for offline builds as well then
> :)
>
> Alex
>
> On Mon, 15 Nov 2021 at 13:59, Jasper Orschulko
> wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > Hi Alexander,
> >
> > "[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher"
> > contains
> > a
> > fix for this issue. Reproducing a build offline with only the
> > DL_DIR
> > should work now.
> >
> > We are currently still lacking the appropriate tests for the
> > fetcher.
> > These will follow soon.
> >
> > - --
> > With best regards
> >
> > Jasper Orschulko
> > DevOps Engineer
> >
> > Tel. +49 30 58 58 14 265
> > Fax +49 30 58 58 14 999
> > jasper.orschu...@iris-sensing.com
> >
> > • • • • • • • • • • • • • • • • • • • • • • • • • •
> >
> > iris-GmbH
> > infrared & intelligent sensors
> > Schnellerstraße 1-5 | 12439 Berlin
> >
> > https://iris-sensing.com/
> >
> >
> >
> >
> >
> > On Thu, 2021-11-11 at 20:20 +0100, Alexander Kanavin wrote:
> > > On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko
> > > wrote:
> > > > So if you have done this initial fetch of your sources and
> > > > stash
> > > > your
> > > > working dir away, you can do an offline build.
> > >
> > > But can you do an offline build without a prepopulated working
> > > dir?
> > > That's the crucial point: offline build should work in a freshly
> > set
> > > up build directory, without sstate cache, and with prepopulated
> > > DL_DIR only.
> > >
> > > Alex
> > -BEGIN PGP SIGNATURE-
> >
> > iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V
> > MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm
> > pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J
> > jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg
> > oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf
> > Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS
> > sY39fiky57aEgMe7g1OZX8oRODowPA==
> > =hpAK
> > -END PGP SIGNATURE-
-BEGIN PGP SIGNATURE-
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSXNkACgkQYgqew07V
MNUKmQf/UYTURPKZ0iY2MKPFB60WdnmsTKJMjotQlcVBUpiwQ6LhSpkfWRqOFLpN
na9cNZbYjGFeDNdL3d0ILOEQ+oVhLY7bQpj5Xb2nDsIWspjlHg1IkkvCVGa9zFb3
XALaDrGBrlad9KNhI4C2CmXjdJZgx3yBuuvK6mSiga5fs44QB/lBb9JOQGYucRt+
QeLkIatvFo1Jxzmc6tNkRr3osgacQroagHj4CISh5W+ezWqtdWHA5ndf4HvCNwJV
hkpyXtxeQDINGVGBsMnJkOIWRSStOviL66Q1h5/dgbJ23FXQuEkM9CyD19kXWSIW
ZSI6as4bzjDqDVPzCPWHDmQdSnJyXQ==
=aNK6
-END PGP SIGNATURE-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158290):
https://lists.openembedded.org/g/openembedded-core/message/158290
Mute This Topic: https://lists.openembedded.org/mt/86840389/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Alexander, "[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher" contains a fix for this issue. Reproducing a build offline with only the DL_DIR should work now. We are currently still lacking the appropriate tests for the fetcher. These will follow soon. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Thu, 2021-11-11 at 20:20 +0100, Alexander Kanavin wrote: > On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko > wrote: > > So if you have done this initial fetch of your sources and stash > > your > > working dir away, you can do an offline build. > > But can you do an offline build without a prepopulated working dir? > That's the crucial point: offline build should work in a freshly set > up build directory, without sstate cache, and with prepopulated > DL_DIR only. > > Alex -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS sY39fiky57aEgMe7g1OZX8oRODowPA== =hpAK -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158286): https://lists.openembedded.org/g/openembedded-core/message/158286 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher
From: Martin Koppehel
- Implement AUTOINC and submodule support for REPO provider
- Implement full srcrev support
- Add comments and fixup empty DL_DIR initialization
- Distinguish between artificial and plain rev
- Comments/documentation
The previous implementation of the repo fetcher could not handle updates
to the repo manifest file, nor deal with dynamic refspecs within this
manifest.
This patch fixes these shortcomings as follows:
During the recipe parsing phase, the repository containing the repo
manifest is cloned. This is done, as we need to parse the XML file
contained within, in order to discover all involved git repositories. A
combined hash is then calculated from the manifest repo, as well as any
git repo specified in the manifest. This hash is used for determining the
necessity of an update.
Additionally, the recipe will throw an error if the repo source is set to
a fixed revision but one or more repositories within the manifest
reference a dynamic refspec. This is done to ensure the reproducibility
of a version-pinned recipe.
Signed-off-by: Jasper Orschulko
---
lib/bb/fetch2/repo.py | 259 +-
1 file changed, 230 insertions(+), 29 deletions(-)
diff --git a/lib/bb/fetch2/repo.py b/lib/bb/fetch2/repo.py
index fa4cb814..706d37fa 100644
--- a/lib/bb/fetch2/repo.py
+++ b/lib/bb/fetch2/repo.py
@@ -3,6 +3,7 @@ BitBake "Fetch" repo (git) implementation
"""
+# Copyright (C) 2021 Martin Koppehel , iris-GmbH infrared &
intelligent sensors
# Copyright (C) 2009 Tom Rini
#
# Based on git.py which is:
@@ -13,9 +14,11 @@ BitBake "Fetch" repo (git) implementation
import os
import bb
+import hashlib
+import xml.etree.ElementTree as ET
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
+
class Repo(FetchMethod):
"""Class to fetch a module or modules from repo (git) repositories"""
@@ -27,46 +30,84 @@ class Repo(FetchMethod):
def urldata_init(self, ud, d):
"""
-We don"t care about the git rev of the manifests repository, but
-we do care about the manifest to use. The default is "default".
-We also care about the branch or tag to be used. The default is
-"master".
+We do care about the rev of the manifests repository, as well as the
+manifest file. However, when SRCREV=AUTOINC, then we use the specified
+branch in SRC_URI, with a fallback to master.
+use sm=fetch to fetch possibly referenced submodules in repositories.
"""
ud.basecmd = d.getVar("FETCHCMD_repo") or "/usr/bin/env repo"
+ud.gitcmd = d.getVar("FETCHCMD_git") or "git -c
core.fsyncobjectfiles=0"
ud.proto = ud.parm.get('protocol', 'git')
ud.branch = ud.parm.get('branch', 'master')
+
+ud.submodules = ud.parm.get('sm', 'fetch')
ud.manifest = ud.parm.get('manifest', 'default.xml')
if not ud.manifest.endswith('.xml'):
ud.manifest += '.xml'
-ud.localfile = d.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host,
ud.path.replace("/", "."), ud.manifest, ud.branch))
+repodir = d.getVar("REPODIR") or (d.getVar("DL_DIR") + "/repo")
+gitsrcname = "%s%s.%s" % (ud.host, ud.path.replace("/", "."),
ud.manifest)
+ud.codir = os.path.join(repodir, d.getVar("BPN"), gitsrcname)
+
+if ud.user:
+ud.username = ud.user + "@"
+else:
+ud.username = ""
+ud.remoteRepo = "%s://%s%s%s" % (ud.proto, ud.username, ud.host,
ud.path)
+
+ud.repodir = os.path.join(ud.codir, "repo")
+# a temporary directory to compute _latest_revision
+ud.tempdir = os.path.join(ud.codir, "temp")
+ud.stampfile = os.path.join(ud.codir, "__hash.txt")
+ud.setup_revisions(d)
+
+# ud.localfile is used to fill localpath, where the downloaded tarball
is stored.
+# in our case, we want something like repo_$GIT_URL_$MANIFEST_$SRCREV
+# todo: do we want the packagename?
+ud.localfile = "repo_%s%s_%s_%s.tar.gz" % (ud.host,
ud.path.replace("/", "."), ud.manifest, ud.revision)
+
+def need_update(self, ud, d):
+if ud.revision.startswith("_"):
+ return True
+return not os.path.exists(ud.localfile)
def download(self, ud, d):
"""Fetch url"""
-if os.access(os.path.join(d.getVar("DL_DIR"), ud.localfile), os.R_OK):
-logger.debug("%s already exists (or was stashed). Skipping repo
init / sync.", ud.localpath)
-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Alex, thanks for your input. You are absolutely correct, this currently does not work. We'll look into this. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Thu, 2021-11-11 at 20:20 +0100, Alexander Kanavin wrote: > On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko > wrote: > > So if you have done this initial fetch of your sources and stash > > your > > working dir away, you can do an offline build. > > But can you do an offline build without a prepopulated working dir? > That's the crucial point: offline build should work in a freshly set > up build directory, without sstate cache, and with prepopulated > DL_DIR only. > > Alex -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGOXJwACgkQYgqew07V MNXaxwf+NDOx5TYR4dY4iCsb0IfdfoB01+H7UDZR2s5GZKKA9TyMKbEqBdZrHBdj PTqZJ339IOXBlks1KPAUci0VOXBdwHoybR2XFMGENTnrj6XA+9Yr96ofSPHqQTWr amD/low0cFvfqZClb+md/FcxhnVK45rFhfLJckKElPiYmKjcghs/PXAWbh+Tw6S7 m5KCq8agPqXnaU/bWDmj2eC9t6t+VT/0OnC0tQtohMbYVd8wul4cf5Ic3JilecjI qCH4rK7EBFfzWhs1WTYDGeXX5iWNmZOsdufpXF0URJa5cpjGA5bhjfyXjoUZPzCR WxErD1cnQSNQqysqWlnB5EgM3y+cBg== =q0zZ -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158217): https://lists.openembedded.org/g/openembedded-core/message/158217 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Peter, during this initial fetch it is true, that the repo fetcher has to download the repo sources from googlesource.com. But during this first fetch the fetcher also needs to download any sources listed in the repo manifest anyway, so internet access is a given. Consequential fetches will only re-run repo if: a) the SRC_URI or SRCREV has changed within the recipe b) the SRCREV is set to AUTOREV Both cases do not apply when reproducing a build offline. So if you have done this initial fetch of your sources and stash your working dir away, you can do an offline build. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Thu, 2021-11-11 at 14:11 +, Peter Kjellerstedt wrote: > > -Original Message- > > From: Jasper Orschulko > > Sent: den 11 november 2021 13:10 > > To: richard.pur...@linuxfoundation.org; openembedded- > > c...@lists.openembedded.org; kweihm...@outlook.com; Peter > > Kjellerstedt > > ; jas...@fancydomain.eu > > Cc: mar...@mko.dev; Daniel Baumgart > > ; > > bitbake-de...@lists.openembedded.org > > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial > > recipe > > for repo 2.17.3 > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > Hi Peter, > > > > > However, this means > > > that only the source for the repo wrapper is available in DL_DIR. > > > So > > > when the repo wrapper executes it will try to go on the Internet > > > to > > > fetch the rest of the repo source, which will fail. > > > > just to clearify: Are we talking about installing repo on a target, > > or > > are we talking about the repo fetcher? > > The repo fetcher, i.e., when repo is used by bitbake. > > > When installing repo on the target, we do not call any "repo" > > command > > within the bitbake build process. We just clone the git repo > > containing > > the wrapper script and install the wrapper script into the sysroot. > > At > > that point the bitbake build process is finished and the user ends > > up > > with just the wrapper script on his device. Just as he would when > > installing repo from the system repositories on debian, arch, etc. > > > > When using the fetcher, the repo sources are fetched during the > > initial > > do_fetch (this also works with `bitbake --runonly=do_fetch`) and > > placed > > into the DL_DIR (in the .repo folder, next to the sources as > > described > > in the repo manifest). So unless the user changes something fetcher > > related in the recipe or deletes his DL_DIR, he can use this build > > environment for an offline build. > > How can it work? The repo source, which is fetched by the repo > wrapper, > will not be in DL_DIR or handled by bitbake at all. This means the > repo wrapper will try to contact googlesource.com directly and fetch > it > when it is run by bitbake, which cannot work since the host is > disconnected from the network. How or from where would it be able to > get the sources it needs? > > > You can try this for yourself, if you like: > > > > 1) clone https://github.com/Jasper-Ben/demo-kas > > 2) (with docker and docker-compose installed) run `make offline- > > build` > > > > - -- > > With best regards > > > > Jasper Orschulko > > DevOps Engineer > > > > Tel. +49 30 58 58 14 265 > > Fax +49 30 58 58 14 999 > > jasper.orschu...@iris-sensing.com > > > > • • • • • • • • • • • • • • • • • • • • • • • • • • > > > > iris-GmbH > > infrared & intelligent sensors > > Schnellerstraße 1-5 | 12439 Berlin > > > > https://iris-sensing.com/ > > //Peter > -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNMesACgkQYgqew07V MNU//Af+IFFqmbtV/k3SscMUWYRNDpnSeK6elwfpwnDi84tQHjRo66Xtrzi12D+H zzya3H87unVY3BWqWnQIE3K4aShquz+p9tQA7zuzlARImCyqAsDQzKddD7hhK7AI 4toNobOqNoJZ2D1kJQyu7o7VsnvbiyvYz8Z8l+t4FF35+bwU26aE0iCFF63toZSi GLvmTzLigYNAu9UUW+UgSUAhHVsaCyOolnGwBdHEJW3NliCZEOLDf8omDygG3xDp UOKcX2Q238sGN2fCwAeS2ABC4ffMNDfrpzAS0Q1qy2WVmheWmmtGoaksW2z3sDZk FKxLlRWNQvPWOy/tY2JLyB1QNyedyg== =c20l -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158186): https://lists.openembedded.org/g/openembedded-core/message/158186 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] fetch2/repo: Implement AUTOREV for repo fetcher
From: Martin Koppehel
- Implement AUTOINC and submodule support for REPO provider
- Implement full srcrev support
- Add comments and fixup empty DL_DIR initialization
- Distinguish between artificial and plain rev
- Comments/documentation
The previous implementation of the repo fetcher could not handle updates
to the repo manifest file, nor deal with dynamic refspecs within this
manifest.
This patch fixes these shortcomings as follows:
During the recipe parsing phase, the repository containing the repo
manifest is cloned. This is done, as we need to parse the XML file
contained within, in order to discover all involved git repositories. A
combined hash is then calculated from the manifest repo, as well as any
git repo specified in the manifest. This hash is used for determining the
necessity of an update.
Additionally, the recipe will throw an error if the repo source is set to
a fixed revision but one or more repositories within the manifest
reference a dynamic refspec. This is done to ensure the reproducibility
of a version-pinned recipe.
Signed-off-by: Jasper Orschulko
---
lib/bb/fetch2/repo.py | 259 +-
1 file changed, 230 insertions(+), 29 deletions(-)
diff --git a/lib/bb/fetch2/repo.py b/lib/bb/fetch2/repo.py
index fa4cb814..b2e1ba8c 100644
--- a/lib/bb/fetch2/repo.py
+++ b/lib/bb/fetch2/repo.py
@@ -3,6 +3,7 @@ BitBake "Fetch" repo (git) implementation
"""
+# Copyright (C) 2021 Martin Koppehel , iris-GmbH infrared &
intelligent sensors
# Copyright (C) 2009 Tom Rini
#
# Based on git.py which is:
@@ -13,9 +14,11 @@ BitBake "Fetch" repo (git) implementation
import os
import bb
+import hashlib
+import xml.etree.ElementTree as ET
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
+
class Repo(FetchMethod):
"""Class to fetch a module or modules from repo (git) repositories"""
@@ -27,46 +30,84 @@ class Repo(FetchMethod):
def urldata_init(self, ud, d):
"""
-We don"t care about the git rev of the manifests repository, but
-we do care about the manifest to use. The default is "default".
-We also care about the branch or tag to be used. The default is
-"master".
+We do care about the rev of the manifests repository, as well as the
+manifest file. However, when SRCREV=AUTOINC, then we use the specified
+branch in SRC_URI, with a fallback to master.
+use sm=fetch to fetch possibly referenced submodules in repositories.
"""
ud.basecmd = d.getVar("FETCHCMD_repo") or "/usr/bin/env repo"
+ud.gitcmd = d.getVar("FETCHCMD_git") or "git -c
core.fsyncobjectfiles=0"
ud.proto = ud.parm.get('protocol', 'git')
ud.branch = ud.parm.get('branch', 'master')
+
+ud.submodules = ud.parm.get('sm', 'fetch')
ud.manifest = ud.parm.get('manifest', 'default.xml')
if not ud.manifest.endswith('.xml'):
ud.manifest += '.xml'
-ud.localfile = d.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host,
ud.path.replace("/", "."), ud.manifest, ud.branch))
+repodir = d.getVar("REPODIR") or (d.getVar("DL_DIR") + "/repo")
+gitsrcname = "%s%s.%s" % (ud.host, ud.path.replace("/", "."),
ud.manifest)
+ud.codir = os.path.join(repodir, d.getVar("BPN"), gitsrcname)
+
+if ud.user:
+ud.username = ud.user + "@"
+else:
+ud.username = ""
+ud.remoteRepo = "%s://%s%s%s" % (ud.proto, ud.username, ud.host,
ud.path)
+
+ud.repodir = os.path.join(ud.codir, "repo")
+# a temporary directory to compute _latest_revision
+ud.tempdir = os.path.join(ud.codir, "temp")
+ud.stampfile = os.path.join(ud.codir, "__hash.txt")
+ud.setup_revisions(d)
+
+# ud.localfile is used to fill localpath, where the downloaded tarball
is stored.
+# in our case, we want something like repo_$GIT_URL_$MANIFEST_$SRCREV
+# todo: do we want the packagename?
+ud.localfile = "repo_%s%s_%s_%s.tar.gz" % (ud.host,
ud.path.replace("/", "."), ud.manifest, d.getVar("SRCREV"))
+
+def need_update(self, ud, d):
+if d.getVar("SRCREV") == "AUTOINC":
+return True
+return os.path.exists(ud.localfile)
def download(self, ud, d):
"""Fetch url"""
-if os.access(os.path.join(d.getVar("DL_DIR"), ud.localfile), os.R_OK):
-logger.debug("%s already exists (or was stashed). Skipping repo
init / sync
Re: [oe-core][PATCH v5 1/2] repo: Add recipe for 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
> Remove " \n".
done in v6.
> You can remove "m 0755" as that is the default for install.
whoops, that sneaked back in as I was addressing Khem's comment :)
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Thu, 2021-11-11 at 11:20 +, Peter Kjellerstedt wrote:
> > -Original Message-
> > From:
> > openembedded-core@lists.openembedded.org
> > > > On Behalf Of Jasper Orschulko via lists.openembedded.org
> > Sent: den 11 november 2021 11:21
> > To: openembedded-core@lists.openembedded.org
> > Cc: mar...@mko.dev; daniel.baumg...@iris-sensing.com;
> > bitbake-de...@lists.openembedded.org; Jasper Orschulko
> >
> > Subject: [oe-core][PATCH v5 1/2] repo: Add recipe for 2.17.3
> >
> > From: Jasper Orschulko
> >
> > Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
> >
> > Signed-off-by: Jasper Orschulko
> > ---
> > meta/conf/distro/include/maintainers.inc | 1 +
> > .../0001-Set-REPO_REV-to-v2.17.3.patch | 35
> > +++
> > .../repo/repo/0001-python3-shebang.patch | 26 ++
> > meta/recipes-devtools/repo/repo_2.17.3.bb | 28 +++
> > 4 files changed, 90 insertions(+)
> > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-REPO_REV-to-v2.17.3.patch
> > create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> >
> > diff --git a/meta/conf/distro/include/maintainers.inc
> > b/meta/conf/distro/include/maintainers.inc
> > index f3e0a75d56..58a0a9615f 100644
> > --- a/meta/conf/distro/include/maintainers.inc
> > +++ b/meta/conf/distro/include/maintainers.inc
> > @@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert
> > Yang "
> > RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
> > RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
> > RECIPE_MAINTAINER:pn-readline = "Hongxu Jia
> > "
> > +RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
> > "
> > RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi
> > "
> > RECIPE_MAINTAINER:pn-rgb = "Unassigned
> > "
> > RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia
> > "
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 00..285b1d3129
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,35 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the default REPO_REV to a SHA1 that
> > corresponds to
> > +the version of the recipe. REPO_REV can still be overwriten by the
> > user,
> > +by specifying the REPO_REV environment variable.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> >
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index b13e34c..31130e9 100755
> > +--- a/repo
> > b/repo
> > +@@ -130,7 +130,7 @@ if not REPO_URL:
> > + REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > + REPO_REV = os.environ.get('REPO_REV')
> > + if not REPO_REV:
> > +- REPO_REV = 'stable'
> > ++ REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
> > +
> > + # increment this whenever we make important changes to this
> > script
> > + VERSION = (2, 8)
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch b/meta/recipes-devtools
Re: [bitbake-devel] [oe-core][PATCH v5 1/2] repo: Add recipe for 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi Jose,
> I think we can replace the patch 0001-Set-REPO_REV-to-v2.17.3.patch
> with a post function
> and with it we can reuse the SRCREV of the recipe. Something like:
>
> do_fix_rev(){
> sed -i "s/REPO_REV = 'stable'/REPO_REV = '${SRCREV}'" ${S}/repo
> }
>
> do_patch[postfuncs] += "do_fix_rev"
Yeah... was thinking about that as well. Wasn't sure though if this
would be considered bad practise. What do the others think about this?
@Richard?
> repo uses git internally, so add it
>
> RDEPENDS:${PN} = "python3 git"
will be fixed in v6.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Thu, 2021-11-11 at 11:00 +, Jose Quaresma wrote:
>
> Jasper Orschulko via lists.openembedded.org
> escreveu no dia
> quinta, 11/11/2021 à(s) 10:21:
> > From: Jasper Orschulko
> >
> > Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
> >
> > Signed-off-by: Jasper Orschulko
> > ---
> > meta/conf/distro/include/maintainers.inc | 1 +
> > .../0001-Set-REPO_REV-to-v2.17.3.patch | 35
> > +++
> > .../repo/repo/0001-python3-shebang.patch | 26 ++
> > meta/recipes-devtools/repo/repo_2.17.3.bb | 28 +++
> > 4 files changed, 90 insertions(+)
> > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-
> > REPO_REV-to-v2.17.3.patch
> > create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> >
> > diff --git a/meta/conf/distro/include/maintainers.inc
> > b/meta/conf/distro/include/maintainers.inc
> > index f3e0a75d56..58a0a9615f 100644
> > --- a/meta/conf/distro/include/maintainers.inc
> > +++ b/meta/conf/distro/include/maintainers.inc
> > @@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert
> > Yang
> > "
> > RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
> > RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
> > RECIPE_MAINTAINER:pn-readline = "Hongxu Jia
> > "
> > +RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
> > "
> > RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi
> > "
> > RECIPE_MAINTAINER:pn-rgb = "Unassigned
> > "
> > RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia
> > "
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 00..285b1d3129
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,35 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the default REPO_REV to a SHA1 that
> > corresponds
> > to
> > +the version of the recipe. REPO_REV can still be overwriten by the
> > user,
> > +by specifying the REPO_REV environment variable.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> >
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index b13e34c..31130e9 100755
> > +--- a/repo
> > b/repo
> > +@@ -130,7 +130,7 @@ if not REPO_URL:
> > + REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > + REPO_REV = os.environ.get('REPO_REV')
> > + if not REPO_REV:
> > +- REPO_REV = 'stable'
> > ++ REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
> > +
> > + # increment this whenever we make important changes to this
> > script
> > + VERSION = (2, 8)
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtool
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Peter, > However, this means > that only the source for the repo wrapper is available in DL_DIR. So > when the repo wrapper executes it will try to go on the Internet to > fetch the rest of the repo source, which will fail. just to clearify: Are we talking about installing repo on a target, or are we talking about the repo fetcher? When installing repo on the target, we do not call any "repo" command within the bitbake build process. We just clone the git repo containing the wrapper script and install the wrapper script into the sysroot. At that point the bitbake build process is finished and the user ends up with just the wrapper script on his device. Just as he would when installing repo from the system repositories on debian, arch, etc. When using the fetcher, the repo sources are fetched during the initial do_fetch (this also works with `bitbake --runonly=do_fetch`) and placed into the DL_DIR (in the .repo folder, next to the sources as described in the repo manifest). So unless the user changes something fetcher related in the recipe or deletes his DL_DIR, he can use this build environment for an offline build. You can try this for yourself, if you like: 1) clone https://github.com/Jasper-Ben/demo-kas 2) (with docker and docker-compose installed) run `make offline-build` - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Thu, 2021-11-11 at 11:34 +, Peter Kjellerstedt wrote: > > -Original Message- > > From: Jasper Orschulko > > Sent: den 11 november 2021 11:05 > > To: richard.pur...@linuxfoundation.org; openembedded- > > c...@lists.openembedded.org; kweihm...@outlook.com; Peter > > Kjellerstedt > > ; jas...@fancydomain.eu > > Cc: mar...@mko.dev; Daniel Baumgart > > ; > > bitbake-de...@lists.openembedded.org > > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial > > recipe > > for repo 2.17.3 > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > Hi Peter, > > > > > How do you avoid the repo wrapper fetching the repo source itself > > > and instead fetch it using the bitbake fetcher? I have seen nothing > > > to that extent in the patches so far. > > > > we don't. This recipe only installs the repo wrapper. The source code > > is downloaded and installed when running `repo init`. > > > > However, in our opinion this is not an issue. When installing repo as > > a > > package to a target, this is the expected behaviour. The target would > > only have the repo wrapper installed, just like on any other linux > > distrobution. The actual usage of repo on the target is decoupled > > from > > the bitbake build process. > > It might be how repo is designed, but it will still break the bitbake > expectations. I.e., if I have an environment with the layers available, > a populated DL_DIR and BB_NO_NETWORK = "1", and then disconnect the > build host from the Internet, I should still be able to source > oe-init-build-env for a new machine and build it. However, this means > that only the source for the repo wrapper is available in DL_DIR. So > when the repo wrapper executes it will try to go on the Internet to > fetch the rest of the repo source, which will fail. > > > When using the repo fetcher, the repo source is fetched during the > > do_fetch stage by running `repo init` (when SRCREV = AUTOREV, changes > > to the recipe or no previous sources available in DL_DIR). By > > executing > > this within the "runfetchcmd" function, this also works with the > > usual > > network features bitbake provides, e.g. proxy. > > If the recipe has not changed and sources are already available from > > a > > previous run, repo will not be rerun. As such, reproducing a build > > offline is also possible. > > > > - -- > > With best regards > > > > Jasper Orschulko > > DevOps Engineer > > > > Tel. +49 30 58 58 14 265 > > Fax +49 30 58 58 14 999 > > jasper.orschu...@iris-sensing.com > > > > • • • • • • • • • • • • • • • • • • • • • • • • • • > > > > iris-GmbH > > infrared & intelligent sensors > > Schnellerstraße 1-5 | 12439 Berlin > > > > https://iris-sensing.com/ > > > > > > > > > > > > On Wed, 2021-11-10 at 23:55 +, Peter Kjellerstedt wrote: > > &g
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Richard, > Even ten recipes using this will show a degradation in parsing speed > and I do > get a lot of complaints when parsing slows down for any reason. The > user doesn't > expect this and it won't be visible what bitbake is doing (sitting at > 99% parsed > for a period). > > Also, the "tend to be small" implies someone will create a huge one > at some > point even if that is a bad idea for whatever reasons, I just know > how these > things end up going :(. I believe this is something that could be addressed in the documentation of the repo fetcher? Something along the lines of "Using the repo fetcher can increase the time spent in the parsing phase, as we need to clone and inspect the manifest repository. To avoid this, keep the size of your manifest repository at a minimum." When this is done, the dalay is barely noticable. We have a testsetup with 24 recipes and reasonable parsing speeds. > * Does it only clone a repo in the AUTOREV case? No, it also clones with fixed refspecs, as we want to ensure that each of the git repos referenced in the manifest also uses fixed refspecs. Otherwise, the sources in the background might change, while the recipe still (correctly) references the same revision of the manifest repo. > * Could it only obtain the manifest file somehow without a clone of > the repo? Unfortunately not really. git does not offer a way to list the content of a file without cloning it beforehand. The only other way I could imagine this to work would require us to keep the manifest file within the meta layer and do a somewhat "hacky" workaround: It is possible to run `repo init` on a local git folder. As such, something like this could work: 1) create temp folder (e.g. /tmp/repo-X) 2) run git init in temp folder 3) install manifest file into temp folder 4) add & commit 5) run repo init -u file://$temp-folder The repo manifest could even still be used outside of yocto, by using the metalayer repo as basis for the repo init: `repo init -u $metalayer-repo -m $path-to-manifest-file`. Not pretty, but it works. So if cloning within the parser is a dealbreaker for you, this option might be worth looking into. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Wed, 2021-11-10 at 16:33 +, Richard Purdie wrote: > On Wed, 2021-11-10 at 13:52 +, Jasper Orschulko wrote: > > Hi Richard, > > > > > When you say "fixed refspec", will that be a definitive sha > > > revision > > > or a tag? > > > We always force resolution of tags as they tend to cause problems > > > and > > > can change even if it is bad form. > > > > that's a good point. Actually, Martin and I have just been > > discussing > > this, as we noticed that this point actually got "lost" during our > > implementation. We are currently working on fixing this. Good to > > know > > how you handle this. I will keep you posted. > > Ok, it is good to be clear on that one. I know the fact we hit the > network for > tags does concern some but it really is the only way to handle them. > > > > This is potentially a big issue. Cloning operations during > > > parsing is > > > pretty > > > horrible. We'd not expect any thing being written out like that > > > during a parse. > > > It would probably work "ok" for one recipe but if you start > > > getting > > > the hundreds > > > of git recipes we have in some layers, it wouldn't scale if we > > > allowed that :(. > > > > > > Not sure what to recommend here but it is definitely problematic. > > > > Just to make sure that we are on the same page: This ONLY affects > > recipes which use the repo fetcher. And it ONLY clones the > > repository > > containing the repo manifest (which tend to be small in size). > > Correct, we are on the same page. This is still quite problematic as > the recipes > are meant to parse quickly and a repository clone is definitely not > expected. > > > So unless developers start using hundreds of repo-based recipes, > > which I > > find a very unlikely scenario, this should not be an issue. > > Even ten recipes using this will show a degradation in parsing speed > and I do > get a lot of complaints when parsing slows down for any reason. The > user doesn't > expect this and it won't be visible what bitbake is doin
Re: [bitbake-devel] [oe-core][PATCH v4 1/2] repo: Add recipe for 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
> to be on safer side perhaps use below single line instead of above
> two.
> install -Dm 0755 ${WORKDIR}/git/repo ${D}${bindir}/repo
Done in v5 of this patch series :)
Also, I modified the REPO_REV patch in such a way, that the enduser
installing repo on a target is still able to override the default
REPO_REV by setting the environment variable.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Wed, 2021-11-10 at 09:20 -0800, Khem Raj wrote:
>
>
> On 11/10/21 7:53 AM, Jasper Orschulko via lists.openembedded.org
> wrote:
> > From: Jasper Orschulko
> >
> > Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
> >
> > Signed-off-by: Jasper Orschulko
> > ---
> > meta/conf/distro/include/maintainers.inc | 1 +
> > .../0001-Set-REPO_REV-to-v2.17.3.patch | 34
> > +++
> > .../repo/repo/0001-python3-shebang.patch | 26 ++
> > meta/recipes-devtools/repo/repo_2.17.3.bb | 29
> >
> > 4 files changed, 90 insertions(+)
> > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-REPO_REV-to-v2.17.3.patch
> > create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> >
> > diff --git a/meta/conf/distro/include/maintainers.inc
> > b/meta/conf/distro/include/maintainers.inc
> > index f3e0a75d56..58a0a9615f 100644
> > --- a/meta/conf/distro/include/maintainers.inc
> > +++ b/meta/conf/distro/include/maintainers.inc
> > @@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert
> > Yang "
> > RECIPE_MAINTAINER:pn-quota = "Anuj Mittal
> > "
> > RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
> > RECIPE_MAINTAINER:pn-readline = "Hongxu Jia
> > "
> > +RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
> > "
> > RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi
> > "
> > RECIPE_MAINTAINER:pn-rgb = "Unassigned
> > "
> > RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia
> > "
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 00..3086f8eb42
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,34 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to a SHA1 that
> > corresponds to
> > +the version of the recipe.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> >
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > + REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> > ++REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
> > + if not REPO_REV:
> > + REPO_REV = 'stable'
> > + # URL to file bug reports for repo tool issues.
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 00..d3888c8bb2
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
> > @@ -0,0 +1,26 @@
> > +From b8e84b202cd302a7c99288d3835dc9c
[oe-core][PATCH v5 2/2] base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko
Add git-native and repo-native as prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/classes/base.bbclass | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..3298bd1952 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,11 @@ python () {
elif uri.scheme == "npm":
d.appendVarFlag('do_fetch', 'depends', '
nodejs-native:do_populate_sysroot')
+elif uri.scheme == "repo":
+needsrcrev = True
+d.appendVarFlag('do_fetch', 'depends', '
repo-native:do_populate_sysroot')
+d.appendVarFlag('do_fetch', 'depends', '
git-native:do_populate_sysroot')
+
# *.lz4 should DEPEND on lz4-native for unpacking
if path.endswith('.lz4'):
d.appendVarFlag('do_unpack', 'depends', '
lz4-native:do_populate_sysroot')
--
2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158157):
https://lists.openembedded.org/g/openembedded-core/message/158157
Mute This Topic: https://lists.openembedded.org/mt/86978425/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH v5 1/2] repo: Add recipe for 2.17.3
From: Jasper Orschulko
Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/conf/distro/include/maintainers.inc | 1 +
.../0001-Set-REPO_REV-to-v2.17.3.patch| 35 +++
.../repo/repo/0001-python3-shebang.patch | 26 ++
meta/recipes-devtools/repo/repo_2.17.3.bb | 28 +++
4 files changed, 90 insertions(+)
create mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
diff --git a/meta/conf/distro/include/maintainers.inc
b/meta/conf/distro/include/maintainers.inc
index f3e0a75d56..58a0a9615f 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert Yang
"
RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
RECIPE_MAINTAINER:pn-readline = "Hongxu Jia "
+RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
"
RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi "
RECIPE_MAINTAINER:pn-rgb = "Unassigned "
RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia "
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 00..285b1d3129
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,35 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the default REPO_REV to a SHA1 that corresponds to
+the version of the recipe. REPO_REV can still be overwriten by the user,
+by specifying the REPO_REV environment variable.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..31130e9 100755
+--- a/repo
b/repo
+@@ -130,7 +130,7 @@ if not REPO_URL:
+ REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+ REPO_REV = os.environ.get('REPO_REV')
+ if not REPO_REV:
+- REPO_REV = 'stable'
++ REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
+
+ # increment this whenever we make important changes to this script
+ VERSION = (2, 8)
+--
+2.33.1
diff --git a/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
new file mode 100644
index 00..d3888c8bb2
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
@@ -0,0 +1,26 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+Yocto does not symlink from python to python3, thus change the shebang from
+python to python3.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 00..cb0af89c09
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git
repositories, does the uploads to revision control systems, and automates parts
of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo;
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
+
+SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+
+SRC_URI += "file://0001-python3-shebang.patch \
+file://0001-Set-REPO_REV-to-v2.17.3.patch"
+
+MIRRORS += "git://gerrit.googlesource.com/git-repo.git
gi
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Peter, > How do you avoid the repo wrapper fetching the repo source itself > and instead fetch it using the bitbake fetcher? I have seen nothing > to that extent in the patches so far. we don't. This recipe only installs the repo wrapper. The source code is downloaded and installed when running `repo init`. However, in our opinion this is not an issue. When installing repo as a package to a target, this is the expected behaviour. The target would only have the repo wrapper installed, just like on any other linux distrobution. The actual usage of repo on the target is decoupled from the bitbake build process. When using the repo fetcher, the repo source is fetched during the do_fetch stage by running `repo init` (when SRCREV = AUTOREV, changes to the recipe or no previous sources available in DL_DIR). By executing this within the "runfetchcmd" function, this also works with the usual network features bitbake provides, e.g. proxy. If the recipe has not changed and sources are already available from a previous run, repo will not be rerun. As such, reproducing a build offline is also possible. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Wed, 2021-11-10 at 23:55 +, Peter Kjellerstedt wrote: > > -Original Message- > > From: bitbake-de...@lists.openembedded.org > de...@lists.openembedded.org> On Behalf Of Jasper Orschulko > > Sent: den 9 november 2021 12:26 > > To: richard.pur...@linuxfoundation.org; openembedded- > > c...@lists.openembedded.org; kweihm...@outlook.com; > > jas...@fancydomain.eu > > Cc: mar...@mko.dev; daniel.baumg...@iris-sensing.net; bitbake- > > de...@lists.openembedded.org > > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial > > recipe > > for repo 2.17.3 > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > Hi Richard, > > > > I think our implementation of the repo fetcher checks the (what I > > believe to be) most relevant parts of your checklist (thanks for the > > write-up). Martin, please corrent me if I'm missing something: > > > > > a) network access for sources is only expected to happen in the > > > do_fetch step. > > > This is not enforced or tested but is required so that we can: > > > > > > i) audit the sources used (i.e. for license/manifest reasons) > > > ii) support offline builds with a suitable cache > > > iii) allow work to continue even with downtime upstream > > > iv) allow for changes upstream in incompatible ways > > > v) allow rebuilding of the software in X years time > > > > check > > > > > b) network access is not expected in do_unpack > > > > check > > How do you avoid the repo wrapper fetching the repo source itself > and instead fetch it using the bitbake fetcher? I have seen nothing > to that extent in the patches so far. > > //Peter > -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGM6rUACgkQYgqew07V MNUCIAf+MXGp9Hnfxmu+8w3BasVsP2N7ttW2FN3Zroiyr/hKrJzeq/qDQwO+/K3U zWZJ85H6L2eTjOP8fPaHbVMRD1jUoYVpYUAE/fN64FbhCBmurVuWFrLo/u6Gy2cU DCd3SIejXidB25EQRoS4Bfl25il1wG4iMw1pCFA6ku5rGhb8q5jvfZECf0Wuhh7X FnMQlI3VZsSk4CakF3g88AFTLFrsQYcN2vDUskdhMfTZpuRA8duIvW4rVgOvHJQT v07rASNR/9yzPRVkzpgPUI9Vl2LA3vEZ3Rccw3jscYHFwkzj0096DO4+jeDwyza5 fFm0dDT9HjGuzTz66c5JL8sdZZi9pw== =AOyp -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158152): https://lists.openembedded.org/g/openembedded-core/message/158152 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [oe-core][PATCH v4 2/2] base.bbclass: Add sysroot deps for repo fetcher
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Unless there are any more comments, I believe this patch series should
be mergable now.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Wed, 2021-11-10 at 16:53 +0100, jas...@fancydomain.eu wrote:
> From: Jasper Orschulko
>
> Add git-native and repo-native as prerequisite for the repo fetcher.
>
> Signed-off-by: Jasper Orschulko
> ---
> meta/classes/base.bbclass | 5 +
> 1 file changed, 5 insertions(+)
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index a65fcc6c1d..3298bd1952 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -665,6 +665,11 @@ python () {
> elif uri.scheme == "npm":
> d.appendVarFlag('do_fetch', 'depends', ' nodejs-
> native:do_populate_sysroot')
>
> + elif uri.scheme == "repo":
> + needsrcrev = True
> + d.appendVarFlag('do_fetch', 'depends', ' repo-
> native:do_populate_sysroot')
> + d.appendVarFlag('do_fetch', 'depends', ' git-
> native:do_populate_sysroot')
> +
> # *.lz4 should DEPEND on lz4-native for unpacking
> if path.endswith('.lz4'):
> d.appendVarFlag('do_unpack', 'depends', ' lz4-
> native:do_populate_sysroot')
-BEGIN PGP SIGNATURE-
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGL66cACgkQYgqew07V
MNWhOwf/c4ITfBijE4DD2gFOWj0OMAQFrdeYC8dQhsm756Kua1m0mPc5Oh61kkB9
og/bsd1qqnuabTcUHW9gVI9Hq45yPXfAy5Gyg6A10owDeCcmNgihCuTGr+sQy64W
li5OcOw59/ok6yPVqE+CaeYoHCGv1VdLR09pSdK1S9kk/3hVNOSRb9umtGb3aR6L
KmrtG71QZVVLBjTrppzIolyuy116cGfIey5zAVmrYPDHLtvwYjnt1NdBFn7OFMFV
7kIXy7aoBMXsuRxTR6mvC9GxJcwOUgsILUWLyQ62VcgNH13G5nR0zHwPamtIOT0R
eaoasSPkMl2wfevOHosrI9tgMI8hIg==
=EDCL
-END PGP SIGNATURE-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158081):
https://lists.openembedded.org/g/openembedded-core/message/158081
Mute This Topic: https://lists.openembedded.org/mt/86960074/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH v4 2/2] base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko
Add git-native and repo-native as prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/classes/base.bbclass | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..3298bd1952 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,11 @@ python () {
elif uri.scheme == "npm":
d.appendVarFlag('do_fetch', 'depends', '
nodejs-native:do_populate_sysroot')
+elif uri.scheme == "repo":
+needsrcrev = True
+d.appendVarFlag('do_fetch', 'depends', '
repo-native:do_populate_sysroot')
+d.appendVarFlag('do_fetch', 'depends', '
git-native:do_populate_sysroot')
+
# *.lz4 should DEPEND on lz4-native for unpacking
if path.endswith('.lz4'):
d.appendVarFlag('do_unpack', 'depends', '
lz4-native:do_populate_sysroot')
--
2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158080):
https://lists.openembedded.org/g/openembedded-core/message/158080
Mute This Topic: https://lists.openembedded.org/mt/86960074/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH v4 1/2] repo: Add recipe for 2.17.3
From: Jasper Orschulko
Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/conf/distro/include/maintainers.inc | 1 +
.../0001-Set-REPO_REV-to-v2.17.3.patch| 34 +++
.../repo/repo/0001-python3-shebang.patch | 26 ++
meta/recipes-devtools/repo/repo_2.17.3.bb | 29
4 files changed, 90 insertions(+)
create mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
diff --git a/meta/conf/distro/include/maintainers.inc
b/meta/conf/distro/include/maintainers.inc
index f3e0a75d56..58a0a9615f 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert Yang
"
RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
RECIPE_MAINTAINER:pn-readline = "Hongxu Jia "
+RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
"
RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi "
RECIPE_MAINTAINER:pn-rgb = "Unassigned "
RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia "
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 00..3086f8eb42
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,34 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the REPO_REV variable to a SHA1 that corresponds to
+the version of the recipe.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index 4cddbf1..cf5f6b1 100755
+--- a/repo
b/repo
+@@ -142,7 +142,7 @@ if __name__ == '__main__':
+ REPO_URL = os.environ.get('REPO_URL', None)
+ if not REPO_URL:
+ REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+-REPO_REV = os.environ.get('REPO_REV')
++REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
+ if not REPO_REV:
+ REPO_REV = 'stable'
+ # URL to file bug reports for repo tool issues.
+--
+2.33.1
diff --git a/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
new file mode 100644
index 00..d3888c8bb2
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
@@ -0,0 +1,26 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+Yocto does not symlink from python to python3, thus change the shebang from
+python to python3.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 00..20f5d465d3
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,29 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git
repositories, does the uploads to revision control systems, and automates parts
of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo;
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
+
+SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+
+SRC_URI += "file://0001-python3-shebang.patch \
+file://0001-Set-REPO_REV-to-v2.17.3.patch"
+
+MIRRORS += "git://gerrit.googlesource.com/git-repo.git
git://github.com/GerritCodeReview/git-repo.git \n"
+
+S = "${
Re: [OE-core] [bitbake-devel] [PATCH 2/2] fetch2: Fix race condition in latest_revision
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I believe the behaviour we observed was actually caused by a bug in one of our earlier patch sets. I spent a couple hours today trying to recreate the issue, without any luck. I'd therefore skip this particular patch. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Mon, 2021-11-08 at 11:28 +0100, Alexander Kanavin wrote: > It helps if you can provide a specific example/test case for this. > > Alex > > On Mon, 8 Nov 2021 at 11:26, Martin Koppehel wrote: > > -- snip > > > I'm afraid I don't understand why this is a race condition? Where > > is the timeout > > > that stops one being set? > > > > The commit message is misleading here, will change that. It is > > actually > > not a race condition as we originally thought, but more a problem > > of > > recursive calls to _latest_revision. > > > > The git fetcher does something similar in _lsremote, where the git > > lsremote would be called recursively to populate the environment, > > and > > there's code in place to prevent that. > > > > In our case, since we persist the revs, we have to filter out empty > > revs > > because otherwise the next repo fetcher invocation would also try > > to > > populate the environment but then revs[key] would already be set to > > an > > empty string, causing a metadata mismatch later. > > > > Therefore we only populate the revision cache if _latest_revision > > actually returns a valid value. > > We did reproduce this with multiple recipes that used up the same > > SRC_URI and therefore influencing each other, this case works fine > > now. > > > > Cheers, > > Martin > > > > > > > > -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGL6P8ACgkQYgqew07V MNVKBQf/Z/PE68CI1L3F1jeHm9jX+DNs1dyA7IB6fOAHdeqYvKNKpbedpHXVHFlp oalj3jIEDJMKE8b8LmH/VrRhLTr68PSEshVIbHOKQQfJmy6FnbQw5VXTMZON92AS phiP4H4jMvr8sx942d+YoVwtk1iMO6+oZwUEx2k5uGiyVeayqMvs0XyrHQY65GhT 7RG9ZSOXutnkF1NCHto4ln5kb8G+MUcS3D57FRbsiGV4XvIaMTqFbn3ub6NP2x5H qMjQwWXLrF1eIQc6hVBXxPnloyPvZESp98/rTDHdBQsJB9DbS2akQ8MZBM+EIiff wtVJRNJa0jG/uDvF3wLCoO+ycHF8SQ== =vnW2 -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158078): https://lists.openembedded.org/g/openembedded-core/message/158078 Mute This Topic: https://lists.openembedded.org/mt/86878868/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi Richard,
> When you say "fixed refspec", will that be a definitive sha revision
> or a tag?
> We always force resolution of tags as they tend to cause problems and
> can change
> even if it is bad form.
that's a good point. Actually, Martin and I have just been discussing
this, as we noticed that this point actually got "lost" during our
implementation. We are currently working on fixing this. Good to know
how you handle this. I will keep you posted.
> This is potentially a big issue. Cloning operations during parsing is
> pretty
> horrible. We'd not expect any thing being written out like that
> during a parse.
> It would probably work "ok" for one recipe but if you start getting
> the hundreds
> of git recipes we have in some layers, it wouldn't scale if we
> allowed that :(.
>
> Not sure what to recommend here but it is definitely problematic.
Just to make sure that we are on the same page: This ONLY affects
recipes which use the repo fetcher. And it ONLY clones the repository
containing the repo manifest (which tend to be small in size). So
unless developers start using hundreds of repo-based recipes, which I
find a very unlikely scenario, this should not be an issue.
Unfortunately, I don't see any other way to access the repo manifest
file, as we need to calculate the commit hashes of the git repos
referenced in the repo manifest file. Otherwise, it is impossible for
us to determinate the necessity of an update when SRCREV =
"${AUTOREV}".
However, I see one potential improvement here. Currently the cloning of
the manifest repo is done on a per-recipe basis. E.g. this means if we
have 10 recipes inheriting a bbclass containing a repo fetcher, we will
clone 10 identical manifest repos. We'll work on improving this.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Wed, 2021-11-10 at 12:46 +, Richard Purdie wrote:
> On Tue, 2021-11-09 at 11:26 +, Jasper Orschulko wrote:
> >
> >
> > > e) fetcher output is deterministic
> > > (i.e. if you fetch configuration XXX now it will match in future
> > > exactly in
> > > a clean build with a new DL_DIR)
> >
> > check. When a fixed refspec is set within the recipe, the fetcher
> > will
> > check, if all repositories in the repo manifest are set to a fixed
> > refspec as well and otherwise throw an error.
>
> When you say "fixed refspec", will that be a definitive sha revision
> or a tag?
> We always force resolution of tags as they tend to cause problems and
> can change
> even if it is bad form.
>
> > > f) network access is expected to work with the standard linux
> > > proxy
> > > variables
> > > environment but only in the do_fetch tasks)
> >
> > this should work, as we keep the GIT_PROXY_COMMAND environment and
> > run
> > repo via runfetchcmd(). Not further tested though.
>
> If you're using runfetchcmd that should work.
>
> >
> > > g) access during parsing has to be minimal, a "git ls-remote" for
> > > anin
> > > AUTOREV
> > > git recipe might be ok but you can't expect to checkout a git
> > > tree
> >
> > unfortunately, do to the nature of repo, we need to clone the
> > manifest
> > repo, so that we can run "git ls-remote" on the referenced git
> > repos
> > and therefore ensure that
>
> This is potentially a big issue. Cloning operations during parsing is
> pretty
> horrible. We'd not expect any thing being written out like that
> during a parse.
> It would probably work "ok" for one recipe but if you start getting
> the hundreds
> of git recipes we have in some layers, it wouldn't scale if we
> allowed that :(.
>
> Not sure what to recommend here but it is definitely problematic.
>
> >
> Cheers,
>
> Richard
>
>
-BEGIN PGP SIGNATURE-
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGLzpwACgkQYgqew07V
MNV3XQf9FDzIq3yihsO5FCEn/QOm7v48VuuOZF/85K6dxRgexCfdHHxkn7zJ0luE
ZxgPDmcXM83HcFh6B5XKis88/vnkU2R+YITgWe9+81l1foQryKTP9u7E2giIHW/F
JpGzxTtTb5F3N0+xjmqnyR7OYEB3TqJ1VFsaLlYdYs/sWaDYbt/9AEtcD39ynCr5
dEEqEgiIk05X03kiNnyUd2jDpy0bAbihqJu7OPzU4zSvNn/+zXRM0CMKDemyONb1
u2lINROQpk98qaVzBTX+uOskQTkMrkRRuuncUY0ggq6pHGz3TRubv15mePYIeLlJ
y2lBBB6O8f/iPesjbvKFa85EeNhdAg==
=4Ewh
-END PGP SIGNATURE-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158077):
https://lists.openembedded.org/g/openembedded-core/message/158077
Mute This Topic: https://lists.openembedded.org/mt/86840389/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Richard, I think our implementation of the repo fetcher checks the (what I believe to be) most relevant parts of your checklist (thanks for the write-up). Martin, please corrent me if I'm missing something: > a) network access for sources is only expected to happen in the > do_fetch step. > This is not enforced or tested but is required so that we can: > > i) audit the sources used (i.e. for license/manifest reasons) > ii) support offline builds with a suitable cache > iii) allow work to continue even with downtime upstream > iv) allow for changes upstream in incompatible ways > v) allow rebuilding of the software in X years time check > b) network access is not expected in do_unpack check > c) you can take DL_DIR and use it as a mirror for offline builds check > d) access to the network is only made when explicitly configured in > recipes > (e.g. use of AUTOREV, or use of git tags which change revision) check > e) fetcher output is deterministic > (i.e. if you fetch configuration XXX now it will match in future > exactly in > a clean build with a new DL_DIR) check. When a fixed refspec is set within the recipe, the fetcher will check, if all repositories in the repo manifest are set to a fixed refspec as well and otherwise throw an error. > f) network access is expected to work with the standard linux proxy > variables > environment but only in the do_fetch tasks) this should work, as we keep the GIT_PROXY_COMMAND environment and run repo via runfetchcmd(). Not further tested though. > g) access during parsing has to be minimal, a "git ls-remote" for an > AUTOREV > git recipe might be ok but you can't expect to checkout a git tree unfortunately, do to the nature of repo, we need to clone the manifest repo, so that we can run "git ls-remote" on the referenced git repos and therefore ensure that > h) we need to provide revision information during parsing such that a > version > for the recipe can be constructed. check. We create a hash value from git ls-remote of the recipe file, as well as the git ls-remote of each repo referenced in the manifest. > i) versions are expected to be able to increase in a way which sorts > allowing > package feeds to operate (see PR server required for git revisions to > sort) check. We use the PR server and didn't notice any issues. > j) API to query for possible version upgrades of a url is highly > desireable to > allow out automated upgrage code to function (it is implied this does > always > have network access) check (if I understand you correctly). We support AUTOINC. > k) Where fixes or changes to behaviour in the fetcher are made, we ask > that > test cases are added (run with "bitbake-selftest bb.tests.fetch"). We > do > have fairly extensive test coverage of the fetcher as it is the only > way > to track all of it's corner cases, it still doesn't give entire > coverage > though sadly. We are currently still missing any tests for the new fetcher. We will add them in the course of the next days. Meanwhile, I have prepared a demo environment so if anymeone is interested in doing some further testing, feel free: https://github.com/Jasper-Ben/demo-kas :) - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ > -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGKWtQACgkQYgqew07V MNU3Lgf+PlDNyHxoCive3HL8V00KlfrsjTcuQVYiODav39n/h/cRvq8F8Wh7NiyM Hjrr9O7MCOZLNGQiW9vBN/R6FbWESZ+sxl2nLdGH98eCYPKBurcHzvIXlJgincc+ dL1HtxDqjdk9TOHfaKAD2zc8U0tY6xu6Hgj7QIAqELt8HAwOQeIherAuIQw7g+KH eWY5xhn8+KrY//RlnnRmDMO20LBY68bdkCOCH3kAIBfcSIzj8fzf1A4PZhGFi8AM dydVCBMrKvPg3bfJnenZrVPijWnb82pOwjWzoZltzYxtCjqAhwevoCr8twGNDbx4 IVPcEqtMWRrf8rpQiHsZFTheap8BdA== =hdir -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158014): https://lists.openembedded.org/g/openembedded-core/message/158014 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH v3 1/2] repo: Add recipe for 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi Jose,
whoops, good find! Yeah... I would also prefer to inject the REPO_REV
as an environment variable within the bitbake build process rather then
hardcoding it with a patch, but I am not sure how and if this would
work? I am open to suggestions! :)
> injecting SRCREV after unpack will be better imo
I am not sure I completely understand what you are saying here. Are you
saying, that the SRCREV in the recipe should be set somewhere else?
Cheers!
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Mon, 2021-11-08 at 12:23 +, Jose Quaresma wrote:
>
>
> Jasper Orschulko via lists.openembedded.org
> escreveu no dia
> segunda, 8/11/2021 à(s) 11:59:
> > From: Jasper Orschulko
> >
> > Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
> >
> > Signed-off-by: Jasper Orschulko
> > ---
> > meta/conf/distro/include/maintainers.inc | 1 +
> > .../0001-Set-REPO_REV-to-v2.17.3.patch | 34
> > +++
> > .../repo/repo/0001-python3-shebang.patch | 26 ++
> > meta/recipes-devtools/repo/repo_2.17.3.bb | 27 +++
> > 4 files changed, 88 insertions(+)
> > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-
> > REPO_REV-to-v2.17.3.patch
> > create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> >
> > diff --git a/meta/conf/distro/include/maintainers.inc
> > b/meta/conf/distro/include/maintainers.inc
> > index f3e0a75d56..58a0a9615f 100644
> > --- a/meta/conf/distro/include/maintainers.inc
> > +++ b/meta/conf/distro/include/maintainers.inc
> > @@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert
> > Yang
> > "
> > RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
> > RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
> > RECIPE_MAINTAINER:pn-readline = "Hongxu Jia
> > "
> > +RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
> > "
> > RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi
> > "
> > RECIPE_MAINTAINER:pn-rgb = "Unassigned
> > "
> > RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia
> > "
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 00..3086f8eb42
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,34 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to a SHA1 that
> > corresponds to
> > +the version of the recipe.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> >
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > + REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> > ++REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
> > + if not REPO_REV:
> > + REPO_REV = 'stable'
> > + # URL to file bug reports for repo tool issues.
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/repo/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 00..d3888c8bb2
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/r
[oe-core][PATCH v3 1/2] repo: Add recipe for 2.17.3
From: Jasper Orschulko
Add a recipe for repo 2.17.3, prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/conf/distro/include/maintainers.inc | 1 +
.../0001-Set-REPO_REV-to-v2.17.3.patch| 34 +++
.../repo/repo/0001-python3-shebang.patch | 26 ++
meta/recipes-devtools/repo/repo_2.17.3.bb | 27 +++
4 files changed, 88 insertions(+)
create mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
diff --git a/meta/conf/distro/include/maintainers.inc
b/meta/conf/distro/include/maintainers.inc
index f3e0a75d56..58a0a9615f 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -652,6 +652,7 @@ RECIPE_MAINTAINER:pn-quilt-native = "Robert Yang
"
RECIPE_MAINTAINER:pn-quota = "Anuj Mittal "
RECIPE_MAINTAINER:pn-re2c = "Khem Raj "
RECIPE_MAINTAINER:pn-readline = "Hongxu Jia "
+RECIPE_MAINTAINER:pn-repo = "Jasper Orschulko
"
RECIPE_MAINTAINER:pn-resolvconf = "Chen Qi "
RECIPE_MAINTAINER:pn-rgb = "Unassigned "
RECIPE_MAINTAINER:pn-rpcbind = "Hongxu Jia "
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 00..3086f8eb42
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,34 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the REPO_REV variable to a SHA1 that corresponds to
+the version of the recipe.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index 4cddbf1..cf5f6b1 100755
+--- a/repo
b/repo
+@@ -142,7 +142,7 @@ if __name__ == '__main__':
+ REPO_URL = os.environ.get('REPO_URL', None)
+ if not REPO_URL:
+ REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+-REPO_REV = os.environ.get('REPO_REV')
++REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
+ if not REPO_REV:
+ REPO_REV = 'stable'
+ # URL to file bug reports for repo tool issues.
+--
+2.33.1
diff --git a/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
new file mode 100644
index 00..d3888c8bb2
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo/0001-python3-shebang.patch
@@ -0,0 +1,26 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+Yocto does not symlink from python to python3, thus change the shebang from
+python to python3.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 00..9e5282cc9b
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git
repositories, does the uploads to revision control systems, and automates parts
of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo;
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
+
+SRC_URI =
"git://gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+SRC_URI += "file://0001-python3-shebang.patch"
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+
+MIRRORS += "git://gerrit.googlesource.com/git-repo.git
git://github.com/GerritCodeReview/git-repo.git \n"
+
+S = "${WORKDIR}/git"
+
+do_install() {
+ install -d ${D}$
[oe-core][PATCH v3 2/2] base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko
Add git-native and repo-native as prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/classes/base.bbclass | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..3298bd1952 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,11 @@ python () {
elif uri.scheme == "npm":
d.appendVarFlag('do_fetch', 'depends', '
nodejs-native:do_populate_sysroot')
+elif uri.scheme == "repo":
+needsrcrev = True
+d.appendVarFlag('do_fetch', 'depends', '
repo-native:do_populate_sysroot')
+d.appendVarFlag('do_fetch', 'depends', '
git-native:do_populate_sysroot')
+
# *.lz4 should DEPEND on lz4-native for unpacking
if path.endswith('.lz4'):
d.appendVarFlag('do_unpack', 'depends', '
lz4-native:do_populate_sysroot')
--
2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157976):
https://lists.openembedded.org/g/openembedded-core/message/157976
Mute This Topic: https://lists.openembedded.org/mt/86903593/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Richard and Peter, I've addressed your comments and will post the v3 patches momentarily. Quick comment regarding the new overrides syntax, or rather the documentation. I just noticed that my go-to page for the mega-manual is outdated (https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) and that the new location now seems to be https://docs.yoctoproject.org/3.4/singleindex.html. There is a redirect in place, e.g. when trying to call a 3.4 version of the mega manual like this https://www.yoctoproject.org/docs/3.4/mega-manual/mega-manual.html. However, this does not extend to the "latest" page, which is just a source for confusion. Is there any chance we could change that? - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Sun, 2021-11-07 at 09:05 +, Richard Purdie wrote: > On Fri, 2021-11-05 at 14:31 +0100, Jasper Orschulko via > lists.openembedded.org > wrote: > > From: Jasper Orschulko > > > > Add a recipe for repo, prerequisite for the repo fetcher. > > > > Signed-off-by: Jasper Orschulko > > --- > > .../repo/files/0001-python3-shebang.patch | 21 > > .../0001-Set-REPO_REV-to-v2.17.3.patch | 33 > > +++ > > meta/recipes-devtools/repo/repo.inc | 25 ++ > > meta/recipes-devtools/repo/repo_2.17.3.bb | 7 > > 4 files changed, 86 insertions(+) > > create mode 100644 meta/recipes-devtools/repo/files/0001-python3- > > shebang.patch > > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set- > > REPO_REV-to-v2.17.3.patch > > create mode 100644 meta/recipes-devtools/repo/repo.inc > > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb > > This basically looks ok to me, I've some minor comments below. > > The patch is missing adding an entry to: > > meta/conf/distro/include/maintainers.inc > > which is required for OE-Core recipes and will trip up automated > testing if we > don't. > > > > > diff --git a/meta/recipes-devtools/repo/files/0001-python3- > > shebang.patch b/meta/recipes-devtools/repo/files/0001-python3- > > shebang.patch > > new file mode 100644 > > index 00..09ccf58264 > > --- /dev/null > > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch > > > I'd put this in a folder called "repo" rather than files. > > > @@ -0,0 +1,21 @@ > > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 > > 2001 > > +From: Jasper Orschulko > > +Date: Tue, 14 Sep 2021 16:46:51 +0200 > > +Subject: [PATCH] python3 shebang > > + > > +--- > > + repo | 2 +- > > + 1 file changed, 1 insertion(+), 1 deletion(-) > > > > Missing Upstream-Status: (you got the second one ok though! :) > > > +diff --git a/repo b/repo > > +index b13e34c..205e0e5 100755 > > +--- a/repo > > b/repo > > +@@ -1,4 +1,4 @@ > > +-#!/usr/bin/env python > > ++#!/usr/bin/env python3 > > + # -*- coding:utf-8 -*- > > + # > > + # Copyright (C) 2008 The Android Open Source Project > > +-- > > +2.33.0 > > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set- > > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo- > > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch > > new file mode 100644 > > index 00..294a3af53a > > --- /dev/null > > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- > > v2.17.3.patch > > @@ -0,0 +1,33 @@ > > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 > > 2001 > > +From: Jasper Orschulko > > +Date: Thu, 4 Nov 2021 16:55:12 +0100 > > +Subject: [PATCH] Set REPO_REV to v2.17.3 > > + > > +repo is an unusual tool because it downloads all of its own Python > > modules > > +using GPG-signed git tags, and stores those files as part of the > > project > > +that it is working with. > > + > > +So in order to have a reproducible repo installation within the > > project > > +folders, we hardcode the REPO_REV variable to this recipes PV. > > + > > +Upstream-Status: Inappropriate [configuration] > > +Signed-off-by: Jasper Orschulko > > +--- > > + repo | 2 +- > > + 1 file changed, 1 insertion(+), 1 deletion(-) > > + > > +diff --git a/repo b/repo > &g
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Alex, we are getting a bit off topic here, but whatever... ;) > What I mean is that I would try harder to find a workable setup while > keeping the recipes for individual items that need to be built. the individual recipes don't really add that much benefit for us, mostly just maintenance overhead and complexity, as we need to constantly ensure that the various components depend on each other in the right order. The only beneficial aspect to the multitude of recipes is the sstate cache. However, as the packages heavily depend on one another, the benefit is marginal, as we have to rebuild most of the application anyways. Additionally, this is not how the developers build the application, so basically we currently have to maintain two separate build workflows. Changing to repo also improves the developer workflow outside of yocto, e.g. thanks to the notion of "topics". The "devtool" option, whilst duable I assume, would again limit the usability to within the Yocto environment. We are currently trying to get to a point, where we manage most of our (automated) workloads (e.g. testing) within Yocto but historically this was not the case and for daily development this is just not really practical. > Particularly this: > "if you want to create another version with different cmake flags, > you would have to create copies of all 50ish recipes" > looks odd. Why would you make copies? Just set the needed flags with > a variable that is set from a global config, > perhaps the distro or local.conf. As far as I understand, this is not reasonably possible (though I might be wrong. TBH, understanding variable scopes in Yocto is a nightmare). To give this some context, our current base setup looks as follows: - - We have one distro.conf - - We have three image recipes with slightly different config + featureset - - AFAIK an image recipe cannot set variables within another recipe, it just "consumes" existing recipes with their configuration. - -> Thus we have three respective recipes with slightly different configuration. If we where to use individual recipes in this setup, we'd have 50ish x 3 recipes. The approach you are describing might circumvent this particular situation but basically just moves the issue around, as you're now unabe to build all three configurations in one go, but need to start juggling your local.confs. Alternatively you might think about using multi-confs. However, each multiconf doubles the parse times for your recipes. As we already use a couple of multiconfs for different machine configs, we would basically increase the time it takes to parse the recipes by ~fifteen-fold (5 pre-existing multi-confs x 3 (or more) disto confs). Additionally, you now need to start worrying about artifact names, as by default the distro name is not included. So yeah... as far as I can tell, there are multiple ways to approach this issue, but none of them seem straight forward nor pretty. Bitbake as it is is just fundamentally not good at handling highly dynamic configurations. The combination with KAS somewhat defuses the situation, but there are still some situations where there is no easy answer. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Fri, 2021-11-05 at 19:45 +0100, Alexander Kanavin wrote: > On Fri, 5 Nov 2021 at 19:05, Jasper Orschulko > wrote: > > > > But that is exactly what we are doing, by integrating the repo > > fetcher into the yocto workflow, thus improving the yocto tooling > > :) > > Why reinvent the wheel, when you can reuse whats already there? You > > wouldn't reinvent git just for yocto, would you? > > > > > What I mean is that I would try harder to find a workable setup while > keeping the recipes for individual items that need to be built. > For instance, devtool is capable of updating source revisions in > recipes automatically, it may not work exactly as you want, but that > can be fixed. > Yes, repo itself is not proprietary, but your organizational workflow > is. > > Particularly this: > "if you want to create another version with different cmake flags, > you would have to create copies of all 50ish recipes" > looks odd. Why would you make copies? Just set the needed flags with > a variable that is set from a global config, > perhaps the distro or local.conf. > > Alex -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFlPMACgkQYgqew07V MNXoCAf/fN4gi1dfx7r4acfbu7kaw9+3rjdMu34J6SX/ahGjCfbTZAg1Twf8N6RZ LBuOpDkXPU77iJuJixZySve5EZc4A2/NS0hqXEpD0aWf8AXaZ
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
Hi Alex, > that you invented a custom, proprietary > workflow that you have to support entirely by > yourselves We are not though. We are integrating an established tool for multi-repository management into yocto. Google repo is not proprietary by the way, it is permissive licensed. It is just a little "special" in it's modus operandi. > instead of finding ways to do what you need > with standard yocto tooling, improving the > tooling where/if necessary. But that is exactly what we are doing, by integrating the repo fetcher into the yocto workflow, thus improving the yocto tooling :) Why reinvent the wheel, when you can reuse whats already there? You wouldn't reinvent git just for yocto, would you? Best regards, Jasper On 5 November 2021 18:46:27 CET, Alexander Kanavin wrote: >On Fri, 5 Nov 2021 at 16:24, Jasper Orschulko < >jasper.orschu...@iris-sensing.com> wrote: > >> >> In our case some binary which is made up from some 50ish separately >> versioned sub-components. We used to have separate recipes for each of >> this components and static link them using DEPENDS. However, this does >> not scale well. E.g. if you want to create another version with >> different cmake flags, you would have to create copies of all 50ish >> recipes. You could move all the components into a single recipe and do >> 50x git fetcher, but keeping this versioned is still painful. >> Especially since the developers mainly don't use yocto but use the SDK >> for cross-compiling. As such this used to mean, that versioning in the >> development workflow and the release workflow worked differently -> >> more maintenance work. >> >> With repo fetcher we can streamline the versioning within the yocto >> build process and the daily developer workflow, as both can fetch the >> repo manifest from a central stored git repo. The idea is, that the >> repo manifest defaults to develop branch for all component repos and if >> we want to create a release we can use repos own tooling to >> automatically create fixed refspecs from the development manifest. >> something like `repo manifest -r -o release-1.0.0.xml`, push that as a >> release to the manifest repo, set this release as refspec in the recipe >> within the meta layer and proceed with the release process. So no more >> manually maintaining the component versioning within the yocto recipes. >> > >I see. I don't particularly endorse that you invented a custom, proprietary >workflow that you have to support entirely by yourselves - instead of >finding ways to do what you need with standard yocto tooling, improving the >tooling where/if necessary. But if that works out ok for you, then fine :) > >Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#157923): https://lists.openembedded.org/g/openembedded-core/message/157923 Mute This Topic: https://lists.openembedded.org/mt/86841424/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH v2 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi Peter,
> Change to RDEPENDS:${PN} and move it to between do_install() and
> BBCLASSEXTEND.
What does this do? The Yocto manual only mentions RDEPENDS_${PN}.
ack to the rest.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Fri, 2021-11-05 at 16:11 +, Peter Kjellerstedt wrote:
> > -Original Message-
> > From:
> > bitbake-de...@lists.openembedded.org > > On Behalf Of Jasper Orschulko via lists.openembedded.org
> > Sent: den 5 november 2021 16:32
> > To: openembedded-core@lists.openembedded.org
> > Cc: mar...@mko.dev; daniel.baumg...@iris-sensing.com;
> > bitbake-de...@lists.openembedded.org; Jasper Orschulko
> >
> > Subject: [bitbake-devel] [oe-core][PATCH v2 1/2] devtools: Initial
> > recipe for repo 2.17.3
>
> Change the Git subject to:
>
> repo: Add recipe for 2.17.3
>
> >
> > From: Jasper Orschulko
> >
> > Add a recipe for repo, prerequisite for the repo fetcher.
> >
> > Signed-off-by: Jasper Orschulko
> > ---
> > .../repo/files/0001-python3-shebang.patch | 21
> > .../0001-Set-REPO_REV-to-v2.17.3.patch | 33
> > +++
> > meta/recipes-devtools/repo/repo.inc | 25 ++
> > meta/recipes-devtools/repo/repo_2.17.3.bb | 7
> > 4 files changed, 86 insertions(+)
> > create mode 100644 meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch
> > create mode 100644 meta/recipes-devtools/repo/repo.inc
> > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> >
> > diff --git a/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch > b/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 00..09ccf58264
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> > @@ -0,0 +1,21 @@
> > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Tue, 14 Sep 2021 16:46:51 +0200
> > +Subject: [PATCH] python3 shebang
> > +
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index b13e34c..205e0e5 100755
> > +--- a/repo
> > b/repo
> > +@@ -1,4 +1,4 @@
> > +-#!/usr/bin/env python
> > ++#!/usr/bin/env python3
> > + # -*- coding:utf-8 -*-
> > + #
> > + # Copyright (C) 2008 The Android Open Source Project
> > +--
> > +2.33.0
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 00..4d76bfc5d2
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,33 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to this recipes PV.
>
> Change "this recipes PV" to "a SHA1 that corresponds to the version
> of the recipe."
>
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > + REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> >
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 SHA rev set in v2 of the patch series :) - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Fri, 2021-11-05 at 15:20 +0100, Konrad Weihmann wrote: > > > On 05.11.21 15:09, Jasper Orschulko wrote: > > Actually, I don't believe this to be an issue for the following > > reasons: > > > > 1. The recipe does nothing more than taking the wrapper script as > > is > > and copying it to the sysroot. As the repo binary is never called > > when > > building, it does not have the "chance" to look for an update > > during > > the yocto offline build process. The outcome of an offline build is > > therefore predictable. > > > > 2. When using repo within the yocto build itself (with the repo > > fetcher), the repo binary is only ever called during the do_fetch > > step, > > which obviously has no issues with any network access. > > > > > > Okay fine that makes sense, even if it is still phoning home. > Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for > a > SHA revision, as I wouldn't bet my life on google not > deleting/removing > tags from their repos - and unpleasantly we wouldn't even notice that > easily -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFTxsACgkQYgqew07V MNW2ugf/RS4zCitkPe9ZEDJvApdZGie1dJmc3gGvcweNv6d6kd1bFd2bS4BYdbiK BOhHF3kMtdznaJNyQlAHBSmysfvTUWymqUahRT3f7+kEuHqrwQcYJFoDqM3tXR4s 4VKYpbRo8klw62XjuBHdwJ8LxJiz5EWQdQF8gdQTl3Tiojp6y2KGDrjo3j/MYjAk 6KfwP0bAca6o/vMF7Jjhbs9sIlCqOkUXrNRbktEonO7hPZxdpgiEC6QPHCq9ONiN 3hHPhV13gd+BTa/7k1GNPSLW4XV6ye7qc26CpqMDing8dxz4RJnzpk2IxTNFHQ0m 95mLdHAygo7o9RJYdz8NIO4hXVqb7Q== =nQVK -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#157907): https://lists.openembedded.org/g/openembedded-core/message/157907 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH v2 2/2] base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko
Add git-native and repo-native as prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/classes/base.bbclass | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..3298bd1952 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,11 @@ python () {
elif uri.scheme == "npm":
d.appendVarFlag('do_fetch', 'depends', '
nodejs-native:do_populate_sysroot')
+elif uri.scheme == "repo":
+needsrcrev = True
+d.appendVarFlag('do_fetch', 'depends', '
repo-native:do_populate_sysroot')
+d.appendVarFlag('do_fetch', 'depends', '
git-native:do_populate_sysroot')
+
# *.lz4 should DEPEND on lz4-native for unpacking
if path.endswith('.lz4'):
d.appendVarFlag('do_unpack', 'depends', '
lz4-native:do_populate_sysroot')
--
2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157906):
https://lists.openembedded.org/g/openembedded-core/message/157906
Mute This Topic: https://lists.openembedded.org/mt/86843544/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH v2 1/2] devtools: Initial recipe for repo 2.17.3
From: Jasper Orschulko
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
.../repo/files/0001-python3-shebang.patch | 21
.../0001-Set-REPO_REV-to-v2.17.3.patch| 33 +++
meta/recipes-devtools/repo/repo.inc | 25 ++
meta/recipes-devtools/repo/repo_2.17.3.bb | 7
4 files changed, 86 insertions(+)
create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch
create mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo.inc
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
diff --git a/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
new file mode 100644
index 00..09ccf58264
--- /dev/null
+++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
@@ -0,0 +1,21 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 00..4d76bfc5d2
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,33 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the REPO_REV variable to this recipes PV.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index 4cddbf1..cf5f6b1 100755
+--- a/repo
b/repo
+@@ -142,7 +142,7 @@ if __name__ == '__main__':
+ REPO_URL = os.environ.get('REPO_URL', None)
+ if not REPO_URL:
+ REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+-REPO_REV = os.environ.get('REPO_REV')
++REPO_REV = '11b30b91df1f0e03b53da970ec2588e85817bacc'
+ if not REPO_REV:
+ REPO_REV = 'stable'
+ # URL to file bug reports for repo tool issues.
+--
+2.33.1
diff --git a/meta/recipes-devtools/repo/repo.inc
b/meta/recipes-devtools/repo/repo.inc
new file mode 100644
index 00..60b32e4d74
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo.inc
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git
repositories, does the uploads to revision control systems, and automates parts
of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo;
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+
+SRC_URI =
"git://g...@gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+MIRRORS = "git://g...@gerrit.googlesource.com/git-repo.git
git://github.com/GerritCodeReview/git-repo.git \n"
+
+SRC_URI += "file://0001-python3-shebang.patch"
+
+S = "${WORKDIR}/git"
+
+RDEPENDS_${PN} = "python3"
+
+do_install() {
+install -d ${D}${bindir}
+install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
+}
+
+BBCLASSEXTEND = "native nativesdk"
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 00..c26264b9e9
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+require recipes-devtools/repo/repo.inc
+
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
--
2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157905):
https://lists.openembedded.org/g/openembedded-core/message/157905
Mute This Topic: https://lists.openembedded.org/mt/86843542/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Yeah. Unfortunately the current repo fetcher is pretty much broken, this patch is more or less a requirement for changes to the repo fetcher we proposed in the bitbake mailinglist. In our case some binary which is made up from some 50ish separately versioned sub-components. We used to have separate recipes for each of this components and static link them using DEPENDS. However, this does not scale well. E.g. if you want to create another version with different cmake flags, you would have to create copies of all 50ish recipes. You could move all the components into a single recipe and do 50x git fetcher, but keeping this versioned is still painful. Especially since the developers mainly don't use yocto but use the SDK for cross-compiling. As such this used to mean, that versioning in the development workflow and the release workflow worked differently -> more maintenance work. With repo fetcher we can streamline the versioning within the yocto build process and the daily developer workflow, as both can fetch the repo manifest from a central stored git repo. The idea is, that the repo manifest defaults to develop branch for all component repos and if we want to create a release we can use repos own tooling to automatically create fixed refspecs from the development manifest. something like `repo manifest -r -o release-1.0.0.xml`, push that as a release to the manifest repo, set this release as refspec in the recipe within the meta layer and proceed with the release process. So no more manually maintaining the component versioning within the yocto recipes. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Fri, 2021-11-05 at 15:20 +0100, Alexander Kanavin wrote: > Wait, what is the use case for the repo fetcher? > > Alex > > On Fri, 5 Nov 2021 at 14:31, Jasper Orschulko via > lists.openembedded.org > wrote: > > From: Jasper Orschulko > > > > Add a recipe for repo, prerequisite for the repo fetcher. > > > > Signed-off-by: Jasper Orschulko > > --- > > .../repo/files/0001-python3-shebang.patch | 21 > > .../0001-Set-REPO_REV-to-v2.17.3.patch | 33 > > +++ > > meta/recipes-devtools/repo/repo.inc | 25 ++ > > meta/recipes-devtools/repo/repo_2.17.3.bb | 7 > > 4 files changed, 86 insertions(+) > > create mode 100644 meta/recipes-devtools/repo/files/0001-python3- > > shebang.patch > > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001- > > Set- > > REPO_REV-to-v2.17.3.patch > > create mode 100644 meta/recipes-devtools/repo/repo.inc > > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb > > > > diff --git a/meta/recipes-devtools/repo/files/0001-python3- > > shebang.patch b/meta/recipes-devtools/repo/files/0001-python3- > > shebang.patch > > new file mode 100644 > > index 00..09ccf58264 > > --- /dev/null > > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch > > @@ -0,0 +1,21 @@ > > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 > > 2001 > > +From: Jasper Orschulko > > +Date: Tue, 14 Sep 2021 16:46:51 +0200 > > +Subject: [PATCH] python3 shebang > > + > > +--- > > + repo | 2 +- > > + 1 file changed, 1 insertion(+), 1 deletion(-) > > + > > +diff --git a/repo b/repo > > +index b13e34c..205e0e5 100755 > > +--- a/repo > > b/repo > > +@@ -1,4 +1,4 @@ > > +-#!/usr/bin/env python > > ++#!/usr/bin/env python3 > > + # -*- coding:utf-8 -*- > > + # > > + # Copyright (C) 2008 The Android Open Source Project > > +-- > > +2.33.0 > > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set- > > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo- > > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch > > new file mode 100644 > > index 00..294a3af53a > > --- /dev/null > > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- > > v2.17.3.patch > > @@ -0,0 +1,33 @@ > > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 > > 2001 > > +From: Jasper Orschulko > > +Date: Thu, 4 Nov 2021 16:55:12 +0100 > > +Subject: [PATCH] Set REPO_REV to v2.17.3 > > + > > +repo is an unusual tool because it downloads all of its own Python > > modules > > +using GPG-signed git tags, and stores those files as part of the > >
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 > Okay fine that makes sense, even if it is still phoning home. True. Bad software design at best. But luckily not an issue from a build system perspective. :) > Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for > a > SHA revision, as I wouldn't bet my life on google not > deleting/removing > tags from their repos - and unpleasantly we wouldn't even notice that > easily Fair point, will do! - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Fri, 2021-11-05 at 15:20 +0100, Konrad Weihmann wrote: > > > On 05.11.21 15:09, Jasper Orschulko wrote: > > Actually, I don't believe this to be an issue for the following > > reasons: > > > > 1. The recipe does nothing more than taking the wrapper script as is > > and copying it to the sysroot. As the repo binary is never called > > when > > building, it does not have the "chance" to look for an update during > > the yocto offline build process. The outcome of an offline build is > > therefore predictable. > > > > 2. When using repo within the yocto build itself (with the repo > > fetcher), the repo binary is only ever called during the do_fetch > > step, > > which obviously has no issues with any network access. > > > > > > Okay fine that makes sense, even if it is still phoning home. > Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for a > SHA revision, as I wouldn't bet my life on google not deleting/removing > tags from their repos - and unpleasantly we wouldn't even notice that > easily -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFRU4ACgkQYgqew07V MNX3ywf+MQAwmH9TVF0izfVVIYhwiTASgZLlG0pwMBXfQw6iIeWAXmO8LccJNrwb fceVMFFXPi1/2SjAdjhOzzh5X7uSGbcVT0G13RpeuYfsnu7HyJ3n6TAdzoXiQtNa PqJqPVsZBEIpHilTYEZqPu/ENHNq74Sj/sQXqM0XjNMTaJKKW7Syc9jbGdEdn+MV gU9iNIQmYWEPSUS2Tp+lgFh9xC3qQOQJZPpcJ5V3hWfAIibaRMed4kGdd3mlJsYj /Aoq+7gXe1iZOXB7OxXX+bk7ueoHCnp7QUPyWFVxslKx/r/faH+f+PwHmJHZp0VD gOrzbjaIK2E2CVnRobaOkSknIffJzQ== =Tfz+ -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#157902): https://lists.openembedded.org/g/openembedded-core/message/157902 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Actually, I don't believe this to be an issue for the following reasons: 1. The recipe does nothing more than taking the wrapper script as is and copying it to the sysroot. As the repo binary is never called when building, it does not have the "chance" to look for an update during the yocto offline build process. The outcome of an offline build is therefore predictable. 2. When using repo within the yocto build itself (with the repo fetcher), the repo binary is only ever called during the do_fetch step, which obviously has no issues with any network access. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Fri, 2021-11-05 at 14:47 +0100, Jasper Orschulko wrote: > Hi Konrad, > > that might be a good point. I patched the REPO_REV to a fixed refspec > within the wrapper tool, as to stop repo from using the latest stable > release. But you are right, and this might not be enough. Will do > some > more digging on this. > -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFOvsACgkQYgqew07V MNVfxQf/VZLnjKv7To1VaqM6dMIX/R3P9cWQtuTxOg6G05Ziooizao5VgQDUXEIy Zs3h5lCFMuXdJlEZc9q2AJQC55BIGqtW8Njo6lNQDPI0Iz51HpXhNgsWdoC3E7qB tphfw+H0WNnIhtDMkK/+hUAZWAsHHUb7I3rAyruE34GKqvEmbBwHo45B0NpgeCqF rNPH7FeI9VvehloZB/5+jnJbrnmyWWogXCNx3PXPlgyrnxyRDuibOUivW6R/MRGd RkSdYvllMgS9ajBLyK/PEyeA6rupEVQquKv0n9ghxZDOzVm6mboCvBMcjxsaSCpR UlFP+h+4sYw0DTfAN9J4UIGWq4VmJA== =P/xG -END PGP SIGNATURE- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#157898): https://lists.openembedded.org/g/openembedded-core/message/157898 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi Konrad,
that might be a good point. I patched the REPO_REV to a fixed refspec
within the wrapper tool, as to stop repo from using the latest stable
release. But you are right, and this might not be enough. Will do some
more digging on this.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
https://iris-sensing.com/
On Fri, 2021-11-05 at 14:35 +0100, Konrad Weihmann wrote:
> Is this implementation of repo being patched against phoning home to
> google everytime and trying to do that pesky auto-update of the tool?
>
> Otherwise this will become an issue when at one point in time an
> (then)
> outdated version of the tool is used in a BB_NO_NETWORK build.
>
> On 05.11.21 14:31, Jasper Orschulko via lists.openembedded.org wrote:
> > From: Jasper Orschulko
> >
> > Add a recipe for repo, prerequisite for the repo fetcher.
> >
> > Signed-off-by: Jasper Orschulko
> > ---
> > .../repo/files/0001-python3-shebang.patch | 21
> > .../0001-Set-REPO_REV-to-v2.17.3.patch | 33
> > +++
> > meta/recipes-devtools/repo/repo.inc | 25 ++
> > meta/recipes-devtools/repo/repo_2.17.3.bb | 7
> > 4 files changed, 86 insertions(+)
> > create mode 100644 meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-REPO_REV-to-v2.17.3.patch
> > create mode 100644 meta/recipes-devtools/repo/repo.inc
> > create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> >
> > diff --git a/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 00..09ccf58264
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> > @@ -0,0 +1,21 @@
> > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Tue, 14 Sep 2021 16:46:51 +0200
> > +Subject: [PATCH] python3 shebang
> > +
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index b13e34c..205e0e5 100755
> > +--- a/repo
> > b/repo
> > +@@ -1,4 +1,4 @@
> > +-#!/usr/bin/env python
> > ++#!/usr/bin/env python3
> > + # -*- coding:utf-8 -*-
> > + #
> > + # Copyright (C) 2008 The Android Open Source Project
> > +--
> > +2.33.0
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 00..294a3af53a
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,33 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to this recipes PV.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> >
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > + REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> > ++REPO_REV = 'v2.17.3'
> > + if not REPO_REV:
> > + REPO_REV = 'stable'
> > + # URL to file bug reports for repo tool issues.
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-
> > devtools
[oe-core][PATCH 2/2] base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko
Add git-native and repo-native as prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
meta/classes/base.bbclass | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..3298bd1952 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,11 @@ python () {
elif uri.scheme == "npm":
d.appendVarFlag('do_fetch', 'depends', '
nodejs-native:do_populate_sysroot')
+elif uri.scheme == "repo":
+needsrcrev = True
+d.appendVarFlag('do_fetch', 'depends', '
repo-native:do_populate_sysroot')
+d.appendVarFlag('do_fetch', 'depends', '
git-native:do_populate_sysroot')
+
# *.lz4 should DEPEND on lz4-native for unpacking
if path.endswith('.lz4'):
d.appendVarFlag('do_unpack', 'depends', '
lz4-native:do_populate_sysroot')
--
2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157895):
https://lists.openembedded.org/g/openembedded-core/message/157895
Mute This Topic: https://lists.openembedded.org/mt/86840280/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
From: Jasper Orschulko
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by: Jasper Orschulko
---
.../repo/files/0001-python3-shebang.patch | 21
.../0001-Set-REPO_REV-to-v2.17.3.patch| 33 +++
meta/recipes-devtools/repo/repo.inc | 25 ++
meta/recipes-devtools/repo/repo_2.17.3.bb | 7
4 files changed, 86 insertions(+)
create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch
create mode 100644
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo.inc
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
diff --git a/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
new file mode 100644
index 00..09ccf58264
--- /dev/null
+++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
@@ -0,0 +1,21 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 00..294a3af53a
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,33 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the REPO_REV variable to this recipes PV.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index 4cddbf1..cf5f6b1 100755
+--- a/repo
b/repo
+@@ -142,7 +142,7 @@ if __name__ == '__main__':
+ REPO_URL = os.environ.get('REPO_URL', None)
+ if not REPO_URL:
+ REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+-REPO_REV = os.environ.get('REPO_REV')
++REPO_REV = 'v2.17.3'
+ if not REPO_REV:
+ REPO_REV = 'stable'
+ # URL to file bug reports for repo tool issues.
+--
+2.33.1
diff --git a/meta/recipes-devtools/repo/repo.inc
b/meta/recipes-devtools/repo/repo.inc
new file mode 100644
index 00..60b32e4d74
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo.inc
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git
repositories, does the uploads to revision control systems, and automates parts
of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo;
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+
+SRC_URI =
"git://g...@gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+MIRRORS = "git://g...@gerrit.googlesource.com/git-repo.git
git://github.com/GerritCodeReview/git-repo.git \n"
+
+SRC_URI += "file://0001-python3-shebang.patch"
+
+S = "${WORKDIR}/git"
+
+RDEPENDS_${PN} = "python3"
+
+do_install() {
+install -d ${D}${bindir}
+install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
+}
+
+BBCLASSEXTEND = "native nativesdk"
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 00..c26264b9e9
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+require recipes-devtools/repo/repo.inc
+
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
--
2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157894):
https://lists.openembedded.org/g/openembedded-core/message/157894
Mute This Topic: https://lists.openembedded.org/mt/86840272/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/2] fetch2: Fix race condition in latest_revision
From: Martin Koppehel Setting latest_revision contained a race condition, where it would be set to an empty string, if the hash calculation function would take to long. Signed-off-by: Jasper Orschulko --- lib/bb/fetch2/__init__.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 6a38cb09..9dc23d05 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1602,7 +1602,9 @@ class FetchMethod(object): try: return revs[key] except KeyError: -revs[key] = rev = self._latest_revision(ud, d, name) +rev = self._latest_revision(ud, d, name) +if rev != '': +revs[key] = rev return rev def sortable_revision(self, ud, d, name): -- 2.33.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#157893): https://lists.openembedded.org/g/openembedded-core/message/157893 Mute This Topic: https://lists.openembedded.org/mt/86840266/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/2] fetch2/repo: Implement AUTOREV for repo fetcher
From: Martin Koppehel
- Implement AUTOINC and submodule support for REPO provider
- Implement full srcrev support
- Add comments and fixup empty DL_DIR initialization
- Distinguish between artificial and plain rev
- Comments/documentation
The previous implementation of the repo fetcher could not handle updates
to the repo manifest file, nor deal with dynamic refspecs within this
manifest.
This patch fixes these shortcomings as follows:
During the recipe parsing phase, the repository containing the repo
manifest is cloned. This is done, as we need to parse the XML file
contained within, in order to discover all involved git repositories. A
combined hash is then calculated from the manifest repo, as well as any
git repo specified in the manifest. This hash is used for determining the
necessity of an update.
Additionally, the recipe will throw an error if the repo source is set to
a fixed revision but one or more repositories within the manifest
reference a dynamic refspec. This is done to ensure the reproducibility
of a version-pinned recipe.
Signed-off-by: Jasper Orschulko
---
lib/bb/fetch2/repo.py | 226 --
1 file changed, 198 insertions(+), 28 deletions(-)
diff --git a/lib/bb/fetch2/repo.py b/lib/bb/fetch2/repo.py
index fa4cb814..22ee5b80 100644
--- a/lib/bb/fetch2/repo.py
+++ b/lib/bb/fetch2/repo.py
@@ -3,6 +3,7 @@ BitBake "Fetch" repo (git) implementation
"""
+# Copyright (C) 2021 Martin Koppehel , iris-GmbH infrared &
intelligent sensors
# Copyright (C) 2009 Tom Rini
#
# Based on git.py which is:
@@ -13,10 +14,13 @@ BitBake "Fetch" repo (git) implementation
import os
import bb
+import hashlib
+import xml.etree.ElementTree as ET
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
from bb.fetch2 import logger
+
class Repo(FetchMethod):
"""Class to fetch a module or modules from repo (git) repositories"""
def supports(self, ud, d):
@@ -27,46 +31,74 @@ class Repo(FetchMethod):
def urldata_init(self, ud, d):
"""
-We don"t care about the git rev of the manifests repository, but
-we do care about the manifest to use. The default is "default".
-We also care about the branch or tag to be used. The default is
-"master".
+We do care about the rev of the manifests repository, as well as the
+manifest file. However, when SRCREV=AUTOINC, then we use the specified
+branch in SRC_URI, with a fallback to master.
+use sm=fetch to fetch possibly referenced submodules in repositories.
"""
ud.basecmd = d.getVar("FETCHCMD_repo") or "/usr/bin/env repo"
+ud.gitcmd = d.getVar("FETCHCMD_git") or "git -c
core.fsyncobjectfiles=0"
ud.proto = ud.parm.get('protocol', 'git')
ud.branch = ud.parm.get('branch', 'master')
+
+ud.submodules = ud.parm.get('sm', 'fetch')
ud.manifest = ud.parm.get('manifest', 'default.xml')
if not ud.manifest.endswith('.xml'):
ud.manifest += '.xml'
-ud.localfile = d.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host,
ud.path.replace("/", "."), ud.manifest, ud.branch))
+repodir = d.getVar("REPODIR") or (d.getVar("DL_DIR") + "/repo")
+gitsrcname = "%s%s.%s" % (ud.host, ud.path.replace("/", "."),
ud.manifest)
+ud.codir = os.path.join(repodir, d.getVar("BPN"), gitsrcname)
+
+if ud.user:
+ud.username = ud.user + "@"
+else:
+ud.username = ""
+ud.remoteRepo = "%s://%s%s%s" % (ud.proto, ud.username, ud.host,
ud.path)
+
+ud.repodir = os.path.join(ud.codir, "repo")
+# a temporary directory to compute _latest_revision
+ud.tempdir = os.path.join(ud.codir, "temp")
+ud.stampfile = os.path.join(ud.codir, "__hash.txt")
+ud.setup_revisions(d)
+
+# ud.localfile is used to fill localpath, where the downloaded tarball
is stored.
+# in our case, we want something like repo_$GIT_URL_$MANIFEST_$SRCREV
+# todo: do we want the packagename?
+ud.localfile = "repo_%s%s_%s_%s.tar.gz" % (ud.host,
ud.path.replace("/", "."), ud.manifest, d.getVar("SRCREV"))
+
+def need_update(self, ud, d):
+if d.getVar("SRCREV") == "AUTOINC":
+return True
+return os.path.exists(ud.localfile)
def download(self, ud, d):
"""Fetch url"""
-if os.access(os.path.join(d.getVar("DL_DIR"), ud.localfile), os.R_OK):
-logger.debug("%s already exists (or was stashed
[oe-core][dunfell][PATCH v3] libx11: Fix CVE-2021-31535
https://lists.x.org/archives/xorg-announce/2021-May/003088.html XLookupColor() and other X libraries function lack proper validation of the length of their string parameters. If those parameters can be controlled by an external application (for instance a color name that can be emitted via a terminal control sequence) it can lead to the emission of extra X protocol requests to the X server. Signed-off-by: Jasper Orschulko --- .../xorg-lib/libx11/CVE-2021-31535.patch | 333 ++ .../recipes-graphics/xorg-lib/libx11_1.6.9.bb | 1 + 2 files changed, 334 insertions(+) create mode 100644 meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch diff --git a/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch new file mode 100644 index 00..97c4c17a8a --- /dev/null +++ b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch @@ -0,0 +1,333 @@ +From 5c539ee6aba5872fcc73aa3d46a4e9a33dc030db Mon Sep 17 00:00:00 2001 +From: Matthieu Herrb +Date: Fri, 19 Feb 2021 15:30:39 +0100 +Subject: [PATCH] Reject string longer than USHRT_MAX before sending them on + the wire + +The X protocol uses CARD16 values to represent the length so +this would overflow. + +CVE-2021-31535 + +Signed-off-by: Matthieu Herrb + +https://lists.x.org/archives/xorg-announce/2021-May/003088.html + +XLookupColor() and other X libraries function lack proper validation +of the length of their string parameters. If those parameters can be +controlled by an external application (for instance a color name that +can be emitted via a terminal control sequence) it can lead to the +emission of extra X protocol requests to the X server. + +Upstream-Status: Backport [https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/8d2e02ae650f00c4a53deb625211a0527126c605] +CVE: CVE-2021-31535 +Signed-off-by: Jasper Orschulko +--- + src/Font.c | 6 -- + src/FontInfo.c | 3 +++ + src/FontNames.c | 3 +++ + src/GetColor.c | 4 + src/LoadFont.c | 4 + src/LookupCol.c | 6 -- + src/ParseCol.c | 5 - + src/QuExt.c | 5 + + src/SetFPath.c | 8 +++- + src/SetHints.c | 7 +++ + src/StNColor.c | 3 +++ + src/StName.c| 7 ++- + 12 files changed, 54 insertions(+), 7 deletions(-) + +diff --git a/src/Font.c b/src/Font.c +index 09d2ae91..3f468e4b 100644 +--- a/src/Font.c b/src/Font.c +@@ -102,6 +102,8 @@ XFontStruct *XLoadQueryFont( + XF86BigfontCodes *extcodes = _XF86BigfontCodes(dpy); + #endif + ++if (strlen(name) >= USHRT_MAX) ++return NULL; + if (_XF86LoadQueryLocaleFont(dpy, name, _result, (Font *)0)) + return font_result; + LockDisplay(dpy); +@@ -662,8 +664,8 @@ int _XF86LoadQueryLocaleFont( + + if (!name) + return 0; +-l = strlen(name); +-if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-') ++l = (int) strlen(name); ++if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-' || l >= USHRT_MAX) + return 0; + charset = NULL; + /* next three lines stolen from _XkbGetCharset() */ +diff --git a/src/FontInfo.c b/src/FontInfo.c +index f870e431..51b48e29 100644 +--- a/src/FontInfo.c b/src/FontInfo.c +@@ -58,6 +58,9 @@ XFontStruct **info) /* RETURN */ + register xListFontsReq *req; + int j; + ++if (strlen(pattern) >= USHRT_MAX) ++return NULL; ++ + LockDisplay(dpy); + GetReq(ListFontsWithInfo, req); + req->maxNames = maxNames; +diff --git a/src/FontNames.c b/src/FontNames.c +index b78792d6..4dac4916 100644 +--- a/src/FontNames.c b/src/FontNames.c +@@ -51,6 +51,9 @@ int *actualCount)/* RETURN */ + register xListFontsReq *req; + unsigned long rlen = 0; + ++if (strlen(pattern) >= USHRT_MAX) ++return NULL; ++ + LockDisplay(dpy); + GetReq(ListFonts, req); + req->maxNames = maxNames; +diff --git a/src/GetColor.c b/src/GetColor.c +index cd0eb9f6..512ac308 100644 +--- a/src/GetColor.c b/src/GetColor.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include + #include "Xlibint.h" + #include "Xcmsint.h" +@@ -48,6 +49,9 @@ XColor *exact_def) /* RETURN */ + XcmsColor cmsColor_exact; + Status ret; + ++if (strlen(colorname) >= USHRT_MAX) ++return (0); ++ + #ifdef XCMS + /* + * Let's Attempt to use Xcms and i18n approach to Parse Color +diff --git a/src/LoadFont.c b/src/LoadFont.c +index f547976b..85735249 100644 +--- a/src/LoadFont.c b/src/LoadFont.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include "Xlibint.h" + + Font +@@ -38,6 +39,9 @@ XLoadFont ( + Font fid; + register xOpenFontReq *req; + ++if (strlen(name) >= USHRT_MAX) ++return (0); ++ +
Re: [oe-core][dunfell][PATCH v2] libx11: Fix CVE-2021-31535
Obviously, patch files in patch files still confuse me... :D v3 on it's way! On 22 June 2021 16:30:40 CEST, Steve Sakoman wrote: >On Tue, Jun 22, 2021 at 2:09 AM Jasper Orschulko via >lists.openembedded.org >wrote: >> >> https://lists.x.org/archives/xorg-announce/2021-May/003088.html >> >> XLookupColor() and other X libraries function lack proper validation >> of the length of their string parameters. If those parameters can be >> controlled by an external application (for instance a color name that >> can be emitted via a terminal control sequence) it can lead to the >> emission of extra X protocol requests to the X server. >> >> Upstream-Status: Backport >[https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/8d2e02ae650f00c4a53deb625211a0527126c605] >> CVE: CVE-2021-31535 >> Signed-off-by: Jasper Orschulko > >These three lines should be in the patch file itself. See the "Patch >name convention and commit message" section at >https://wiki.yoctoproject.org/wiki/Security for details. > >Thanks! > >Steve > >> --- >> .../xorg-lib/libx11/CVE-2021-31535.patch | 322 >++ >> .../recipes-graphics/xorg-lib/libx11_1.6.9.bb | 1 + >> 2 files changed, 323 insertions(+) >> create mode 100644 >meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch >> >> diff --git >a/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch >b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch >> new file mode 100644 >> index 00..1112320acf >> --- /dev/null >> +++ b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch >> @@ -0,0 +1,322 @@ >> +From 5c539ee6aba5872fcc73aa3d46a4e9a33dc030db Mon Sep 17 00:00:00 >2001 >> +From: Matthieu Herrb >> +Date: Fri, 19 Feb 2021 15:30:39 +0100 >> +Subject: [PATCH] Reject string longer than USHRT_MAX before sending >them on >> + the wire >> + >> +The X protocol uses CARD16 values to represent the length so >> +this would overflow. >> + >> +CVE-2021-31535 >> + >> +Signed-off-by: Matthieu Herrb >> +Signed-off-by: Jasper Orschulko >> +--- >> + src/Font.c | 6 -- >> + src/FontInfo.c | 3 +++ >> + src/FontNames.c | 3 +++ >> + src/GetColor.c | 4 >> + src/LoadFont.c | 4 >> + src/LookupCol.c | 6 -- >> + src/ParseCol.c | 5 - >> + src/QuExt.c | 5 + >> + src/SetFPath.c | 8 +++- >> + src/SetHints.c | 7 +++ >> + src/StNColor.c | 3 +++ >> + src/StName.c| 7 ++- >> + 12 files changed, 54 insertions(+), 7 deletions(-) >> + >> +diff --git a/src/Font.c b/src/Font.c >> +index 09d2ae91..3f468e4b 100644 >> +--- a/src/Font.c >> b/src/Font.c >> +@@ -102,6 +102,8 @@ XFontStruct *XLoadQueryFont( >> + XF86BigfontCodes *extcodes = _XF86BigfontCodes(dpy); >> + #endif >> + >> ++if (strlen(name) >= USHRT_MAX) >> ++return NULL; >> + if (_XF86LoadQueryLocaleFont(dpy, name, _result, (Font >*)0)) >> + return font_result; >> + LockDisplay(dpy); >> +@@ -662,8 +664,8 @@ int _XF86LoadQueryLocaleFont( >> + >> + if (!name) >> + return 0; >> +-l = strlen(name); >> +-if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-') >> ++l = (int) strlen(name); >> ++if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-' || l >= >USHRT_MAX) >> + return 0; >> + charset = NULL; >> + /* next three lines stolen from _XkbGetCharset() */ >> +diff --git a/src/FontInfo.c b/src/FontInfo.c >> +index f870e431..51b48e29 100644 >> +--- a/src/FontInfo.c >> b/src/FontInfo.c >> +@@ -58,6 +58,9 @@ XFontStruct **info) /* RETURN */ >> + register xListFontsReq *req; >> + int j; >> + >> ++if (strlen(pattern) >= USHRT_MAX) >> ++return NULL; >> ++ >> + LockDisplay(dpy); >> + GetReq(ListFontsWithInfo, req); >> + req->maxNames = maxNames; >> +diff --git a/src/FontNames.c b/src/FontNames.c >> +index b78792d6..4dac4916 100644 >> +--- a/src/FontNames.c >> b/src/FontNames.c >> +@@ -51,6 +51,9 @@ int *actualCount)/* RETURN */ >> + register xListFontsReq *req; >> + unsigned long rlen = 0; >> + >> ++if (strlen(pattern) >= USHRT_MAX) >> ++return NULL; >> ++ >> + LockDisplay(dpy); >> + GetReq(ListFonts, req); >> + req->maxNames = maxNames; >> +diff --git a/src/GetColor.c b/src/GetColor.c >> +index
[oe-core][dunfell][PATCH v2] libx11: Fix CVE-2021-31535
https://lists.x.org/archives/xorg-announce/2021-May/003088.html XLookupColor() and other X libraries function lack proper validation of the length of their string parameters. If those parameters can be controlled by an external application (for instance a color name that can be emitted via a terminal control sequence) it can lead to the emission of extra X protocol requests to the X server. Upstream-Status: Backport [https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/8d2e02ae650f00c4a53deb625211a0527126c605] CVE: CVE-2021-31535 Signed-off-by: Jasper Orschulko --- .../xorg-lib/libx11/CVE-2021-31535.patch | 322 ++ .../recipes-graphics/xorg-lib/libx11_1.6.9.bb | 1 + 2 files changed, 323 insertions(+) create mode 100644 meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch diff --git a/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch new file mode 100644 index 00..1112320acf --- /dev/null +++ b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch @@ -0,0 +1,322 @@ +From 5c539ee6aba5872fcc73aa3d46a4e9a33dc030db Mon Sep 17 00:00:00 2001 +From: Matthieu Herrb +Date: Fri, 19 Feb 2021 15:30:39 +0100 +Subject: [PATCH] Reject string longer than USHRT_MAX before sending them on + the wire + +The X protocol uses CARD16 values to represent the length so +this would overflow. + +CVE-2021-31535 + +Signed-off-by: Matthieu Herrb +Signed-off-by: Jasper Orschulko +--- + src/Font.c | 6 -- + src/FontInfo.c | 3 +++ + src/FontNames.c | 3 +++ + src/GetColor.c | 4 + src/LoadFont.c | 4 + src/LookupCol.c | 6 -- + src/ParseCol.c | 5 - + src/QuExt.c | 5 + + src/SetFPath.c | 8 +++- + src/SetHints.c | 7 +++ + src/StNColor.c | 3 +++ + src/StName.c| 7 ++- + 12 files changed, 54 insertions(+), 7 deletions(-) + +diff --git a/src/Font.c b/src/Font.c +index 09d2ae91..3f468e4b 100644 +--- a/src/Font.c b/src/Font.c +@@ -102,6 +102,8 @@ XFontStruct *XLoadQueryFont( + XF86BigfontCodes *extcodes = _XF86BigfontCodes(dpy); + #endif + ++if (strlen(name) >= USHRT_MAX) ++return NULL; + if (_XF86LoadQueryLocaleFont(dpy, name, _result, (Font *)0)) + return font_result; + LockDisplay(dpy); +@@ -662,8 +664,8 @@ int _XF86LoadQueryLocaleFont( + + if (!name) + return 0; +-l = strlen(name); +-if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-') ++l = (int) strlen(name); ++if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-' || l >= USHRT_MAX) + return 0; + charset = NULL; + /* next three lines stolen from _XkbGetCharset() */ +diff --git a/src/FontInfo.c b/src/FontInfo.c +index f870e431..51b48e29 100644 +--- a/src/FontInfo.c b/src/FontInfo.c +@@ -58,6 +58,9 @@ XFontStruct **info) /* RETURN */ + register xListFontsReq *req; + int j; + ++if (strlen(pattern) >= USHRT_MAX) ++return NULL; ++ + LockDisplay(dpy); + GetReq(ListFontsWithInfo, req); + req->maxNames = maxNames; +diff --git a/src/FontNames.c b/src/FontNames.c +index b78792d6..4dac4916 100644 +--- a/src/FontNames.c b/src/FontNames.c +@@ -51,6 +51,9 @@ int *actualCount)/* RETURN */ + register xListFontsReq *req; + unsigned long rlen = 0; + ++if (strlen(pattern) >= USHRT_MAX) ++return NULL; ++ + LockDisplay(dpy); + GetReq(ListFonts, req); + req->maxNames = maxNames; +diff --git a/src/GetColor.c b/src/GetColor.c +index cd0eb9f6..512ac308 100644 +--- a/src/GetColor.c b/src/GetColor.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include + #include "Xlibint.h" + #include "Xcmsint.h" +@@ -48,6 +49,9 @@ XColor *exact_def) /* RETURN */ + XcmsColor cmsColor_exact; + Status ret; + ++if (strlen(colorname) >= USHRT_MAX) ++return (0); ++ + #ifdef XCMS + /* + * Let's Attempt to use Xcms and i18n approach to Parse Color +diff --git a/src/LoadFont.c b/src/LoadFont.c +index f547976b..85735249 100644 +--- a/src/LoadFont.c b/src/LoadFont.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include "Xlibint.h" + + Font +@@ -38,6 +39,9 @@ XLoadFont ( + Font fid; + register xOpenFontReq *req; + ++if (strlen(name) >= USHRT_MAX) ++return (0); ++ + if (_XF86LoadQueryLocaleFont(dpy, name, (XFontStruct **)0, )) + return fid; + +diff --git a/src/LookupCol.c b/src/LookupCol.c +index f7f969f5..cd9b1368 100644 +--- a/src/LookupCol.c b/src/LookupCol.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include + #include "Xlibint.h" + #inc
[oe-core][dunfell][PATCH] libx11: Fix CVE-2021-31535
https://lists.x.org/archives/xorg-announce/2021-May/003088.html XLookupColor() and other X libraries function lack proper validation of the length of their string parameters. If those parameters can be controlled by an external application (for instance a color name that can be emitted via a terminal control sequence) it can lead to the emission of extra X protocol requests to the X server. Upstream-Status: Backported [https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/8d2e02ae650f00c4a53deb625211a0527126c605] CVE: CVE-2021-31535 Signed-off-by: Jasper Orschulko --- .../xorg-lib/libx11/CVE-2021-31535.patch | 322 ++ .../recipes-graphics/xorg-lib/libx11_1.6.9.bb | 1 + 2 files changed, 323 insertions(+) create mode 100644 meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch diff --git a/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch new file mode 100644 index 00..1112320acf --- /dev/null +++ b/meta/recipes-graphics/xorg-lib/libx11/CVE-2021-31535.patch @@ -0,0 +1,322 @@ +From 5c539ee6aba5872fcc73aa3d46a4e9a33dc030db Mon Sep 17 00:00:00 2001 +From: Matthieu Herrb +Date: Fri, 19 Feb 2021 15:30:39 +0100 +Subject: [PATCH] Reject string longer than USHRT_MAX before sending them on + the wire + +The X protocol uses CARD16 values to represent the length so +this would overflow. + +CVE-2021-31535 + +Signed-off-by: Matthieu Herrb +Signed-off-by: Jasper Orschulko +--- + src/Font.c | 6 -- + src/FontInfo.c | 3 +++ + src/FontNames.c | 3 +++ + src/GetColor.c | 4 + src/LoadFont.c | 4 + src/LookupCol.c | 6 -- + src/ParseCol.c | 5 - + src/QuExt.c | 5 + + src/SetFPath.c | 8 +++- + src/SetHints.c | 7 +++ + src/StNColor.c | 3 +++ + src/StName.c| 7 ++- + 12 files changed, 54 insertions(+), 7 deletions(-) + +diff --git a/src/Font.c b/src/Font.c +index 09d2ae91..3f468e4b 100644 +--- a/src/Font.c b/src/Font.c +@@ -102,6 +102,8 @@ XFontStruct *XLoadQueryFont( + XF86BigfontCodes *extcodes = _XF86BigfontCodes(dpy); + #endif + ++if (strlen(name) >= USHRT_MAX) ++return NULL; + if (_XF86LoadQueryLocaleFont(dpy, name, _result, (Font *)0)) + return font_result; + LockDisplay(dpy); +@@ -662,8 +664,8 @@ int _XF86LoadQueryLocaleFont( + + if (!name) + return 0; +-l = strlen(name); +-if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-') ++l = (int) strlen(name); ++if (l < 2 || name[l - 1] != '*' || name[l - 2] != '-' || l >= USHRT_MAX) + return 0; + charset = NULL; + /* next three lines stolen from _XkbGetCharset() */ +diff --git a/src/FontInfo.c b/src/FontInfo.c +index f870e431..51b48e29 100644 +--- a/src/FontInfo.c b/src/FontInfo.c +@@ -58,6 +58,9 @@ XFontStruct **info) /* RETURN */ + register xListFontsReq *req; + int j; + ++if (strlen(pattern) >= USHRT_MAX) ++return NULL; ++ + LockDisplay(dpy); + GetReq(ListFontsWithInfo, req); + req->maxNames = maxNames; +diff --git a/src/FontNames.c b/src/FontNames.c +index b78792d6..4dac4916 100644 +--- a/src/FontNames.c b/src/FontNames.c +@@ -51,6 +51,9 @@ int *actualCount)/* RETURN */ + register xListFontsReq *req; + unsigned long rlen = 0; + ++if (strlen(pattern) >= USHRT_MAX) ++return NULL; ++ + LockDisplay(dpy); + GetReq(ListFonts, req); + req->maxNames = maxNames; +diff --git a/src/GetColor.c b/src/GetColor.c +index cd0eb9f6..512ac308 100644 +--- a/src/GetColor.c b/src/GetColor.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include + #include "Xlibint.h" + #include "Xcmsint.h" +@@ -48,6 +49,9 @@ XColor *exact_def) /* RETURN */ + XcmsColor cmsColor_exact; + Status ret; + ++if (strlen(colorname) >= USHRT_MAX) ++return (0); ++ + #ifdef XCMS + /* + * Let's Attempt to use Xcms and i18n approach to Parse Color +diff --git a/src/LoadFont.c b/src/LoadFont.c +index f547976b..85735249 100644 +--- a/src/LoadFont.c b/src/LoadFont.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include "Xlibint.h" + + Font +@@ -38,6 +39,9 @@ XLoadFont ( + Font fid; + register xOpenFontReq *req; + ++if (strlen(name) >= USHRT_MAX) ++return (0); ++ + if (_XF86LoadQueryLocaleFont(dpy, name, (XFontStruct **)0, )) + return fid; + +diff --git a/src/LookupCol.c b/src/LookupCol.c +index f7f969f5..cd9b1368 100644 +--- a/src/LookupCol.c b/src/LookupCol.c +@@ -27,6 +27,7 @@ in this Software without prior written authorization from The Open Group. + #ifdef HAVE_CONFIG_H + #include + #endif ++#include + #include + #include "Xlibint.h" + #inc
Re: [oe-core][dunfell][PATCH] libxml2: Fix CVE-2021-3518
Hi Steve,
sorry about that. Accidental checkout of dunfell-next. I sent a new patch.
Best regards,
Jasper
On 21 June 2021 17:26:14 CEST, Steve Sakoman wrote:
>Sadly this patch won't apply.
>
>Could you rebase it on the current head of dunfell? It seems you
>generated this patch with an older version of dunfell that is missing
>"libxml: fix CVE-2021-3517 CVE-2021-3537":
>
>https://git.openembedded.org/openembedded-core-contrib/commit/?h=stable/dunfell-nut=f177c0ec321f005dd9ce63aec2d700fd53c993ff
>
>Thanks again for helping with CVEs!
>
>Steve
>
>On Mon, Jun 21, 2021 at 4:11 AM Jasper Orschulko via
>lists.openembedded.org
>wrote:
>>
>> There's a flaw in libxml2 in versions before 2.9.11. An attacker who
>is able to submit a crafted file to be processed by an application
>linked with libxml2 could trigger a use-after-free. The greatest impact
>from this flaw is to confidentiality, integrity, and availability.
>>
>> Upstream-Status: Backport [from fedora:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1954243]
>>
>> Signed-off-by: Jasper Orschulko
>> ---
>> .../libxml/libxml2/CVE-2021-3518.patch| 108
>++
>> meta/recipes-core/libxml/libxml2_2.9.10.bb| 1 +
>> 2 files changed, 109 insertions(+)
>> create mode 100644
>meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
>>
>> diff --git a/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
>b/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
>> new file mode 100644
>> index 00..c22cccf1b1
>> --- /dev/null
>> +++ b/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
>> @@ -0,0 +1,108 @@
>> +From ac82a514e16eb81b4506e2cba1a1ee45b9f025b5 Mon Sep 17 00:00:00
>2001
>> +From: Nick Wellnhofer
>> +Date: Wed, 10 Jun 2020 16:34:52 +0200
>> +Subject: [PATCH 1/2] Don't recurse into xi:include children in
>> + xmlXIncludeDoProcess
>> +
>> +Otherwise, nested xi:include nodes might result in a use-after-free
>> +if XML_PARSE_NOXINCNODE is specified.
>> +
>> +Found with libFuzzer and ASan.
>> +
>> +The upstream patch 752e5f71d7cea2ca5a7e7c0b8f72ed04ce654be4 has been
>modified,
>> +as to avoid unnecessary modifications to fallback files.
>> +
>> +Signed-off-by: Jasper Orschulko
>> +---
>> + xinclude.c | 24 ++--
>> + 1 file changed, 10 insertions(+), 14 deletions(-)
>> +
>> +diff --git a/xinclude.c b/xinclude.c
>> +index ba850fa5..f260c1a7 100644
>> +--- a/xinclude.c
>> b/xinclude.c
>> +@@ -2392,21 +2392,19 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr
>ctxt, xmlDocPtr doc, xmlNodePtr tree) {
>> + * First phase: lookup the elements in the document
>> + */
>> + cur = tree;
>> +-if (xmlXIncludeTestNode(ctxt, cur) == 1)
>> +- xmlXIncludePreProcessNode(ctxt, cur);
>> + while ((cur != NULL) && (cur != tree->parent)) {
>> + /* TODO: need to work on entities -> stack */
>> +- if ((cur->children != NULL) &&
>> +- (cur->children->type != XML_ENTITY_DECL) &&
>> +- (cur->children->type != XML_XINCLUDE_START) &&
>> +- (cur->children->type != XML_XINCLUDE_END)) {
>> +- cur = cur->children;
>> +- if (xmlXIncludeTestNode(ctxt, cur))
>> +- xmlXIncludePreProcessNode(ctxt, cur);
>> +- } else if (cur->next != NULL) {
>> ++if (xmlXIncludeTestNode(ctxt, cur) == 1) {
>> ++xmlXIncludePreProcessNode(ctxt, cur);
>> ++} else if ((cur->children != NULL) &&
>> ++ (cur->children->type != XML_ENTITY_DECL) &&
>> ++ (cur->children->type != XML_XINCLUDE_START) &&
>> ++ (cur->children->type != XML_XINCLUDE_END)) {
>> ++cur = cur->children;
>> ++continue;
>> ++}
>> ++ if (cur->next != NULL) {
>> + cur = cur->next;
>> +- if (xmlXIncludeTestNode(ctxt, cur))
>> +- xmlXIncludePreProcessNode(ctxt, cur);
>> + } else {
>> + if (cur == tree)
>> + break;
>> +@@ -2416,8 +2414,6 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr ctxt,
>xmlDocPtr doc, xmlNodePtr tree) {
>> + break; /* do */
>> + if (cur->next != NULL) {
>> + cur = cur->next;
>> +- if (xmlXIncludeTestNode(ctxt, cur))
>> +- xmlXIncludePreProcessNode(ctx
[oe-core][dunfell][PATCH v2] libxml2: Fix CVE-2021-3518
There's a flaw in libxml2 in versions before 2.9.11. An attacker who is able to
submit a crafted file to be processed by an application linked with libxml2
could trigger a use-after-free. The greatest impact from this flaw is to
confidentiality, integrity, and availability.
Upstream-Status: Backport [from fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1954243]
Signed-off-by: Jasper Orschulko
---
.../libxml/libxml2/CVE-2021-3518.patch| 108 ++
meta/recipes-core/libxml/libxml2_2.9.10.bb| 1 +
2 files changed, 109 insertions(+)
create mode 100644 meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
diff --git a/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
b/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
new file mode 100644
index 00..c22cccf1b1
--- /dev/null
+++ b/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
@@ -0,0 +1,108 @@
+From ac82a514e16eb81b4506e2cba1a1ee45b9f025b5 Mon Sep 17 00:00:00 2001
+From: Nick Wellnhofer
+Date: Wed, 10 Jun 2020 16:34:52 +0200
+Subject: [PATCH 1/2] Don't recurse into xi:include children in
+ xmlXIncludeDoProcess
+
+Otherwise, nested xi:include nodes might result in a use-after-free
+if XML_PARSE_NOXINCNODE is specified.
+
+Found with libFuzzer and ASan.
+
+The upstream patch 752e5f71d7cea2ca5a7e7c0b8f72ed04ce654be4 has been modified,
+as to avoid unnecessary modifications to fallback files.
+
+Signed-off-by: Jasper Orschulko
+---
+ xinclude.c | 24 ++--
+ 1 file changed, 10 insertions(+), 14 deletions(-)
+
+diff --git a/xinclude.c b/xinclude.c
+index ba850fa5..f260c1a7 100644
+--- a/xinclude.c
b/xinclude.c
+@@ -2392,21 +2392,19 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr ctxt,
xmlDocPtr doc, xmlNodePtr tree) {
+ * First phase: lookup the elements in the document
+ */
+ cur = tree;
+-if (xmlXIncludeTestNode(ctxt, cur) == 1)
+- xmlXIncludePreProcessNode(ctxt, cur);
+ while ((cur != NULL) && (cur != tree->parent)) {
+ /* TODO: need to work on entities -> stack */
+- if ((cur->children != NULL) &&
+- (cur->children->type != XML_ENTITY_DECL) &&
+- (cur->children->type != XML_XINCLUDE_START) &&
+- (cur->children->type != XML_XINCLUDE_END)) {
+- cur = cur->children;
+- if (xmlXIncludeTestNode(ctxt, cur))
+- xmlXIncludePreProcessNode(ctxt, cur);
+- } else if (cur->next != NULL) {
++if (xmlXIncludeTestNode(ctxt, cur) == 1) {
++xmlXIncludePreProcessNode(ctxt, cur);
++} else if ((cur->children != NULL) &&
++ (cur->children->type != XML_ENTITY_DECL) &&
++ (cur->children->type != XML_XINCLUDE_START) &&
++ (cur->children->type != XML_XINCLUDE_END)) {
++cur = cur->children;
++continue;
++}
++ if (cur->next != NULL) {
+ cur = cur->next;
+- if (xmlXIncludeTestNode(ctxt, cur))
+- xmlXIncludePreProcessNode(ctxt, cur);
+ } else {
+ if (cur == tree)
+ break;
+@@ -2416,8 +2414,6 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr ctxt, xmlDocPtr
doc, xmlNodePtr tree) {
+ break; /* do */
+ if (cur->next != NULL) {
+ cur = cur->next;
+- if (xmlXIncludeTestNode(ctxt, cur))
+- xmlXIncludePreProcessNode(ctxt, cur);
+ break; /* do */
+ }
+ } while (cur != NULL);
+--
+2.32.0
+
+
+From 3ad5ac1e39e3cd42f838c1cd27ffd4e9b79e6121 Mon Sep 17 00:00:00 2001
+From: Nick Wellnhofer
+Date: Thu, 22 Apr 2021 19:26:28 +0200
+Subject: [PATCH 2/2] Fix user-after-free with `xmllint --xinclude --dropdtd`
+
+The --dropdtd option can leave dangling pointers in entity reference
+nodes. Make sure to skip these nodes when processing XIncludes.
+
+This also avoids scanning entity declarations and even modifying
+them inadvertently during XInclude processing.
+
+Move from a block list to an allow list approach to avoid descending
+into other node types that can't contain elements.
+
+Fixes #237.
+
+Signed-off-by: Jasper Orschulko
+---
+ xinclude.c | 5 ++---
+ 1 file changed, 2 insertions(+), 3 deletions(-)
+
+diff --git a/xinclude.c b/xinclude.c
+index f260c1a7..d7648529 100644
+--- a/xinclude.c
b/xinclude.c
+@@ -2397,9 +2397,8 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr ctxt, xmlDocPtr
doc, xmlNodePtr tree) {
+ if (xmlXIncludeTestNode(ctxt, cur) == 1) {
+ xmlXIncludePreProcessNode(ctxt, cur);
+ } else if ((cur->children != NULL) &&
+- (cur->children->type != XML_ENTITY_DECL) &&
+- (cur->children->type != XML_XINCLUDE_START) &&
+- (cur->children->type != XML_XINCLUDE_END)) {
++
[oe-core][dunfell][PATCH] libxml2: Fix CVE-2021-3518
There's a flaw in libxml2 in versions before 2.9.11. An attacker who is able to
submit a crafted file to be processed by an application linked with libxml2
could trigger a use-after-free. The greatest impact from this flaw is to
confidentiality, integrity, and availability.
Upstream-Status: Backport [from fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1954243]
Signed-off-by: Jasper Orschulko
---
.../libxml/libxml2/CVE-2021-3518.patch| 108 ++
meta/recipes-core/libxml/libxml2_2.9.10.bb| 1 +
2 files changed, 109 insertions(+)
create mode 100644 meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
diff --git a/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
b/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
new file mode 100644
index 00..c22cccf1b1
--- /dev/null
+++ b/meta/recipes-core/libxml/libxml2/CVE-2021-3518.patch
@@ -0,0 +1,108 @@
+From ac82a514e16eb81b4506e2cba1a1ee45b9f025b5 Mon Sep 17 00:00:00 2001
+From: Nick Wellnhofer
+Date: Wed, 10 Jun 2020 16:34:52 +0200
+Subject: [PATCH 1/2] Don't recurse into xi:include children in
+ xmlXIncludeDoProcess
+
+Otherwise, nested xi:include nodes might result in a use-after-free
+if XML_PARSE_NOXINCNODE is specified.
+
+Found with libFuzzer and ASan.
+
+The upstream patch 752e5f71d7cea2ca5a7e7c0b8f72ed04ce654be4 has been modified,
+as to avoid unnecessary modifications to fallback files.
+
+Signed-off-by: Jasper Orschulko
+---
+ xinclude.c | 24 ++--
+ 1 file changed, 10 insertions(+), 14 deletions(-)
+
+diff --git a/xinclude.c b/xinclude.c
+index ba850fa5..f260c1a7 100644
+--- a/xinclude.c
b/xinclude.c
+@@ -2392,21 +2392,19 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr ctxt,
xmlDocPtr doc, xmlNodePtr tree) {
+ * First phase: lookup the elements in the document
+ */
+ cur = tree;
+-if (xmlXIncludeTestNode(ctxt, cur) == 1)
+- xmlXIncludePreProcessNode(ctxt, cur);
+ while ((cur != NULL) && (cur != tree->parent)) {
+ /* TODO: need to work on entities -> stack */
+- if ((cur->children != NULL) &&
+- (cur->children->type != XML_ENTITY_DECL) &&
+- (cur->children->type != XML_XINCLUDE_START) &&
+- (cur->children->type != XML_XINCLUDE_END)) {
+- cur = cur->children;
+- if (xmlXIncludeTestNode(ctxt, cur))
+- xmlXIncludePreProcessNode(ctxt, cur);
+- } else if (cur->next != NULL) {
++if (xmlXIncludeTestNode(ctxt, cur) == 1) {
++xmlXIncludePreProcessNode(ctxt, cur);
++} else if ((cur->children != NULL) &&
++ (cur->children->type != XML_ENTITY_DECL) &&
++ (cur->children->type != XML_XINCLUDE_START) &&
++ (cur->children->type != XML_XINCLUDE_END)) {
++cur = cur->children;
++continue;
++}
++ if (cur->next != NULL) {
+ cur = cur->next;
+- if (xmlXIncludeTestNode(ctxt, cur))
+- xmlXIncludePreProcessNode(ctxt, cur);
+ } else {
+ if (cur == tree)
+ break;
+@@ -2416,8 +2414,6 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr ctxt, xmlDocPtr
doc, xmlNodePtr tree) {
+ break; /* do */
+ if (cur->next != NULL) {
+ cur = cur->next;
+- if (xmlXIncludeTestNode(ctxt, cur))
+- xmlXIncludePreProcessNode(ctxt, cur);
+ break; /* do */
+ }
+ } while (cur != NULL);
+--
+2.32.0
+
+
+From 3ad5ac1e39e3cd42f838c1cd27ffd4e9b79e6121 Mon Sep 17 00:00:00 2001
+From: Nick Wellnhofer
+Date: Thu, 22 Apr 2021 19:26:28 +0200
+Subject: [PATCH 2/2] Fix user-after-free with `xmllint --xinclude --dropdtd`
+
+The --dropdtd option can leave dangling pointers in entity reference
+nodes. Make sure to skip these nodes when processing XIncludes.
+
+This also avoids scanning entity declarations and even modifying
+them inadvertently during XInclude processing.
+
+Move from a block list to an allow list approach to avoid descending
+into other node types that can't contain elements.
+
+Fixes #237.
+
+Signed-off-by: Jasper Orschulko
+---
+ xinclude.c | 5 ++---
+ 1 file changed, 2 insertions(+), 3 deletions(-)
+
+diff --git a/xinclude.c b/xinclude.c
+index f260c1a7..d7648529 100644
+--- a/xinclude.c
b/xinclude.c
+@@ -2397,9 +2397,8 @@ xmlXIncludeDoProcess(xmlXIncludeCtxtPtr ctxt, xmlDocPtr
doc, xmlNodePtr tree) {
+ if (xmlXIncludeTestNode(ctxt, cur) == 1) {
+ xmlXIncludePreProcessNode(ctxt, cur);
+ } else if ((cur->children != NULL) &&
+- (cur->children->type != XML_ENTITY_DECL) &&
+- (cur->children->type != XML_XINCLUDE_START) &&
+- (cur->children->type != XML_XINCLUDE_END)) {
++
[OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
expat < 4.0 is vulnerable to billion laughs attacks (see
[https://github.com/libexpat/libexpat/issues/34]). This patch backports
the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8
and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream.
Additionally, the SRC_URI had to be adjusted due to renaming of the
source archive
Signed-off-by: Jasper Orschulko
Upstream-Status: Submitted
[https://lists.openembedded.org/g/openembedded-core/message/153030?p=,,,20,0,0,0::Created,,Jasper,20,2,0,83581993]
---
.../expat/expat/CVE-2013-0340.patch | 1758 +
.../expat/expat/libtool-tag.patch | 41 +-
meta/recipes-core/expat/expat_2.2.9.bb| 12 +-
3 files changed, 1782 insertions(+), 29 deletions(-)
create mode 100644 meta/recipes-core/expat/expat/CVE-2013-0340.patch
diff --git a/meta/recipes-core/expat/expat/CVE-2013-0340.patch
b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
new file mode 100644
index 00..5ef749719d
--- /dev/null
+++ b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
@@ -0,0 +1,1758 @@
+From a644ccf25392523b1329872310e24d0fc5f40629 Mon Sep 17 00:00:00 2001
+From: Sebastian Pipping
+Date: Mon, 19 Apr 2021 21:42:51 +0200
+Subject: [PATCH] expat: Backport fix for CVE-2013-0340
+
+Issue: https://github.com/libexpat/libexpat/issues/34
+
+This patch cherry-picks the following commits from upstream release
+2.4.0 onto 2.2.9:
+
+- b1d039607d3d8a042bf0466bfcc1c0f104e353c8
+- 60959f2b491876199879d97c8ed956eabb0c2e73
+
+Upstream-Status: Backport
+CVE: CVE-2013-0340
+Signed-off-by: Jasper Orschulko
+---
+ lib/expat.h | 21 +-
+ lib/internal.h| 30 +
+ lib/libexpat.def |3 +
+ lib/libexpatw.def |3 +
+ lib/xmlparse.c| 1147 +--
+ 5 files changed, 1143 insertions(+), 61 deletions(-)
+
+diff --git a/lib/expat.h b/lib/expat.h
+index 48a6e2a3..0fb70d9d 100644
+--- a/lib/expat.h
b/lib/expat.h
+@@ -115,7 +115,9 @@ enum XML_Error {
+ XML_ERROR_RESERVED_PREFIX_XMLNS,
+ XML_ERROR_RESERVED_NAMESPACE_URI,
+ /* Added in 2.2.1. */
+- XML_ERROR_INVALID_ARGUMENT
++ XML_ERROR_INVALID_ARGUMENT,
++ /* Added in 2.4.0. */
++ XML_ERROR_AMPLIFICATION_LIMIT_BREACH
+ };
+
+ enum XML_Content_Type {
+@@ -997,7 +999,10 @@ enum XML_FeatureEnum {
+ XML_FEATURE_SIZEOF_XML_LCHAR,
+ XML_FEATURE_NS,
+ XML_FEATURE_LARGE_SIZE,
+- XML_FEATURE_ATTR_INFO
++ XML_FEATURE_ATTR_INFO,
++ /* Added in Expat 2.4.0. */
++ XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT,
++ XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT
+ /* Additional features must be added to the end of this enum. */
+ };
+
+@@ -1010,6 +1015,18 @@ typedef struct {
+ XMLPARSEAPI(const XML_Feature *)
+ XML_GetFeatureList(void);
+
++#ifdef XML_DTD
++/* Added in Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionMaximumAmplification(
++XML_Parser parser, float maximumAmplificationFactor);
++
++/* Added in Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionActivationThreshold(
++XML_Parser parser, unsigned long long activationThresholdBytes);
++#endif
++
+ /* Expat follows the semantic versioning convention.
+See http://semver.org.
+ */
+diff --git a/lib/internal.h b/lib/internal.h
+index 60913dab..d8b31fa2 100644
+--- a/lib/internal.h
b/lib/internal.h
+@@ -101,10 +101,40 @@
+ # endif
+ #endif
+
++#include // ULONG_MAX
++
++#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
++# define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
++# if defined(_WIN64) // Note: modifier "td" does not work for MinGW
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "I64d"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#else
++# define EXPAT_FMT_ULL(midpart) "%" midpart "llu"
++# if ! defined(ULONG_MAX)
++#error Compiler did not define ULONG_MAX for us
++# elif ULONG_MAX == 18446744073709551615u // 2^64-1
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "ld"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#endif
++
+ #ifndef UNUSED_P
+ # define UNUSED_P(p) (void)p
+ #endif
+
++/* NOTE BEGIN If you ever patch these defaults to greater values
++ for non-attack XML payload in your environment,
++ please file a bug report with libexpat. Thank you!
++*/
++#define EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT
\
++ 100.0f
++#define EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT
\
++ 8388608 // 8 MiB, 2^23
++/* NOTE END */
++
+ #ifdef __cplusplus
+ extern "C" {
+ #endif
+diff --git a/lib/libexpat.def b/lib/libexpat.def
+index 16faf595..5aefa6df 100644
+--- a/lib/libexpat.def
b/lib/libexpat.def
+@@ -76,
Re: [OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
expat < 4.0 is vulnerable to billion laughs attacks (see
[https://github.com/libexpat/libexpat/issues/34]). This patch backports
the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8
and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream.
Additionally, the SRC_URI had to be adjusted due to renaming of the
source archive
Signed-off-by: Jasper Orschulko
Upstream-Status: Submitted
[https://lists.openembedded.org/g/openembedded-core/message/153030?p=,,,20,0,0,0::Created,,Jasper,20,2,0,83581993
]
---
.../expat/expat/CVE-2013-0340.patch | 1758 +
.../expat/expat/libtool-tag.patch | 41 +-
meta/recipes-core/expat/expat_2.2.9.bb| 12 +-
3 files changed, 1782 insertions(+), 29 deletions(-)
create mode 100644 meta/recipes-core/expat/expat/CVE-2013-0340.patch
diff --git a/meta/recipes-core/expat/expat/CVE-2013-0340.patch
b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
new file mode 100644
index 00..5ef749719d
--- /dev/null
+++ b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
@@ -0,0 +1,1758 @@
+From a644ccf25392523b1329872310e24d0fc5f40629 Mon Sep 17 00:00:00 2001
+From: Sebastian Pipping
+Date: Mon, 19 Apr 2021 21:42:51 +0200
+Subject: [PATCH] expat: Backport fix for CVE-2013-0340
+
+Issue: https://github.com/libexpat/libexpat/issues/34
+
+This patch cherry-picks the following commits from upstream release
+2.4.0 onto 2.2.9:
+
+- b1d039607d3d8a042bf0466bfcc1c0f104e353c8
+- 60959f2b491876199879d97c8ed956eabb0c2e73
+
+Upstream-Status: Backport
+CVE: CVE-2013-0340
+Signed-off-by: Jasper Orschulko
+---
+ lib/expat.h | 21 +-
+ lib/internal.h| 30 +
+ lib/libexpat.def |3 +
+ lib/libexpatw.def |3 +
+ lib/xmlparse.c| 1147 +--
+ 5 files changed, 1143 insertions(+), 61 deletions(-)
+
+diff --git a/lib/expat.h b/lib/expat.h
+index 48a6e2a3..0fb70d9d 100644
+--- a/lib/expat.h
b/lib/expat.h
+@@ -115,7 +115,9 @@ enum XML_Error {
+ XML_ERROR_RESERVED_PREFIX_XMLNS,
+ XML_ERROR_RESERVED_NAMESPACE_URI,
+ /* Added in 2.2.1. */
+- XML_ERROR_INVALID_ARGUMENT
++ XML_ERROR_INVALID_ARGUMENT,
++ /* Added in 2.4.0. */
++ XML_ERROR_AMPLIFICATION_LIMIT_BREACH
+ };
+
+ enum XML_Content_Type {
+@@ -997,7 +999,10 @@ enum XML_FeatureEnum {
+ XML_FEATURE_SIZEOF_XML_LCHAR,
+ XML_FEATURE_NS,
+ XML_FEATURE_LARGE_SIZE,
+- XML_FEATURE_ATTR_INFO
++ XML_FEATURE_ATTR_INFO,
++ /* Added in Expat 2.4.0. */
++
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFA
ULT,
++
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAU
LT
+ /* Additional features must be added to the end of this enum. */
+ };
+
+@@ -1010,6 +1015,18 @@ typedef struct {
+ XMLPARSEAPI(const XML_Feature *)
+ XML_GetFeatureList(void);
+
++#ifdef XML_DTD
++/* Added in Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionMaximumAmplification(
++XML_Parser parser, float maximumAmplificationFactor);
++
++/* Added in Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionActivationThreshold(
++XML_Parser parser, unsigned long long activationThresholdBytes);
++#endif
++
+ /* Expat follows the semantic versioning convention.
+See http://semver.org.
+ */
+diff --git a/lib/internal.h b/lib/internal.h
+index 60913dab..d8b31fa2 100644
+--- a/lib/internal.h
b/lib/internal.h
+@@ -101,10 +101,40 @@
+ # endif
+ #endif
+
++#include // ULONG_MAX
++
++#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
++# define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
++# if defined(_WIN64) // Note: modifier "td" does not work for MinGW
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "I64d"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#else
++# define EXPAT_FMT_ULL(midpart) "%" midpart "llu"
++# if ! defined(ULONG_MAX)
++#error Compiler did not define ULONG_MAX for us
++# elif ULONG_MAX == 18446744073709551615u // 2^64-1
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "ld"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#endif
++
+ #ifndef UNUSED_P
+ # define UNUSED_P(p) (void)p
+ #endif
+
++/* NOTE BEGIN If you ever patch these defaults to greater values
++ for non-attack XML payload in your environment,
++ please file a bug report with libexpat. Thank you!
++*/
++#define
EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT
\
++ 100.0f
++#define
EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT
\
++ 8388608 // 8 MiB, 2^23
++/* NOTE END */
++
+ #ifdef __cplusplus
+ extern "C" {
+ #endif
+diff --git a/lib/libexpat.def b/lib/libexpat.def
+index 16faf595..5aefa6df 100644
+--- a/lib/libexpat.def
b/lib/libexpat.def
+@@ -76,
Re: [OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
I just noticed (additionally to the fact that I messed up the path in
my patch), that the original do_configure_prepend task actually is not
necessary, as there is no ${S}/conftools/libtool.m4 in the 2.9.9
release (neither git, nor sourceforge). While removing a non-existing
file does no harm, I will provide a new patch tomorrow without this
task, for tidiness' sake. ;)
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Ostendstraße 1-14 | 12459 Berlin
https://iris-sensing.com/
On Wed, 2021-06-16 at 20:20 +0200, Jasper Orschulko wrote:
> Revision of the the patch file. Please verify. :)
>
-BEGIN PGP SIGNATURE-
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmDKaXMACgkQYgqew07V
MNXfFQf8C5Lh2OG7tDsP6uQcLEV/J+ieCWN2ylKH5lARVzEPQB5TpVGfgcbdrqPr
66Ia3NS/gKDHtpKDigBOpYau4jFC71252Hpfap13/OiH53/+1es3hwXm5k4xtYYL
WU8iAG7wlKwrj8zSljeElOvOw0EiDLaX/dnhtNKboquKxAgJrQkGG2a3G4KlFQ50
W4xR0Jrx67/UkWJLic1h51vc1RGw7zeDbOwJ+xl+2uXDGCjRtQHmXChpBSInAMjP
r0uza47Oi/+XQGuVYAdYR12lp89Vl7EGAvoy6seKablkVSu7zBMxBi70GyrQdKFw
eM7ixMdqSS1MZ6zdI/64Aaq9XB1wgg==
=EY5+
-END PGP SIGNATURE-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153045):
https://lists.openembedded.org/g/openembedded-core/message/153045
Mute This Topic: https://lists.openembedded.org/mt/83581993/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Revision of the the patch file. Please verify. :)
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Ostendstraße 1-14 | 12459 Berlin
https://iris-sensing.com/
On Wed, 2021-06-16 at 18:19 +0000, Jasper Orschulko wrote:
> expat < 4.0 is vulnerable to billion laughs attacks (see
> [https://github.com/libexpat/libexpat/issues/34]). This patch
> backports
> the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8
> and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream.
>
> Additionally, the SRC_URI had to be adjusted due to renaming of the
> source archive
>
> Signed-off-by: Jasper Orschulko
> ---
> .../expat/expat/CVE-2013-0340.patch | 1758
> +
> .../expat/expat/libtool-tag.patch | 41 +-
> meta/recipes-core/expat/expat_2.2.9.bb | 10 +-
> 3 files changed, 1783 insertions(+), 26 deletions(-)
> create mode 100644 meta/recipes-core/expat/expat/CVE-2013-0340.patch
>
> diff --git a/meta/recipes-core/expat/expat/CVE-2013-0340.patch
> b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
> new file mode 100644
> index 00..5ef749719d
> --- /dev/null
> +++ b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
> @@ -0,0 +1,1758 @@
> +From a644ccf25392523b1329872310e24d0fc5f40629 Mon Sep 17 00:00:00
> 2001
> +From: Sebastian Pipping
> +Date: Mon, 19 Apr 2021 21:42:51 +0200
> +Subject: [PATCH] expat: Backport fix for CVE-2013-0340
> +
> +Issue: https://github.com/libexpat/libexpat/issues/34
> +
> +This patch cherry-picks the following commits from upstream release
> +2.4.0 onto 2.2.9:
> +
> +- b1d039607d3d8a042bf0466bfcc1c0f104e353c8
> +- 60959f2b491876199879d97c8ed956eabb0c2e73
> +
> +Upstream-Status: Backport
> +CVE: CVE-2013-0340
> +Signed-off-by: Jasper Orschulko
> +---
> + lib/expat.h | 21 +-
> + lib/internal.h | 30 +
> + lib/libexpat.def | 3 +
> + lib/libexpatw.def | 3 +
> + lib/xmlparse.c | 1147 +--
> + 5 files changed, 1143 insertions(+), 61 deletions(-)
> +
> +diff --git a/lib/expat.h b/lib/expat.h
> +index 48a6e2a3..0fb70d9d 100644
> +--- a/lib/expat.h
> b/lib/expat.h
> +@@ -115,7 +115,9 @@ enum XML_Error {
> + XML_ERROR_RESERVED_PREFIX_XMLNS,
> + XML_ERROR_RESERVED_NAMESPACE_URI,
> + /* Added in 2.2.1. */
> +- XML_ERROR_INVALID_ARGUMENT
> ++ XML_ERROR_INVALID_ARGUMENT,
> ++ /* Added in 2.4.0. */
> ++ XML_ERROR_AMPLIFICATION_LIMIT_BREACH
> + };
> +
> + enum XML_Content_Type {
> +@@ -997,7 +999,10 @@ enum XML_FeatureEnum {
> + XML_FEATURE_SIZEOF_XML_LCHAR,
> + XML_FEATURE_NS,
> + XML_FEATURE_LARGE_SIZE,
> +- XML_FEATURE_ATTR_INFO
> ++ XML_FEATURE_ATTR_INFO,
> ++ /* Added in Expat 2.4.0. */
> ++
> XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DE
> FA
> ULT,
> ++
> XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEF
> AU
> LT
> + /* Additional features must be added to the end of this enum. */
> + };
> +
> +@@ -1010,6 +1015,18 @@ typedef struct {
> + XMLPARSEAPI(const XML_Feature *)
> + XML_GetFeatureList(void);
> +
> ++#ifdef XML_DTD
> ++/* Added in Expat 2.4.0. */
> ++XMLPARSEAPI(XML_Bool)
> ++XML_SetBillionLaughsAttackProtectionMaximumAmplification(
> ++ XML_Parser parser, float maximumAmplificationFactor);
> ++
> ++/* Added in Expat 2.4.0. */
> ++XMLPARSEAPI(XML_Bool)
> ++XML_SetBillionLaughsAttackProtectionActivationThreshold(
> ++ XML_Parser parser, unsigned long long
> activationThresholdBytes);
> ++#endif
> ++
> + /* Expat follows the semantic versioning convention.
> + See http://semver.org.
> + */
> +diff --git a/lib/internal.h b/lib/internal.h
> +index 60913dab..d8b31fa2 100644
> +--- a/lib/internal.h
> b/lib/internal.h
> +@@ -101,10 +101,40 @@
> + # endif
> + #endif
> +
> ++#include // ULONG_MAX
> ++
> ++#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
> ++# define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
> ++# if defined(_WIN64) // Note: modifier "td" does not work for
> MinGW
> ++# define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "I64d"
> ++# else
> ++# define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
> ++# endif
> ++#else
> ++# define EXPAT_FMT_ULL(midpart) "%" midpart "llu"
> ++# if ! defined(ULONG_MAX)
> ++# error Compiler did not define ULONG_MAX for us
>
Re: [OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
expat < 4.0 is vulnerable to billion laughs attacks (see
[https://github.com/libexpat/libexpat/issues/34]). This patch backports
the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8
and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream.
Additionally, the SRC_URI had to be adjusted due to renaming of the
source archive
Signed-off-by: Jasper Orschulko
---
.../expat/expat/CVE-2013-0340.patch | 1758 +
.../expat/expat/libtool-tag.patch | 41 +-
meta/recipes-core/expat/expat_2.2.9.bb| 10 +-
3 files changed, 1783 insertions(+), 26 deletions(-)
create mode 100644 meta/recipes-core/expat/expat/CVE-2013-0340.patch
diff --git a/meta/recipes-core/expat/expat/CVE-2013-0340.patch
b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
new file mode 100644
index 00..5ef749719d
--- /dev/null
+++ b/meta/recipes-core/expat/expat/CVE-2013-0340.patch
@@ -0,0 +1,1758 @@
+From a644ccf25392523b1329872310e24d0fc5f40629 Mon Sep 17 00:00:00 2001
+From: Sebastian Pipping
+Date: Mon, 19 Apr 2021 21:42:51 +0200
+Subject: [PATCH] expat: Backport fix for CVE-2013-0340
+
+Issue: https://github.com/libexpat/libexpat/issues/34
+
+This patch cherry-picks the following commits from upstream release
+2.4.0 onto 2.2.9:
+
+- b1d039607d3d8a042bf0466bfcc1c0f104e353c8
+- 60959f2b491876199879d97c8ed956eabb0c2e73
+
+Upstream-Status: Backport
+CVE: CVE-2013-0340
+Signed-off-by: Jasper Orschulko
+---
+ lib/expat.h | 21 +-
+ lib/internal.h| 30 +
+ lib/libexpat.def |3 +
+ lib/libexpatw.def |3 +
+ lib/xmlparse.c| 1147 +--
+ 5 files changed, 1143 insertions(+), 61 deletions(-)
+
+diff --git a/lib/expat.h b/lib/expat.h
+index 48a6e2a3..0fb70d9d 100644
+--- a/lib/expat.h
b/lib/expat.h
+@@ -115,7 +115,9 @@ enum XML_Error {
+ XML_ERROR_RESERVED_PREFIX_XMLNS,
+ XML_ERROR_RESERVED_NAMESPACE_URI,
+ /* Added in 2.2.1. */
+- XML_ERROR_INVALID_ARGUMENT
++ XML_ERROR_INVALID_ARGUMENT,
++ /* Added in 2.4.0. */
++ XML_ERROR_AMPLIFICATION_LIMIT_BREACH
+ };
+
+ enum XML_Content_Type {
+@@ -997,7 +999,10 @@ enum XML_FeatureEnum {
+ XML_FEATURE_SIZEOF_XML_LCHAR,
+ XML_FEATURE_NS,
+ XML_FEATURE_LARGE_SIZE,
+- XML_FEATURE_ATTR_INFO
++ XML_FEATURE_ATTR_INFO,
++ /* Added in Expat 2.4.0. */
++
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFA
ULT,
++
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAU
LT
+ /* Additional features must be added to the end of this enum. */
+ };
+
+@@ -1010,6 +1015,18 @@ typedef struct {
+ XMLPARSEAPI(const XML_Feature *)
+ XML_GetFeatureList(void);
+
++#ifdef XML_DTD
++/* Added in Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionMaximumAmplification(
++XML_Parser parser, float maximumAmplificationFactor);
++
++/* Added in Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionActivationThreshold(
++XML_Parser parser, unsigned long long activationThresholdBytes);
++#endif
++
+ /* Expat follows the semantic versioning convention.
+See http://semver.org.
+ */
+diff --git a/lib/internal.h b/lib/internal.h
+index 60913dab..d8b31fa2 100644
+--- a/lib/internal.h
b/lib/internal.h
+@@ -101,10 +101,40 @@
+ # endif
+ #endif
+
++#include // ULONG_MAX
++
++#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
++# define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
++# if defined(_WIN64) // Note: modifier "td" does not work for MinGW
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "I64d"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#else
++# define EXPAT_FMT_ULL(midpart) "%" midpart "llu"
++# if ! defined(ULONG_MAX)
++#error Compiler did not define ULONG_MAX for us
++# elif ULONG_MAX == 18446744073709551615u // 2^64-1
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "ld"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#endif
++
+ #ifndef UNUSED_P
+ # define UNUSED_P(p) (void)p
+ #endif
+
++/* NOTE BEGIN If you ever patch these defaults to greater values
++ for non-attack XML payload in your environment,
++ please file a bug report with libexpat. Thank you!
++*/
++#define
EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT
\
++ 100.0f
++#define
EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT
\
++ 8388608 // 8 MiB, 2^23
++/* NOTE END */
++
+ #ifdef __cplusplus
+ extern "C" {
+ #endif
+diff --git a/lib/libexpat.def b/lib/libexpat.def
+index 16faf595..5aefa6df 100644
+--- a/lib/libexpat.def
b/lib/libexpat.def
+@@ -76,3 +76,6 @@ EXPORTS
+ XML_SetHashSalt @67
+ ; added with version 2.2.5
+ _INTERNAL_trim_to_complete_utf8_characters @68
++; added with version 2.4.0
++ XML_S
Re: [OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 P.S.: I was looking at https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Example:_CVE_patch_header and this page as far as I can tell only mentions the patch header convention, not the file name itself. Maybe this needs an update? :) - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Ostendstraße 1-14 | 12459 Berlin https://iris-sensing.com/ On Wed, 2021-06-16 at 05:09 -1000, Steve Sakoman wrote: > On Wed, Jun 16, 2021 at 4:49 AM Jasper Orschulko > wrote: > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > P.S.: I am not too familiar with expat, this particular CVE, not with > > the practise of backporting security patches, so someone(TM) should > > definitely take a closer look at this first. > > Will do! > > A few initial comments: > > 1. Please don't PGP sign patch emails :-) > 2. Change the patch file name to CVE-2013-0340.patch > > Other than that it looks OK at first glance. > > For reference the patch requirements for CVE's are outlined at: > > https://wiki.yoctoproject.org/wiki/Security > > in the "Patch name convention and commit message" section. > > Thanks for helping with CVEs! > > Steve > > > > > > With best regards > > > > Jasper Orschulko > > DevOps Engineer > > > > Tel. +49 30 58 58 14 265 > > Fax +49 30 58 58 14 999 > > jasper.orschu...@iris-sensing.com > > > > • • • • • • • • • • • • • • • • • • • • • • • • • • > > > > iris-GmbH > > infrared & intelligent sensors > > Ostendstraße 1-14 | 12459 Berlin > > > > https://iris-sensing.com/ > > > > > > > > > > On Wed, 2021-06-16 at 14:44 +, Jasper Orschulko wrote: > > > expat < 4.0 is vulnerable to billion laughs attacks (see > > > [https://github.com/libexpat/libexpat/issues/34]). This patch > > > backports > > > the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8 > > > and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream. > > > > > > Additionally, the SRC_URI had to be adjusted due to renaming of the > > > source archive > > > > > > Signed-off-by: Jasper Orschulko > > > --- > > > ...expat-Backport-fix-for-CVE-2013-0340.patch | 1758 > > > + > > > meta/recipes-core/expat/expat_2.2.9.bb | 3 +- > > > 2 files changed, 1760 insertions(+), 1 deletion(-) > > > create mode 100644 meta/recipes-core/expat/expat/0001-expat- > > > Backport- > > > fix-for-CVE-2013-0340.patch > > > > > > diff --git a/meta/recipes-core/expat/expat/0001-expat-Backport-fix- > > > for- > > > CVE-2013-0340.patch b/meta/recipes-core/expat/expat/0001-expat- > > > Backport-fix-for-CVE-2013-0340.patch > > > new file mode 100644 > > > index 00..b2ca066d96 > > > --- /dev/null > > > +++ b/meta/recipes-core/expat/expat/0001-expat-Backport-fix-for- > > > CVE- > > > 2013-0340.patch > > > @@ -0,0 +1,1758 @@ > > > +From 6f68eb0439f3c1807a143ff8c8972e74d404d8f0 Mon Sep 17 00:00:00 > > > 2001 > > > +From: Sebastian Pipping > > > +Date: Mon, 19 Apr 2021 21:42:51 +0200 > > > +Subject: [PATCH] expat: Backport fix for CVE-2013-0340 > > > + > > > +Issue: https://github.com/libexpat/libexpat/issues/34 > > > + > > > +This patch cherry-picks the following commits from upstream > > > release > > > +2.4.0 onto 2.2.9: > > > + > > > +- b1d039607d3d8a042bf0466bfcc1c0f104e353c8 > > > +- 60959f2b491876199879d97c8ed956eabb0c2e73 > > > + > > > +Upstream-Status: Backport > > > +CVE: CVE-2013-0340 > > > +Signed-off-by: Jasper Orschulko > > > > > > +--- > > > + expat/lib/expat.h | 21 +- > > > + expat/lib/internal.h | 30 + > > > + expat/lib/libexpat.def | 3 + > > > + expat/lib/libexpatw.def | 3 + > > > + expat/lib/xmlparse.c | 1147 > > > +- > > > - > > > + 5 files changed, 1143 insertions(+), 61 deletions(-) > > > + > > > +diff --git a/expat/lib/expat.h b/expat/lib/expat.h > > > +index 48a6e2a3..796086c2 100644 > > > +--- a/expat/lib/expat.h > > > b/expat/lib/expat.h > > > +@@
Re: [OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Steve! Thanks for the quick feedback! I just noticed that the archive folder structure from sourceforge differs to to the git content, thus the "inner" patch currently fails. Oops! I'm thinking about setting the git repository as SRC_URI, as the expat project is currently moving away from sourceforge towards github. Also, we would not be affected by random archive renaming ;) What do you think? - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 jasper.orschu...@iris-sensing.com • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Ostendstraße 1-14 | 12459 Berlin https://iris-sensing.com/ On Wed, 2021-06-16 at 05:09 -1000, Steve Sakoman wrote: > On Wed, Jun 16, 2021 at 4:49 AM Jasper Orschulko > wrote: > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > P.S.: I am not too familiar with expat, this particular CVE, not > > with > > the practise of backporting security patches, so someone(TM) should > > definitely take a closer look at this first. > > Will do! > > A few initial comments: > > 1. Please don't PGP sign patch emails :-) > 2. Change the patch file name to CVE-2013-0340.patch > > Other than that it looks OK at first glance. > > For reference the patch requirements for CVE's are outlined at: > > https://wiki.yoctoproject.org/wiki/Security > > in the "Patch name convention and commit message" section. > > Thanks for helping with CVEs! > > Steve > > > > > > With best regards > > > > Jasper Orschulko > > DevOps Engineer > > > > Tel. +49 30 58 58 14 265 > > Fax +49 30 58 58 14 999 > > jasper.orschu...@iris-sensing.com > > > > • • • • • • • • • • • • • • • • • • • • • • • • • • > > > > iris-GmbH > > infrared & intelligent sensors > > Ostendstraße 1-14 | 12459 Berlin > > > > https://iris-sensing.com/ > > > > > > > > > > On Wed, 2021-06-16 at 14:44 +, Jasper Orschulko wrote: > > > expat < 4.0 is vulnerable to billion laughs attacks (see > > > [https://github.com/libexpat/libexpat/issues/34]). This patch > > > backports > > > the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8 > > > and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream. > > > > > > Additionally, the SRC_URI had to be adjusted due to renaming of > > > the > > > source archive > > > > > > Signed-off-by: Jasper Orschulko > > > > > > --- > > > ...expat-Backport-fix-for-CVE-2013-0340.patch | 1758 > > > + > > > meta/recipes-core/expat/expat_2.2.9.bb | 3 +- > > > 2 files changed, 1760 insertions(+), 1 deletion(-) > > > create mode 100644 meta/recipes-core/expat/expat/0001-expat- > > > Backport- > > > fix-for-CVE-2013-0340.patch > > > > > > diff --git a/meta/recipes-core/expat/expat/0001-expat-Backport- > > > fix- > > > for- > > > CVE-2013-0340.patch b/meta/recipes-core/expat/expat/0001-expat- > > > Backport-fix-for-CVE-2013-0340.patch > > > new file mode 100644 > > > index 00..b2ca066d96 > > > --- /dev/null > > > +++ b/meta/recipes-core/expat/expat/0001-expat-Backport-fix-for- > > > CVE- > > > 2013-0340.patch > > > @@ -0,0 +1,1758 @@ > > > +From 6f68eb0439f3c1807a143ff8c8972e74d404d8f0 Mon Sep 17 > > > 00:00:00 > > > 2001 > > > +From: Sebastian Pipping > > > +Date: Mon, 19 Apr 2021 21:42:51 +0200 > > > +Subject: [PATCH] expat: Backport fix for CVE-2013-0340 > > > + > > > +Issue: https://github.com/libexpat/libexpat/issues/34 > > > + > > > +This patch cherry-picks the following commits from upstream > > > release > > > +2.4.0 onto 2.2.9: > > > + > > > +- b1d039607d3d8a042bf0466bfcc1c0f104e353c8 > > > +- 60959f2b491876199879d97c8ed956eabb0c2e73 > > > + > > > +Upstream-Status: Backport > > > +CVE: CVE-2013-0340 > > > +Signed-off-by: Jasper Orschulko > > > > > > +--- > > > + expat/lib/expat.h | 21 +- > > > + expat/lib/internal.h | 30 + > > > + expat/lib/libexpat.def | 3 + > > > + expat/lib/libexpatw.def | 3 + > > > + expat/lib/xmlparse.c | 1147 > > > +- > > > - > > > + 5 file
Re: [OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
P.S.: I am not too familiar with expat, this particular CVE, not with
the practise of backporting security patches, so someone(TM) should
definitely take a closer look at this first.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Ostendstraße 1-14 | 12459 Berlin
https://iris-sensing.com/
On Wed, 2021-06-16 at 14:44 +0000, Jasper Orschulko wrote:
> expat < 4.0 is vulnerable to billion laughs attacks (see
> [https://github.com/libexpat/libexpat/issues/34]). This patch
> backports
> the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8
> and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream.
>
> Additionally, the SRC_URI had to be adjusted due to renaming of the
> source archive
>
> Signed-off-by: Jasper Orschulko
> ---
> ...expat-Backport-fix-for-CVE-2013-0340.patch | 1758
> +
> meta/recipes-core/expat/expat_2.2.9.bb | 3 +-
> 2 files changed, 1760 insertions(+), 1 deletion(-)
> create mode 100644 meta/recipes-core/expat/expat/0001-expat-
> Backport-
> fix-for-CVE-2013-0340.patch
>
> diff --git a/meta/recipes-core/expat/expat/0001-expat-Backport-fix-
> for-
> CVE-2013-0340.patch b/meta/recipes-core/expat/expat/0001-expat-
> Backport-fix-for-CVE-2013-0340.patch
> new file mode 100644
> index 00..b2ca066d96
> --- /dev/null
> +++ b/meta/recipes-core/expat/expat/0001-expat-Backport-fix-for-CVE-
> 2013-0340.patch
> @@ -0,0 +1,1758 @@
> +From 6f68eb0439f3c1807a143ff8c8972e74d404d8f0 Mon Sep 17 00:00:00
> 2001
> +From: Sebastian Pipping
> +Date: Mon, 19 Apr 2021 21:42:51 +0200
> +Subject: [PATCH] expat: Backport fix for CVE-2013-0340
> +
> +Issue: https://github.com/libexpat/libexpat/issues/34
> +
> +This patch cherry-picks the following commits from upstream release
> +2.4.0 onto 2.2.9:
> +
> +- b1d039607d3d8a042bf0466bfcc1c0f104e353c8
> +- 60959f2b491876199879d97c8ed956eabb0c2e73
> +
> +Upstream-Status: Backport
> +CVE: CVE-2013-0340
> +Signed-off-by: Jasper Orschulko
> +---
> + expat/lib/expat.h | 21 +-
> + expat/lib/internal.h | 30 +
> + expat/lib/libexpat.def | 3 +
> + expat/lib/libexpatw.def | 3 +
> + expat/lib/xmlparse.c | 1147
> +-
> -
> + 5 files changed, 1143 insertions(+), 61 deletions(-)
> +
> +diff --git a/expat/lib/expat.h b/expat/lib/expat.h
> +index 48a6e2a3..796086c2 100644
> +--- a/expat/lib/expat.h
> b/expat/lib/expat.h
> +@@ -115,7 +115,9 @@ enum XML_Error {
> + XML_ERROR_RESERVED_PREFIX_XMLNS,
> + XML_ERROR_RESERVED_NAMESPACE_URI,
> + /* Added in 2.2.1. */
> +- XML_ERROR_INVALID_ARGUMENT
> ++ XML_ERROR_INVALID_ARGUMENT,
> ++ /* Backported from 2.4.0. */
> ++ XML_ERROR_AMPLIFICATION_LIMIT_BREACH
> + };
> +
> + enum XML_Content_Type {
> +@@ -997,7 +999,10 @@ enum XML_FeatureEnum {
> + XML_FEATURE_SIZEOF_XML_LCHAR,
> + XML_FEATURE_NS,
> + XML_FEATURE_LARGE_SIZE,
> +- XML_FEATURE_ATTR_INFO
> ++ XML_FEATURE_ATTR_INFO,
> ++ /* Added in Expat 2.4.0. */
> ++
> XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DE
> FA
> ULT,
> ++
> XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEF
> AU
> LT
> + /* Additional features must be added to the end of this enum. */
> + };
> +
> +@@ -1010,6 +1015,18 @@ typedef struct {
> + XMLPARSEAPI(const XML_Feature *)
> + XML_GetFeatureList(void);
> +
> ++#ifdef XML_DTD
> ++/* Backported from Expat 2.4.0. */
> ++XMLPARSEAPI(XML_Bool)
> ++XML_SetBillionLaughsAttackProtectionMaximumAmplification(
> ++ XML_Parser parser, float maximumAmplificationFactor);
> ++
> ++/* Backported from Expat 2.4.0. */
> ++XMLPARSEAPI(XML_Bool)
> ++XML_SetBillionLaughsAttackProtectionActivationThreshold(
> ++ XML_Parser parser, unsigned long long
> activationThresholdBytes);
> ++#endif
> ++
> + /* Expat follows the semantic versioning convention.
> + See http://semver.org.
> + */
> +diff --git a/expat/lib/internal.h b/expat/lib/internal.h
> +index 60913dab..d8b31fa2 100644
> +--- a/expat/lib/internal.h
> b/expat/lib/internal.h
> +@@ -101,10 +101,40 @@
> + # endif
> + #endif
> +
> ++#include // ULONG_MAX
> ++
> ++#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
> ++# define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
> ++# if defined(_WIN64) // Note: modifier "td" does not work for
> MinGW
> ++# define EXPAT_FMT_PTRDIFF_T(midpart) "%"
[OE-core][dunfell][PATCH] expat: fix CVE-2013-0340
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
expat < 4.0 is vulnerable to billion laughs attacks (see
[https://github.com/libexpat/libexpat/issues/34]). This patch backports
the commits b1d039607d3d8a042bf0466bfcc1c0f104e353c8
and 60959f2b491876199879d97c8ed956eabb0c2e73 from upstream.
Additionally, the SRC_URI had to be adjusted due to renaming of the
source archive
Signed-off-by: Jasper Orschulko
- ---
...expat-Backport-fix-for-CVE-2013-0340.patch | 1758 +
meta/recipes-core/expat/expat_2.2.9.bb|3 +-
2 files changed, 1760 insertions(+), 1 deletion(-)
create mode 100644 meta/recipes-core/expat/expat/0001-expat-Backport-
fix-for-CVE-2013-0340.patch
diff --git a/meta/recipes-core/expat/expat/0001-expat-Backport-fix-for-
CVE-2013-0340.patch b/meta/recipes-core/expat/expat/0001-expat-
Backport-fix-for-CVE-2013-0340.patch
new file mode 100644
index 00..b2ca066d96
- --- /dev/null
+++ b/meta/recipes-core/expat/expat/0001-expat-Backport-fix-for-CVE-
2013-0340.patch
@@ -0,0 +1,1758 @@
+From 6f68eb0439f3c1807a143ff8c8972e74d404d8f0 Mon Sep 17 00:00:00 2001
+From: Sebastian Pipping
+Date: Mon, 19 Apr 2021 21:42:51 +0200
+Subject: [PATCH] expat: Backport fix for CVE-2013-0340
+
+Issue: https://github.com/libexpat/libexpat/issues/34
+
+This patch cherry-picks the following commits from upstream release
+2.4.0 onto 2.2.9:
+
+- b1d039607d3d8a042bf0466bfcc1c0f104e353c8
+- 60959f2b491876199879d97c8ed956eabb0c2e73
+
+Upstream-Status: Backport
+CVE: CVE-2013-0340
+Signed-off-by: Jasper Orschulko
+---
+ expat/lib/expat.h | 21 +-
+ expat/lib/internal.h| 30 +
+ expat/lib/libexpat.def |3 +
+ expat/lib/libexpatw.def |3 +
+ expat/lib/xmlparse.c| 1147 +-
- -
+ 5 files changed, 1143 insertions(+), 61 deletions(-)
+
+diff --git a/expat/lib/expat.h b/expat/lib/expat.h
+index 48a6e2a3..796086c2 100644
+--- a/expat/lib/expat.h
b/expat/lib/expat.h
+@@ -115,7 +115,9 @@ enum XML_Error {
+ XML_ERROR_RESERVED_PREFIX_XMLNS,
+ XML_ERROR_RESERVED_NAMESPACE_URI,
+ /* Added in 2.2.1. */
+- XML_ERROR_INVALID_ARGUMENT
++ XML_ERROR_INVALID_ARGUMENT,
++ /* Backported from 2.4.0. */
++ XML_ERROR_AMPLIFICATION_LIMIT_BREACH
+ };
+
+ enum XML_Content_Type {
+@@ -997,7 +999,10 @@ enum XML_FeatureEnum {
+ XML_FEATURE_SIZEOF_XML_LCHAR,
+ XML_FEATURE_NS,
+ XML_FEATURE_LARGE_SIZE,
+- XML_FEATURE_ATTR_INFO
++ XML_FEATURE_ATTR_INFO,
++ /* Added in Expat 2.4.0. */
++
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFA
ULT,
++
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAU
LT
+ /* Additional features must be added to the end of this enum. */
+ };
+
+@@ -1010,6 +1015,18 @@ typedef struct {
+ XMLPARSEAPI(const XML_Feature *)
+ XML_GetFeatureList(void);
+
++#ifdef XML_DTD
++/* Backported from Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionMaximumAmplification(
++XML_Parser parser, float maximumAmplificationFactor);
++
++/* Backported from Expat 2.4.0. */
++XMLPARSEAPI(XML_Bool)
++XML_SetBillionLaughsAttackProtectionActivationThreshold(
++XML_Parser parser, unsigned long long activationThresholdBytes);
++#endif
++
+ /* Expat follows the semantic versioning convention.
+See http://semver.org.
+ */
+diff --git a/expat/lib/internal.h b/expat/lib/internal.h
+index 60913dab..d8b31fa2 100644
+--- a/expat/lib/internal.h
b/expat/lib/internal.h
+@@ -101,10 +101,40 @@
+ # endif
+ #endif
+
++#include // ULONG_MAX
++
++#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
++# define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
++# if defined(_WIN64) // Note: modifier "td" does not work for MinGW
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "I64d"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#else
++# define EXPAT_FMT_ULL(midpart) "%" midpart "llu"
++# if ! defined(ULONG_MAX)
++#error Compiler did not define ULONG_MAX for us
++# elif ULONG_MAX == 18446744073709551615u // 2^64-1
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "ld"
++# else
++#define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
++# endif
++#endif
++
+ #ifndef UNUSED_P
+ # define UNUSED_P(p) (void)p
+ #endif
+
++/* NOTE BEGIN If you ever patch these defaults to greater values
++ for non-attack XML payload in your environment,
++ please file a bug report with libexpat. Thank you!
++*/
++#define
EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT
\
++ 100.0f
++#define
EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT
\
++ 8388608 // 8 MiB, 2^23
++/* NOTE END */
++
+ #ifdef __cplusplus
+ extern "C" {
+ #endif
+diff --git a/expat/lib/libexpat.def b/expat/lib/libexpat.def
+index 16faf595..b5e5
[OE-core][dunfell][hardknott] expat: upstream package renamed
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi,
recently the expat upstream sources have been renamed to
${PN}-{PV}-RENAMED-VULNERABLE-PLEASE-USE-2.4.1-INSTEAD.tar.xz
which causes the fetch to fail. This effects all layers except master.
IMO we now have two options:
1. change the SRC_URI and backport security fixes (which seems sensible
for the dunfell LTS branch)
2. update the recipes to use 2.4.1
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
jasper.orschu...@iris-sensing.com
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Ostendstraße 1-14 | 12459 Berlin
https://iris-sensing.com/
-BEGIN PGP SIGNATURE-
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmDJ4sYACgkQYgqew07V
MNVEMAf9GQsWGjXK1vX+EQAU63VejQexh+KogMmHbRgT2e/9DwOXxs78zfL0/BLh
wWg9ijseom12nApdZHzjeSGs6ZAJUVVIDJfEqjfHAT06ft1yJ0K06tf06zXxr2bI
YTu9yXwn4wugXwmAjfEKn5j0lUMZvgGaE2PsTi40u70GjAK5dTTzZ29apwkfQOG9
qSZ75aL1DjEa2RM8LmgmHZlpFSIZsBTHPgf1WQmlcOupURSRAMJ2wqPDUuu3Zy1Q
IeMiEF6sQVI7M5m69Ia8U2t9l/5eR4x8vfnJJEbBmC8EJAwbDp987dey35Q5cex7
N0ocaNz+Gei2Aenot+qd30mgE1so1w==
=7Zqx
-END PGP SIGNATURE-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153005):
https://lists.openembedded.org/g/openembedded-core/message/153005
Mute This Topic: https://lists.openembedded.org/mt/83578374/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
